Skip to content

Commit 159b2dd

Browse files
committed
docs: add Copy SQL export to TsFile for table model (V2.0.11)
1 parent 886cc58 commit 159b2dd

4 files changed

Lines changed: 582 additions & 44 deletions

File tree

‎src/UserGuide/Master/Table/Tools-System/Data-Export-Tool_apache.md‎

Lines changed: 146 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -1,32 +1,39 @@
11
# Data Export
22

33
## 1. Function Overview
4-
The data export tool `export-data.sh/bat` is located in the `tools` directory and can export query results from specified SQL statements into CSV, SQL, or TsFile (open-source time-series file format) formats. Its specific functionalities are as follows:
4+
5+
IoTDB supports two methods for data export:
6+
7+
* Data Export Tool: `export-data.sh/bat` is located in the `tools` directory and can export query results of specified SQL statements into CSV, SQL, and TsFile (open-source time-series file format) formats.
8+
* Copy SQL Export to TsFile: Write query results back to a TsFile at a specified path via SQL.
59

610
<table style="text-align: left;">
711
<tbody>
812
<tr> <th>File Format</th>
9-
<th>IoTDB Tool</th>
13+
<th>IoTDB Tool</th>
1014
<th>Description</th>
1115
</tr>
1216
<tr>
13-
<td>CSV</td>
14-
<td rowspan="3">export-data.sh/bat</td>
15-
<td>Plain text format for storing structured data. Must follow the CSV format specified below.</td>
17+
<td>CSV</td>
18+
<td rowspan="3">export-data.sh/bat</td>
19+
<td>Plain text format for storing structured data. Must follow the CSV format specified below.</td>
1620
</tr>
1721
<tr>
18-
<td>SQL</td>
19-
<td>File containing custom SQL statements.</td>
22+
<td>SQL</td>
23+
<td>File containing custom SQL statements.</td>
2024
</tr>
2125
<tr>
22-
<td >TsFile</td>
26+
<td rowspan="2">TsFile</td>
27+
<td>Open-source time-series file format.</td>
28+
</tr>
29+
<tr>
30+
<td>Copy SQL</td>
2331
<td>Open-source time-series file format.</td>
24-
</tr>
32+
</tr>
2533
</tbody>
2634
</table>
27-
2835

29-
## 2. Detailed Features
36+
## 2. Data Export Tool
3037
### 2.1 Common Parameters
3138
| Short | Full Parameter | Description | Required | Default |
3239
|----------------|--------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| ----------------- |----------------------------------------------|
@@ -167,3 +174,131 @@ Parse error: Missing required option: db
167174
> /tools/export-data.sh -ft tsfile -sql_dialect table -t /path/export/dir -start_time 0
168175
Parse error: Missing required option: db
169176
```
177+
178+
## 3. Copy SQL
179+
180+
> **Note: This feature is supported since V2.0.11.**
181+
182+
### 3.1 Command
183+
184+
```SQL
185+
// ---------------------------------------- Copy Statement ---------------------------------------------------------
186+
copyToStatement
187+
: COPY '(' query ')' TO fileName=string ((WITH)? copyToStatementOptions)?
188+
| COPY tableName=qualifiedName ('(' tableColumns=identifierList ')')? TO fileName=string ((WITH)? copyToStatementOptions)?
189+
;
190+
191+
copyToStatementOptions
192+
: '(' copyToStatementOption (',' copyToStatementOption)* ')'
193+
;
194+
195+
copyToStatementOption
196+
: FORMAT identifier
197+
| TABLE identifier
198+
| TAGS '(' identifierList ')'
199+
| TIME identifier
200+
| MEMORY_THRESHOLD memory=INTEGER_VALUE
201+
;
202+
```
203+
204+
#### Parameters
205+
206+
| Name | Description | Default |
207+
|---|---|---|
208+
| FORMAT | Export format. Currently only TsFile is supported. | TsFile |
209+
| TABLE | Specifies the table name in the generated TsFile. | If the query SQL involves only one table, that table name is used; otherwise, `default` is used. |
210+
| TIME | Specifies which column in the result set is used as the TIME column.<br>When manually specified: an error is reported if the column type is not TIMESTAMP or the specified column cannot be found.<br>When not manually specified: an error is also reported if a column named `time` exists but its type is not TIMESTAMP.<br>The time column is constructed with the following priority:<br>1. The column with the same name as the time column of the single table involved in the query<br>2. The column named "time" with the TIMESTAMP type as the time column<br>3. Use the current number of written rows for the corresponding device as time to generate the time column, with the column name "time" | - |
211+
| TAGS | Specifies which columns are TAG columns. When there are multiple TAG columns, the order in the final generated table is consistent with the specified order.<br>When manually specified: an error is reported if a column does not exist, a column type is not STRING, or duplicate column names exist.<br>If the query involves only one table and all tag columns of the table can be found in the query result set, they are inferred as the tag columns of that table; otherwise, the default value is an empty list, meaning all columns except the TIME column are treated as FIELD columns | Empty list |
212+
| MEMORY_THRESHOLD | Used for memory control when generating the TsFile (unit: byte). An error is reported when the manually specified value is less than or equal to 0. | 32MB |
213+
214+
#### Result Set
215+
216+
| Column | Data Type | Description |
217+
|---|---|---|
218+
| path | STRING | Absolute path of the generated target file |
219+
| row_count | INT64 | Total number of written rows |
220+
| device_count | INT64 | Number of generated devices |
221+
| size_in_bytes | INT64 | Size of the generated target file |
222+
| table_name | STRING | Table name in the target file. If it is auto-generated, it will be marked with `(auto_gen)`. |
223+
| time_column | STRING | Name of the time column of the table in the target file. If it is auto-generated, it will be marked with `(auto_gen)`. |
224+
| tag_columns | STRING | Names of the tag columns of the table in the target file, separated by `,`. |
225+
226+
#### Other Notes
227+
228+
* File generation location:
229+
* If a file name is specified, the generated TsFile is saved under `${dn_data_dirs}/copy_to` of the DataNode directly connected to the client. When multiple directories are configured, the file is generated according to the strategy of the configuration item `dn_multi_dir_strategy`.
230+
* If a path is specified, the file is saved under the specified path.
231+
* Possible exceptions during execution:
232+
* An error is reported when out-of-order timestamps exist while writing to the TsFile according to the given schema
233+
* An error is reported when the file name is invalid or the target file already exists
234+
* Duplicate column names exist in the query result
235+
* Insufficient disk space
236+
237+
### 3.2 Examples
238+
239+
Taking table1 in the [Sample Data](../Reference/Sample-Data.md) as an example
240+
241+
1. Export all data in table1 to the file copysql1.tsfile via a select statement
242+
243+
```SQL
244+
IoTDB:database1> copy (select * from table1) to 'copysql1.tsfile'
245+
+-----------------------------------------------------+---------+------------+-------------+----------+-----------+-----------------------------+
246+
| path|row_count|device_count|size_in_bytes|table_name|time_column| tag_columns|
247+
+-----------------------------------------------------+---------+------------+-------------+----------+-----------+-----------------------------+
248+
| /iotdb/data/datanode/data/copy_to/copysql1.tsfile| 18| 6| 4636| table1| time|[region, plant_id, device_id]|
249+
+-----------------------------------------------------+---------+------------+-------------+----------+-----------+-----------------------------+
250+
Total line number = 1
251+
It costs 0.336s
252+
```
253+
254+
2. Export all data in table1 to the file copysql2.tsfile via the table name
255+
256+
```SQL
257+
IoTDB:database1> copy table1 to 'copysql2.tsfile'
258+
+-----------------------------------------------------+---------+------------+-------------+----------+-----------+-----------------------------+
259+
| path|row_count|device_count|size_in_bytes|table_name|time_column| tag_columns|
260+
+-----------------------------------------------------+---------+------------+-------------+----------+-----------+-----------------------------+
261+
| /iotdb/data/datanode/data/copy_to/copysql2.tsfile| 18| 6| 4636| table1| time|[region, plant_id, device_id]|
262+
+-----------------------------------------------------+---------+------------+-------------+----------+-----------+-----------------------------+
263+
Total line number = 1
264+
It costs 0.048s
265+
```
266+
267+
3. Export part of the data in table1 to the file copysql3.tsfile via the table name (columns)
268+
269+
```SQL
270+
IoTDB:database1> copy table1 (device_id,temperature) to 'copysql3.tsfile'
271+
+-----------------------------------------------------+---------+------------+-------------+----------+--------------+-----------+
272+
| path|row_count|device_count|size_in_bytes|table_name| time_column|tag_columns|
273+
+-----------------------------------------------------+---------+------------+-------------+----------+--------------+-----------+
274+
| /iotdb/data/datanode/data/copy_to/copysql3.tsfile| 18| 1| 558| table1|time(auto_gen)| []|
275+
+-----------------------------------------------------+---------+------------+-------------+----------+--------------+-----------+
276+
Total line number = 1
277+
It costs 0.064s
278+
```
279+
280+
4. Export the aggregation results of part of the data in table1 to the file copysql4.tsfile via a select statement
281+
282+
```SQL
283+
IoTDB:database1> copy (select count(temperature), count(humidity) from table1 group by device_id) to 'copysql4.tsfile'
284+
+-----------------------------------------------------+---------+------------+-------------+----------+--------------+-----------+
285+
| path|row_count|device_count|size_in_bytes|table_name| time_column|tag_columns|
286+
+-----------------------------------------------------+---------+------------+-------------+----------+--------------+-----------+
287+
| /iotdb/data/datanode/data/copy_to/copysql4.tsfile| 2| 1| 543| table1|time(auto_gen)| []|
288+
+-----------------------------------------------------+---------+------------+-------------+----------+--------------+-----------+
289+
Total line number = 1
290+
It costs 0.155s
291+
```
292+
293+
5. Export part of the data in table1 to the file copysql5.tsfile via a select statement, and specify the target table, time column, and tag columns
294+
295+
```SQL
296+
IoTDB:database1> copy (select time,region,device_id,temperature from table1 order by time) to 'copysql5.tsfile' (TABLE copytable, TIME time, TAGS (region,device_id))
297+
+-----------------------------------------------------+---------+------------+-------------+----------+-----------+-------------------+
298+
| path|row_count|device_count|size_in_bytes|table_name|time_column| tag_columns|
299+
+-----------------------------------------------------+---------+------------+-------------+----------+-----------+-------------------+
300+
| /iotdb/data/datanode/data/copy_to/copysql5.tsfile| 18| 4| 1199| copytable| time|[region, device_id]|
301+
+-----------------------------------------------------+---------+------------+-------------+----------+-----------+-------------------+
302+
Total line number = 1
303+
It costs 0.047s
304+
```

0 commit comments

Comments
 (0)