You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: src/UserGuide/Master/Tree/User-Manual/Streaming_apache.md
+9-9Lines changed: 9 additions & 9 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -64,17 +64,17 @@ The design of user programming interfaces for stream processing plugins follows
64
64
65
65
In the user programming interface of stream processing plugins, events abstract the write operations of database data. Events are captured by the local stream processing engine and passed sequentially through the three stages of stream processing, namely Pipe Source, Pipe Processor, and Pipe Sink plugins. User logic is triggered and executed within these three plugins.
66
66
67
-
To accommodate both low-latency stream processing in low-load scenarios and high-throughput stream processing in high-load scenarios at the edge, the stream processing engine dynamically chooses the processing objects from operation logs and data files. Therefore, the user programming interface for stream processing requires the user to provide the handling logic for two types of events: TabletInsertionEvent for operation log write events and TsFileInsertionEvent for data file write events.
67
+
To accommodate both low-latency stream processing in low-load scenarios and high-throughput stream processing in high-load scenarios at the edge, the stream processing engine dynamically chooses whether to process real-time writes or data files. Therefore, the user programming interface for stream processing requires users to provide the handling logic for two types of events: Tablet write events (`TabletInsertionEvent`) and TsFile write events (`TsFileInsertionEvent`).
The TabletInsertionEvent is a high-level data abstraction for user write requests, which provides the ability to manipulate the underlying data of the write request by providing a unified operation interface.
71
+
The Tablet write event (`TabletInsertionEvent`) is a high-level data abstraction for user write requests. By providing a unified operation interface, it enables users to process the written data.
72
72
73
-
For different database deployments, the underlying storage structure corresponding to the operation log write event is different. For stand-alone deployment scenarios, the operation log write event is an encapsulation of write-ahead log (WAL) entries; for distributed deployment scenarios, the operation log write event is an encapsulation of individual node consensus protocol operation log entries.
73
+
Since V2.0.5, the stream processing framework obtains real-time written data directly from write requests and encapsulates it as a `TabletInsertionEvent`, instead of reading the data from operation logs. This change does not affect the `TabletInsertionEvent` interface or the usage of existing plugins.
74
74
75
-
For write operations generated by different write request interfaces of the database, the data structure of the request structure corresponding to the operation log write event is also different.IoTDB provides many write interfaces such as InsertRecord, InsertRecords, InsertTablet, InsertTablets, and so on, and each kind of write request uses a completely different serialisation method to generate a write request. completely different serialisation methods and generate different binary entries.
75
+
For write operations generated through different database write interfaces, the data structures corresponding to Tablet write events also differ. IoTDB provides multiple write interfaces, such as `InsertRecord`, `InsertRecords`, `InsertTablet`, and `InsertTablets`. The data structures of different write requests also vary.
76
76
77
-
The existence of operation log write events provides users with a unified view of data operations, which shields the implementation differences of the underlying data structures, greatly reduces the programming threshold for users, and improves the ease of use of the functionality.
77
+
Tablet write events provide users with a unified view of data operations, hiding implementation differences among underlying data structures, greatly lowering the programming barrier, and improving usability.
78
78
79
79
```java
80
80
/** TabletInsertionEvent is used to define the event of data insertion. */
@@ -98,9 +98,9 @@ public interface TabletInsertionEvent extends Event {
The TsFileInsertionEvent represents a high-level abstraction of the database's disk flush operation and is a collection of multiple TabletInsertionEvents.
103
+
The TsFile write event (`TsFileInsertionEvent`) is a high-level abstraction of database file flush operations and a collection of multiple Tablet write events (`TabletInsertionEvent`).
104
104
105
105
IoTDB's storage engine is based on the LSM (Log-Structured Merge) structure. When data is written, the write operations are first flushed to log-structured files, while the written data is also stored in memory. When the memory reaches its capacity limit, a flush operation is triggered, converting the data in memory into a database file while deleting the previously written log entries. During the conversion from memory data to database file data, two compression processes, encoding compression and universal compression, are applied. As a result, the data in the database file occupies less space compared to the original data in memory.
106
106
@@ -110,7 +110,7 @@ In summary, the data file write event appears in the event stream of stream proc
110
110
111
111
1. Historical data extraction: Before a stream processing task starts, all persisted write data exists in the form of TsFiles. When collecting historical data at the beginning of a stream processing task, the historical data is abstracted as TsFileInsertionEvent.
112
112
113
-
2. Real-time data extraction: During the execution of a stream processing task, if the speed of processing the log entries representing real-time operations is slower than the rate of write requests, the unprocessed log entries will be persisted to disk in the form of TsFiles. When these data are extracted by the stream processing engine, they are abstracted as TsFileInsertionEvent.
113
+
2. Real-time data extraction: While a stream processing task is running, real-time writes are usually processed as `TabletInsertionEvent`s. When unprocessed data accumulates, the stream processing engine can extract data in batches from TsFiles that have already been flushed to disk and process it as `TsFileInsertionEvent`s.
Copy file name to clipboardExpand all lines: src/UserGuide/V1.3.x/User-Manual/Streaming_apache.md
+9-9Lines changed: 9 additions & 9 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -64,17 +64,17 @@ The design of user programming interfaces for stream processing plugins follows
64
64
65
65
In the user programming interface of stream processing plugins, events abstract the write operations of database data. Events are captured by the local stream processing engine and passed sequentially through the three stages of stream processing, namely Pipe Source, Pipe Processor, and Pipe Sink plugins. User logic is triggered and executed within these three plugins.
66
66
67
-
To accommodate both low-latency stream processing in low-load scenarios and high-throughput stream processing in high-load scenarios at the edge, the stream processing engine dynamically chooses the processing objects from operation logs and data files. Therefore, the user programming interface for stream processing requires the user to provide the handling logic for two types of events: TabletInsertionEvent for operation log write events and TsFileInsertionEvent for data file write events.
67
+
To accommodate both low-latency stream processing in low-load scenarios and high-throughput stream processing in high-load scenarios at the edge, the stream processing engine dynamically chooses whether to process real-time writes or data files. Therefore, the user programming interface for stream processing requires users to provide the handling logic for two types of events: Tablet write events (`TabletInsertionEvent`) and TsFile write events (`TsFileInsertionEvent`).
The TabletInsertionEvent is a high-level data abstraction for user write requests, which provides the ability to manipulate the underlying data of the write request by providing a unified operation interface.
71
+
The Tablet write event (`TabletInsertionEvent`) is a high-level data abstraction for user write requests. By providing a unified operation interface, it enables users to process the written data.
72
72
73
-
For different database deployments, the underlying storage structure corresponding to the operation log write event is different. For stand-alone deployment scenarios, the operation log write event is an encapsulation of write-ahead log (WAL) entries; for distributed deployment scenarios, the operation log write event is an encapsulation of individual node consensus protocol operation log entries.
73
+
Since V1.3.5, the stream processing framework obtains real-time written data directly from write requests and encapsulates it as a `TabletInsertionEvent`, instead of reading the data from operation logs. This change does not affect the `TabletInsertionEvent` interface or the usage of existing plugins.
74
74
75
-
For write operations generated by different write request interfaces of the database, the data structure of the request structure corresponding to the operation log write event is also different.IoTDB provides many write interfaces such as InsertRecord, InsertRecords, InsertTablet, InsertTablets, and so on, and each kind of write request uses a completely different serialisation method to generate a write request. completely different serialisation methods and generate different binary entries.
75
+
For write operations generated through different database write interfaces, the data structures corresponding to Tablet write events also differ. IoTDB provides multiple write interfaces, such as `InsertRecord`, `InsertRecords`, `InsertTablet`, and `InsertTablets`. The data structures of different write requests also vary.
76
76
77
-
The existence of operation log write events provides users with a unified view of data operations, which shields the implementation differences of the underlying data structures, greatly reduces the programming threshold for users, and improves the ease of use of the functionality.
77
+
Tablet write events provide users with a unified view of data operations, hiding implementation differences among underlying data structures, greatly lowering the programming barrier, and improving usability.
78
78
79
79
```java
80
80
/** TabletInsertionEvent is used to define the event of data insertion. */
@@ -98,9 +98,9 @@ public interface TabletInsertionEvent extends Event {
The TsFileInsertionEvent represents a high-level abstraction of the database's disk flush operation and is a collection of multiple TabletInsertionEvents.
103
+
The TsFile write event (`TsFileInsertionEvent`) is a high-level abstraction of database file flush operations and a collection of multiple Tablet write events (`TabletInsertionEvent`).
104
104
105
105
IoTDB's storage engine is based on the LSM (Log-Structured Merge) structure. When data is written, the write operations are first flushed to log-structured files, while the written data is also stored in memory. When the memory reaches its capacity limit, a flush operation is triggered, converting the data in memory into a database file while deleting the previously written log entries. During the conversion from memory data to database file data, two compression processes, encoding compression and universal compression, are applied. As a result, the data in the database file occupies less space compared to the original data in memory.
106
106
@@ -110,7 +110,7 @@ In summary, the data file write event appears in the event stream of stream proc
110
110
111
111
1. Historical data extraction: Before a stream processing task starts, all persisted write data exists in the form of TsFiles. When collecting historical data at the beginning of a stream processing task, the historical data is abstracted as TsFileInsertionEvent.
112
112
113
-
2. Real-time data extraction: During the execution of a stream processing task, if the speed of processing the log entries representing real-time operations is slower than the rate of write requests, the unprocessed log entries will be persisted to disk in the form of TsFiles. When these data are extracted by the stream processing engine, they are abstracted as TsFileInsertionEvent.
113
+
2. Real-time data extraction: While a stream processing task is running, real-time writes are usually processed as `TabletInsertionEvent`s. When unprocessed data accumulates, the stream processing engine can extract data in batches from TsFiles that have already been flushed to disk and process it as `TsFileInsertionEvent`s.
Copy file name to clipboardExpand all lines: src/UserGuide/dev-1.3/User-Manual/Streaming_apache.md
+9-9Lines changed: 9 additions & 9 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -64,17 +64,17 @@ The design of user programming interfaces for stream processing plugins follows
64
64
65
65
In the user programming interface of stream processing plugins, events abstract the write operations of database data. Events are captured by the local stream processing engine and passed sequentially through the three stages of stream processing, namely Pipe Source, Pipe Processor, and Pipe Sink plugins. User logic is triggered and executed within these three plugins.
66
66
67
-
To accommodate both low-latency stream processing in low-load scenarios and high-throughput stream processing in high-load scenarios at the edge, the stream processing engine dynamically chooses the processing objects from operation logs and data files. Therefore, the user programming interface for stream processing requires the user to provide the handling logic for two types of events: TabletInsertionEvent for operation log write events and TsFileInsertionEvent for data file write events.
67
+
To accommodate both low-latency stream processing in low-load scenarios and high-throughput stream processing in high-load scenarios at the edge, the stream processing engine dynamically chooses whether to process real-time writes or data files. Therefore, the user programming interface for stream processing requires users to provide the handling logic for two types of events: Tablet write events (`TabletInsertionEvent`) and TsFile write events (`TsFileInsertionEvent`).
The TabletInsertionEvent is a high-level data abstraction for user write requests, which provides the ability to manipulate the underlying data of the write request by providing a unified operation interface.
71
+
The Tablet write event (`TabletInsertionEvent`) is a high-level data abstraction for user write requests. By providing a unified operation interface, it enables users to process the written data.
72
72
73
-
For different database deployments, the underlying storage structure corresponding to the operation log write event is different. For stand-alone deployment scenarios, the operation log write event is an encapsulation of write-ahead log (WAL) entries; for distributed deployment scenarios, the operation log write event is an encapsulation of individual node consensus protocol operation log entries.
73
+
Since V1.3.5, the stream processing framework obtains real-time written data directly from write requests and encapsulates it as a `TabletInsertionEvent`, instead of reading the data from operation logs. This change does not affect the `TabletInsertionEvent` interface or the usage of existing plugins.
74
74
75
-
For write operations generated by different write request interfaces of the database, the data structure of the request structure corresponding to the operation log write event is also different.IoTDB provides many write interfaces such as InsertRecord, InsertRecords, InsertTablet, InsertTablets, and so on, and each kind of write request uses a completely different serialisation method to generate a write request. completely different serialisation methods and generate different binary entries.
75
+
For write operations generated through different database write interfaces, the data structures corresponding to Tablet write events also differ. IoTDB provides multiple write interfaces, such as `InsertRecord`, `InsertRecords`, `InsertTablet`, and `InsertTablets`. The data structures of different write requests also vary.
76
76
77
-
The existence of operation log write events provides users with a unified view of data operations, which shields the implementation differences of the underlying data structures, greatly reduces the programming threshold for users, and improves the ease of use of the functionality.
77
+
Tablet write events provide users with a unified view of data operations, hiding implementation differences among underlying data structures, greatly lowering the programming barrier, and improving usability.
78
78
79
79
```java
80
80
/** TabletInsertionEvent is used to define the event of data insertion. */
@@ -98,9 +98,9 @@ public interface TabletInsertionEvent extends Event {
The TsFileInsertionEvent represents a high-level abstraction of the database's disk flush operation and is a collection of multiple TabletInsertionEvents.
103
+
The TsFile write event (`TsFileInsertionEvent`) is a high-level abstraction of database file flush operations and a collection of multiple Tablet write events (`TabletInsertionEvent`).
104
104
105
105
IoTDB's storage engine is based on the LSM (Log-Structured Merge) structure. When data is written, the write operations are first flushed to log-structured files, while the written data is also stored in memory. When the memory reaches its capacity limit, a flush operation is triggered, converting the data in memory into a database file while deleting the previously written log entries. During the conversion from memory data to database file data, two compression processes, encoding compression and universal compression, are applied. As a result, the data in the database file occupies less space compared to the original data in memory.
106
106
@@ -110,7 +110,7 @@ In summary, the data file write event appears in the event stream of stream proc
110
110
111
111
1. Historical data extraction: Before a stream processing task starts, all persisted write data exists in the form of TsFiles. When collecting historical data at the beginning of a stream processing task, the historical data is abstracted as TsFileInsertionEvent.
112
112
113
-
2. Real-time data extraction: During the execution of a stream processing task, if the speed of processing the log entries representing real-time operations is slower than the rate of write requests, the unprocessed log entries will be persisted to disk in the form of TsFiles. When these data are extracted by the stream processing engine, they are abstracted as TsFileInsertionEvent.
113
+
2. Real-time data extraction: While a stream processing task is running, real-time writes are usually processed as `TabletInsertionEvent`s. When unprocessed data accumulates, the stream processing engine can extract data in batches from TsFiles that have already been flushed to disk and process it as `TsFileInsertionEvent`s.
0 commit comments