Skip to main content
Version: 2.5

Stream Load

1. Does Stream Load support identifying column names held in the first row of the source data file? Or, does Stream Load support skipping the first row during data reading?​

Stream Load does not support identifying column names held in the first row of the source data file. Stream Load considers the first row to be normal data like the other rows. Additionally, Stream Load does not support skipping the first row during data reading. If the first row holds column names, take one of the following actions:

  • Modify the settings of the tool that you use to export the data. Then, re-export the data as a source data file that does not hold column names in the first row.
  • Use commands such as sed -i '1d' filename to delete the first row of the source data file.
  • In the load command or statement, use -H "where: <column_name> != '<column_name>'" to filter out the first row of the source data file. <column_name> is any of the column names held in the first row. Note that StarRocks first transforms and then filters the source data. Therefore, if the column names in the first row fail to be transformed into their matching destination data types, NULL values are returned for them. This means the destination StarRocks table cannot contain columns that are set to NOT NULL.
  • In the load command or statement, add -H "max_filter_ratio:0.01" to set a maximum error tolerance that is 1% or lower but can tolerate more than 1 error row, thereby allowing StarRocks to ignore the data transformation failures in the first row. In this case, the Stream Load job can still succeed even if ErrorURL is returned to indicate error rows. Do not set max_filter_ratio to a large value. If you set max_filter_ratio to a large value, some important data quality issues may be missed.

2. The data to be loaded into the partition column is not of standard DATE or INT type. For example, the data is in a format like 202106.00. How do I transform the data if I load it by using Stream Load?​

StarRocks supports transforming data at loading. For more information, see Transform data at loading.

Suppose that you want to load a CSV-formatted source data file named TEST and the file consists of four columns, NO, DATE, VERSION, and PRICE, among which the data from the DATE column is in a non-standard format such as 202106.00. If you want to use DATE as the partition column in StarRocks, you need to first create a StarRocks table, for example, one that consists of the following four columns: NO, VERSION, PRICE, and DATE. Then, you need to specify the data type of the DATE column of the StarRocks table as DATE, DATETIME, or INT. Finally, when you create a Stream Load job, you need to specify the following setting in the load command or statement to transform data from the source DATE column's data type to the destination column's data type:

-H "columns: NO,DATE_1, VERSION, PRICE, DATE=LEFT(DATE_1,6)"

In the preceding example, DATE_1 can be considered to be a temporarily named column mapping the destination DATE column, and the final results loaded into the destination DATE column are computed by the left() function. Note that you must first list the temporary names of the source columns and then use functions to transform data. The functions supported are scalar functions, including non-aggregate functions and window functions.

3. What do I do if my Stream Load job reports the "body exceed max size: 10737418240, limit: 10737418240" error?​

The size of the source data file exceeds 10 GB, which is the maximum file size supported by Stream Load. Take one of the following actions:

  • Use seq -w 0 n to split the source data file into smaller files.
  • Use curl -XPOST http://be_host:http_port/api/update_config?streaming_load_max_mb=<file_size> to adjust the value of the BE configuration item streaming_load_max_mb to increase the maximum file size.