Data loading

🗃️ Loading overview

4 items

🗃️ Load from Object Storage

8 items

📄️ Load data from a local file system

StarRocks provides two methods of loading data from a local file system:

📄️ Load data from HDFS

StarRocks provides the following options for loading data from HDFS:

🗃️ Load data from Apache Kafka

2 items

🗃️ Load data from Apache Spark

2 items

📄️ Load data using INSERT

This topic describes how to load data into StarRocks by using a SQL statement - INSERT.

📄️ SQL Transaction

Start a simple SQL transaction to commit multiple INSERT statements in a batch.

📄️ Load data using Stream Load transaction interface

From v2.4 onwards, StarRocks provides a Stream Load transaction interface to implement two-phase commit (2PC) for transactions that are run to load data from external systems such as Apache Flink® and Apache Kafka®. The Stream Load transaction interface helps improve the performance of highly concurrent stream loads.

📄️ Realtime synchronization from MySQL

StarRocks supports multiple methods to synchronize data from MySQL to StarRocks in real time, delivering low latency real-time analytics of massive data.

📄️ Realtime synchronization from PostgreSQL

How to use a Flink CDC pipeline to capture PostgreSQL changes and stream them into StarRocks Primary Key tables in real time.

📄️ Continuously load data from Apache Flink®

StarRocks provides a self-developed connector named StarRocks Connector for Apache Flink® (Flink connector for short) to help you load data into a StarRocks table by using Flink. The basic principle is to accumulate the data and then load it all at a time into StarRocks through STREAM LOAD.

📄️ Continuously load data from Apache® Pulsar™

As of StarRocks version 2.5, Routine Load supports continuously loading data from Apache® Pulsar™. Pulsar is distributed, open source pub-sub messaging and streaming platform with a store-compute separation architecture. Loading data from Pulsar via Routine Load is similar to loading data from Apache Kafka. This topic uses CSV-formatted data as an example to introduce how to load data from Apache Pulsar via Routine Load.

📄️ Change data through loading

Primary Key tables provided by StarRocks allow you to make data changes to StarRocks tables by running Stream Load, Broker Load, or Routine Load jobs. These data changes include inserts, updates, and deletions. However, Primary Key tables do not support changing data by using Spark Load or INSERT.

📄️ Transform data at loading

StarRocks supports data transformation at loading.

📄️ Load data using tools

StarRocks and its ecosystem partners offer the following tools to help you seamlessly integrate StarRocks with external databases.

📄️ Strict mode

Strict mode is an optional property that you can configure for data loads. It affects the loading behavior and the final loaded data.

📄️ Load data from HDFS or cloud storage

StarRocks provides the loading method MySQL-based Broker Load to help you load a large amount of data from HDFS or cloud storage into StarRocks.

📄️ Introduction

You can import semi-structured data (for example, JSON) by using stream load or routine load.