Version: Latest-3.3

Delta Lake catalog

A Delta Lake catalog is a kind of external catalog that enables you to query data from Delta Lake without ingestion.

Also, you can directly transform and load data from Delta Lake by using INSERT INTO based on Delta Lake catalogs. StarRocks supports Delta Lake catalogs from v2.5 onwards.

To ensure successful SQL workloads on your Delta Lake cluster, your StarRocks cluster must be able to access the storage system and metastore of your Delta Lake cluster. StarRocks supports the following storage systems and metastores:

Distributed file system (HDFS) or object storage like AWS S3, Microsoft Azure Storage, Google GCS, or other S3-compatible storage system (for example, MinIO)
Metastore like Hive metastore or AWS Glue

NOTE

If you choose AWS S3 as storage, you can use HMS or AWS Glue as metastore. If you choose any other storage system, you can only use HMS as metastore.

Usage notes

The file format of Delta Lake that StarRocks supports is Parquet. Parquet files support the following compression formats: SNAPPY, LZ4, ZSTD, GZIP, and NO_COMPRESSION.
The data types of Delta Lake that StarRocks does not support are MAP and STRUCT.

Integration preparations

Before you create a Delta Lake catalog, make sure your StarRocks cluster can integrate with the storage system and metastore of your Delta Lake cluster.

AWS IAM

If your Delta Lake cluster uses AWS S3 as storage or AWS Glue as metastore, choose your suitable authentication method and make the required preparations to ensure that your StarRocks cluster can access the related AWS cloud resources.

The following authentication methods are recommended:

Instance profile
Assumed role
IAM user

Of the above-mentioned three authentication methods, instance profile is the most widely used.

For more information, see Preparation for authentication in AWS IAM.

HDFS

If you choose HDFS as storage, configure your StarRocks cluster as follows:

(Optional) Set the username that is used to access your HDFS cluster and Hive metastore. By default, StarRocks uses the username of the FE and BE or CN processes to access your HDFS cluster and Hive metastore. You can also set the username by adding export HADOOP_USER_NAME="<user_name>" at the beginning of the fe/conf/hadoop_env.sh file of each FE and at the beginning of the be/conf/hadoop_env.sh file of each BE or the cn/conf/hadoop_env.sh file of each CN. After you set the username in these files, restart each FE and each BE or CN to make the parameter settings take effect. You can set only one username for each StarRocks cluster.
When you query Delta Lake data, the FEs and BEs or CNs of your StarRocks cluster use the HDFS client to access your HDFS cluster. In most cases, you do not need to configure your StarRocks cluster to achieve that purpose, and StarRocks starts the HDFS client using the default configurations. You need to configure your StarRocks cluster only in the following situations:
- High availability (HA) is enabled for your HDFS cluster: Add the hdfs-site.xml file of your HDFS cluster to the $FE_HOME/conf path of each FE and to the $BE_HOME/conf path of each BE or the $CN_HOME/conf path of each CN.
- View File System (ViewFs) is enabled for your HDFS cluster: Add the core-site.xml file of your HDFS cluster to the $FE_HOME/conf path of each FE and to the $BE_HOME/conf path of each BE or the $CN_HOME/conf path of each CN.

NOTE

If an error indicating an unknown host is returned when you send a query, you must add the mapping between the host names and IP addresses of your HDFS cluster nodes to the /etc/hosts path.

Kerberos authentication

If Kerberos authentication is enabled for your HDFS cluster or Hive metastore, configure your StarRocks cluster as follows:

Run the kinit -kt keytab_path principal command on each FE and each BE or CN to obtain Ticket Granting Ticket (TGT) from Key Distribution Center (KDC). To run this command, you must have the permissions to access your HDFS cluster and Hive metastore. Note that accessing KDC with this command is time-sensitive. Therefore, you need to use cron to run this command periodically.
Add JAVA_OPTS="-Djava.security.krb5.conf=/etc/krb5.conf" to the $FE_HOME/conf/fe.conf file of each FE and to the $BE_HOME/conf/be.conf file of each BE or the $CN_HOME/conf/cn.conf file of each CN. In this example, /etc/krb5.conf is the save path of the krb5.conf file. You can modify the path based on your needs.

Create a Delta Lake catalog

Syntax

CREATE EXTERNAL CATALOG <catalog_name>
[COMMENT <comment>]
PROPERTIES
(
    "type" = "deltalake",
    MetastoreParams,
    StorageCredentialParams,
    MetadataUpdateParams
)

Parameters

catalog_name

The name of the Delta Lake catalog. The naming conventions are as follows:

The name can contain letters, digits (0-9), and underscores (_). It must start with a letter.
The name is case-sensitive and cannot exceed 1023 characters in length.

comment

The description of the Delta Lake catalog. This parameter is optional.

type

The type of your data source. Set the value to deltalake.

MetastoreParams

A set of parameters about how StarRocks integrates with the metastore of your data source.

Hive metastore

If you choose Hive metastore as the metastore of your data source, configure MetastoreParams as follows:

"hive.metastore.type" = "hive",
"hive.metastore.uris" = "<hive_metastore_uri>"

NOTE

Before querying Delta Lake data, you must add the mapping between the host names and IP addresses of your Hive metastore nodes to the /etc/hosts path. Otherwise, StarRocks may fail to access your Hive metastore when you start a query.

The following table describes the parameter you need to configure in MetastoreParams.

Parameter	Required	Description
hive.metastore.type	Yes	The type of metastore that you use for your Delta Lake cluster. Set the value to `hive`.
hive.metastore.uris	Yes	The URI of your Hive metastore. Format: `thrift://<metastore_IP_address>:<metastore_port>`. If high availability (HA) is enabled for your Hive metastore, you can specify multiple metastore URIs and separate them with commas (`,`), for example, `"thrift://<metastore_IP_address_1>:<metastore_port_1>,thrift://<metastore_IP_address_2>:<metastore_port_2>,thrift://<metastore_IP_address_3>:<metastore_port_3>"`.

AWS Glue

If you choose AWS Glue as the metastore of your data source, which is supported only when you choose AWS S3 as storage, take one of the following actions:

To choose the instance profile-based authentication method, configure MetastoreParams as follows:

"hive.metastore.type" = "glue",
"aws.glue.use_instance_profile" = "true",
"aws.glue.region" = "<aws_glue_region>"

To choose the assumed role-based authentication method, configure MetastoreParams as follows:

"hive.metastore.type" = "glue",
"aws.glue.use_instance_profile" = "true",
"aws.glue.iam_role_arn" = "<iam_role_arn>",
"aws.glue.region" = "<aws_glue_region>"

To choose the IAM user-based authentication method, configure MetastoreParams as follows:

"hive.metastore.type" = "glue",
"aws.glue.use_instance_profile" = "false",
"aws.glue.access_key" = "<iam_user_access_key>",
"aws.glue.secret_key" = "<iam_user_secret_key>",
"aws.glue.region" = "<aws_s3_region>"

The following table describes the parameters you need to configure in MetastoreParams.

Parameter	Required	Description
hive.metastore.type	Yes	The type of metastore that you use for your Delta Lake cluster. Set the value to `glue`.
aws.glue.use_instance_profile	Yes	Specifies whether to enable the instance profile-based authentication method and the assumed role-based authentication method. Valid values: `true` and `false`. Default value: `false`.
aws.glue.iam_role_arn	No	The ARN of the IAM role that has privileges on your AWS Glue Data Catalog. If you use the assumed role-based authentication method to access AWS Glue, you must specify this parameter.
aws.glue.region	Yes	The region in which your AWS Glue Data Catalog resides. Example: `us-west-1`.
aws.glue.access_key	No	The access key of your AWS IAM user. If you use the IAM user-based authentication method to access AWS Glue, you must specify this parameter.
aws.glue.secret_key	No	The secret key of your AWS IAM user. If you use the IAM user-based authentication method to access AWS Glue, you must specify this parameter.

For information about how to choose an authentication method for accessing AWS Glue and how to configure an access control policy in the AWS IAM Console, see Authentication parameters for accessing AWS Glue.

StorageCredentialParams

A set of parameters about how StarRocks integrates with your storage system. This parameter set is optional.

If you use HDFS as storage, you do not need to configure StorageCredentialParams.

If you use AWS S3, other S3-compatible storage system, Microsoft Azure Storage, or Google GCS as storage, you must configure StorageCredentialParams.

AWS S3

If you choose AWS S3 as storage for your Delta Lake cluster, take one of the following actions:

To choose the instance profile-based authentication method, configure StorageCredentialParams as follows:
```
"aws.s3.use_instance_profile" = "true",
"aws.s3.region" = "<aws_s3_region>"
```

To choose the assumed role-based authentication method, configure StorageCredentialParams as follows:

"aws.s3.use_instance_profile" = "true",
"aws.s3.iam_role_arn" = "<iam_role_arn>",
"aws.s3.region" = "<aws_s3_region>"

To choose the IAM user-based authentication method, configure StorageCredentialParams as follows:

"aws.s3.use_instance_profile" = "false",
"aws.s3.access_key" = "<iam_user_access_key>",
"aws.s3.secret_key" = "<iam_user_secret_key>",
"aws.s3.region" = "<aws_s3_region>"

The following table describes the parameters you need to configure in StorageCredentialParams.

Parameter	Required	Description
aws.s3.use_instance_profile	Yes	Specifies whether to enable the instance profile-based authentication method and the assumed role-based authentication method. Valid values: `true` and `false`. Default value: `false`.
aws.s3.iam_role_arn	No	The ARN of the IAM role that has privileges on your AWS S3 bucket. If you use the assumed role-based authentication method to access AWS S3, you must specify this parameter.
aws.s3.region	Yes	The region in which your AWS S3 bucket resides. Example: `us-west-1`.
aws.s3.access_key	No	The access key of your IAM user. If you use the IAM user-based authentication method to access AWS S3, you must specify this parameter.
aws.s3.secret_key	No	The secret key of your IAM user. If you use the IAM user-based authentication method to access AWS S3, you must specify this parameter.

For information about how to choose an authentication method for accessing AWS S3 and how to configure an access control policy in AWS IAM Console, see Authentication parameters for accessing AWS S3.

S3-compatible storage system

Delta Lake catalogs support S3-compatible storage systems from v2.5 onwards.

If you choose an S3-compatible storage system, such as MinIO, as storage for your Delta Lake cluster, configure StorageCredentialParams as follows to ensure a successful integration:

"aws.s3.enable_ssl" = "false",
"aws.s3.enable_path_style_access" = "true",
"aws.s3.endpoint" = "<s3_endpoint>",
"aws.s3.access_key" = "<iam_user_access_key>",
"aws.s3.secret_key" = "<iam_user_secret_key>"

The following table describes the parameters you need to configure in StorageCredentialParams.

Parameter	Required	Description
aws.s3.enable_ssl	Yes	Specifies whether to enable SSL connection. Valid values: `true` and `false`. Default value: `true`.
aws.s3.enable_path_style_access	Yes	Specifies whether to enable path-style access. Valid values: `true` and `false`. Default value: `false`. For MinIO, you must set the value to `true`. Path-style URLs use the following format: `https://s3.<region_code>.amazonaws.com/<bucket_name>/<key_name>`. For example, if you create a bucket named `DOC-EXAMPLE-BUCKET1` in the US West (Oregon) Region, and you want to access the `alice.jpg` object in that bucket, you can use the following path-style URL: `https://s3.us-west-2.amazonaws.com/DOC-EXAMPLE-BUCKET1/alice.jpg`.
aws.s3.endpoint	Yes	The endpoint that is used to connect to your S3-compatible storage system instead of AWS S3.
aws.s3.access_key	Yes	The access key of your IAM user.
aws.s3.secret_key	Yes	The secret key of your IAM user.

Microsoft Azure Storage

Delta Lake catalogs support Microsoft Azure Storage from v3.0 onwards.

Azure Blob Storage

If you choose Blob Storage as storage for your Delta Lake cluster, take one of the following actions:

To choose the Shared Key authentication method, configure StorageCredentialParams as follows:
```
"azure.blob.storage_account" = "<storage_account_name>",
"azure.blob.shared_key" = "<storage_account_shared_key>"
```
The following table describes the parameters you need to configure in StorageCredentialParams.

Parameter Required Description
azure.blob.storage_account Yes The username of your Blob Storage account.
azure.blob.shared_key Yes The shared key of your Blob Storage account.

To choose the SAS Token authentication method, configure StorageCredentialParams as follows:

"azure.blob.storage_account" = "<storage_account_name>",
"azure.blob.container" = "<container_name>",
"azure.blob.sas_token" = "<storage_account_SAS_token>"

The following table describes the parameters you need to configure in StorageCredentialParams.

Parameter	Required	Description
azure.blob.storage_account	Yes	The username of your Blob Storage account.
azure.blob.container	Yes	The name of the blob container that stores your data.
azure.blob.sas_token	Yes	The SAS token that is used to access your Blob Storage account.

Azure Data Lake Storage Gen2

If you choose Data Lake Storage Gen2 as storage for your Delta Lake cluster, take one of the following actions:

To choose the Managed Identity authentication method, configure StorageCredentialParams as follows:

"azure.adls2.oauth2_use_managed_identity" = "true",
"azure.adls2.oauth2_tenant_id" = "<service_principal_tenant_id>",
"azure.adls2.oauth2_client_id" = "<service_client_id>"

The following table describes the parameters you need to configure in StorageCredentialParams.

Parameter	Required	Description
azure.adls2.oauth2_use_managed_identity	Yes	Specifies whether to enable the Managed Identity authentication method. Set the value to `true`.
azure.adls2.oauth2_tenant_id	Yes	The ID of the tenant whose data you want to access.
azure.adls2.oauth2_client_id	Yes	The client (application) ID of the managed identity.

To choose the Shared Key authentication method, configure StorageCredentialParams as follows:

"azure.adls2.storage_account" = "<storage_account_name>",
"azure.adls2.shared_key" = "<storage_account_shared_key>"

The following table describes the parameters you need to configure in StorageCredentialParams.

Parameter	Required	Description
azure.adls2.storage_account	Yes	The username of your Data Lake Storage Gen2 storage account.
azure.adls2.shared_key	Yes	The shared key of your Data Lake Storage Gen2 storage account.

To choose the Service Principal authentication method, configure StorageCredentialParams as follows:

"azure.adls2.oauth2_client_id" = "<service_client_id>",
"azure.adls2.oauth2_client_secret" = "<service_principal_client_secret>",
"azure.adls2.oauth2_client_endpoint" = "<service_principal_client_endpoint>"

The following table describes the parameters you need to configure in StorageCredentialParams.

Parameter	Required	Description
azure.adls2.oauth2_client_id	Yes	The client (application) ID of the service principal.
azure.adls2.oauth2_client_secret	Yes	The value of the new client (application) secret created.
azure.adls2.oauth2_client_endpoint	Yes	The OAuth 2.0 token endpoint (v1) of the service principal or application.

Azure Data Lake Storage Gen1

If you choose Data Lake Storage Gen1 as storage for your Delta Lake cluster, take one of the following actions:

To choose the Managed Service Identity authentication method, configure StorageCredentialParams as follows:
```
"azure.adls1.use_managed_service_identity" = "true"
```
The following table describes the parameters you need to configure in StorageCredentialParams.

Parameter Required Description
azure.adls1.use_managed_service_identity Yes Specifies whether to enable the Managed Service Identity authentication method. Set the value to true.

Parameter	Required	Description
azure.adls1.use_managed_service_identity	Yes	Specifies whether to enable the Managed Service Identity authentication method. Set the value to `true`.

To choose the Service Principal authentication method, configure StorageCredentialParams as follows:

"azure.adls1.oauth2_client_id" = "<application_client_id>",
"azure.adls1.oauth2_credential" = "<application_client_credential>",
"azure.adls1.oauth2_endpoint" = "<OAuth_2.0_authorization_endpoint_v2>"

The following table describes the parameters you need to configure in StorageCredentialParams.

Parameter	Required	Description
azure.adls1.oauth2_client_id	Yes	The client (application) ID of the service principal.
azure.adls1.oauth2_credential	Yes	The value of the new client (application) secret created.
azure.adls1.oauth2_endpoint	Yes	The OAuth 2.0 token endpoint (v1) of the service principal or application.

Google GCS

Delta Lake catalogs support Google GCS from v3.0 onwards.

If you choose Google GCS as storage for your Delta Lake cluster, take one of the following actions:

To choose the VM-based authentication method, configure StorageCredentialParams as follows:
```
"gcp.gcs.use_compute_engine_service_account" = "true"
```
The following table describes the parameters you need to configure in StorageCredentialParams.

Parameter Default value Value example Description
gcp.gcs.use_compute_engine_service_account false true Specifies whether to directly use the service account that is bound to your Compute Engine.

To choose the service account-based authentication method, configure StorageCredentialParams as follows:

"gcp.gcs.service_account_email" = "<google_service_account_email>",
"gcp.gcs.service_account_private_key_id" = "<google_service_private_key_id>",
"gcp.gcs.service_account_private_key" = "<google_service_private_key>",

The following table describes the parameters you need to configure in StorageCredentialParams.

Parameter	Default value	Value example	Description
gcp.gcs.service_account_email	""	"user@hello.iam.gserviceaccount.com"	The email address in the JSON file generated at the creation of the service account.
gcp.gcs.service_account_private_key_id	""	"61d257bd8479547cb3e04f0b9b6b9ca07af3b7ea"	The private key ID in the JSON file generated at the creation of the service account.
gcp.gcs.service_account_private_key	""	"-----BEGIN PRIVATE KEY----xxxx-----END PRIVATE KEY-----\n"	The private key in the JSON file generated at the creation of the service account.

To choose the impersonation-based authentication method, configure StorageCredentialParams as follows:

Make a VM instance impersonate a service account:

"gcp.gcs.use_compute_engine_service_account" = "true",
"gcp.gcs.impersonation_service_account" = "<assumed_google_service_account_email>"

The following table describes the parameters you need to configure in StorageCredentialParams.

Parameter	Default value	Value example	Description
gcp.gcs.use_compute_engine_service_account	false	true	Specifies whether to directly use the service account that is bound to your Compute Engine.
gcp.gcs.impersonation_service_account	""	"hello"	The service account that you want to impersonate.

Make a service account (temporarily named as meta service account) impersonate another service account (temporarily named as data service account):

"gcp.gcs.service_account_email" = "<google_service_account_email>",
"gcp.gcs.service_account_private_key_id" = "<meta_google_service_account_email>",
"gcp.gcs.service_account_private_key" = "<meta_google_service_account_email>",
"gcp.gcs.impersonation_service_account" = "<data_google_service_account_email>"

The following table describes the parameters you need to configure in StorageCredentialParams.

Parameter	Default value	Value example	Description
gcp.gcs.service_account_email	""	"user@hello.iam.gserviceaccount.com"	The email address in the JSON file generated at the creation of the meta service account.
gcp.gcs.service_account_private_key_id	""	"61d257bd8479547cb3e04f0b9b6b9ca07af3b7ea"	The private key ID in the JSON file generated at the creation of the meta service account.
gcp.gcs.service_account_private_key	""	"-----BEGIN PRIVATE KEY----xxxx-----END PRIVATE KEY-----\n"	The private key in the JSON file generated at the creation of the meta service account.
gcp.gcs.impersonation_service_account	""	"hello"	The data service account that you want to impersonate.

MetadataUpdateParams

A set of parameters about how StarRocks updates the cached metadata of Delta Lake. This parameter set is optional.

From v3.3.3 onwards, Delta Lake Catalog supports Metadata Local Cache and Retrieval. In most cases, you can ignore MetadataUpdateParams and do not need to tune the policy parameters in it, because the default values of these parameters already provide you with an out-of-the-box performance.

However, if the frequency of data updates in Delta Lake is high, you can tune these parameters to further optimize the performance of automatic asynchronous updates.

NOTE

In most cases, if your Delta Lake data is updated at a granularity of 1 hour or less, the data update frequency is considered high.

| Parameter | Unit | Default | Description | |----------------------------------------------------| -------- | ------------------------------------------------------------ | | enable_deltalake_table_cache | - | true | Whether to enable Table Cache in the metadata cache for Delta Lake. | | enable_deltalake_json_meta_cache | - | true | Whether to enable cache for Delta Log JSON files. | | deltalake_json_meta_cache_ttl_sec | Second | 48 * 60 * 60 | Time-To-Live (TTL) for the Delta Log JSON file cache. | | deltalake_json_meta_cache_memory_usage_ratio | - | 0.1 | The maximum ratio of JVM Heap size occupied by the Delta Log JSON file cache. | | enable_deltalake_checkpoint_meta_cache | - | true | Whether to enable cache for Delta Log Checkpoint files. | | deltalake_checkpoint_meta_cache_ttl_sec | Second | 48 * 60 * 60 | Time-To-Live (TTL) for the Delta Log Checkpoint file cache. | | deltalake_checkpoint_meta_cache_memory_usage_ratio | - | 0.1 | The maximum ratio of JVM Heap size occupied by the Delta Log Checkpoint file cache. |

Examples

The following examples create a Delta Lake catalog named deltalake_catalog_hms or deltalake_catalog_glue, depending on the type of metastore you use, to query data from your Delta Lake cluster.

HDFS

If you use HDFS as storage, run a command like below:

CREATE EXTERNAL CATALOG deltalake_catalog_hms
PROPERTIES
(
    "type" = "deltalake",
    "hive.metastore.type" = "hive",
    "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083"
);

AWS S3

If you choose instance profile-based credential

If you use Hive metastore in your Delta Lake cluster, run a command like below:

CREATE EXTERNAL CATALOG deltalake_catalog_hms
PROPERTIES
(
    "type" = "deltalake",
    "hive.metastore.type" = "hive",
    "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083",
    "aws.s3.use_instance_profile" = "true",
    "aws.s3.region" = "us-west-2"
);

If you use AWS Glue in your Amazon EMR Delta Lake cluster, run a command like below:

CREATE EXTERNAL CATALOG deltalake_catalog_glue
PROPERTIES
(
    "type" = "deltalake",
    "hive.metastore.type" = "glue",
    "aws.glue.use_instance_profile" = "true",
    "aws.glue.region" = "us-west-2",
    "aws.s3.use_instance_profile" = "true",
    "aws.s3.region" = "us-west-2"
);

If you choose assumed role-based credential

If you use Hive metastore in your Delta Lake cluster, run a command like below:

CREATE EXTERNAL CATALOG deltalake_catalog_hms
PROPERTIES
(
    "type" = "deltalake",
    "hive.metastore.type" = "hive",
    "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083",
    "aws.s3.use_instance_profile" = "true",
    "aws.s3.iam_role_arn" = "arn:aws:iam::081976408565:role/test_s3_role",
    "aws.s3.region" = "us-west-2"
);

If you use AWS Glue in your Amazon EMR Delta Lake cluster, run a command like below:

CREATE EXTERNAL CATALOG deltalake_catalog_glue
PROPERTIES
(
    "type" = "deltalake",
    "hive.metastore.type" = "glue",
    "aws.glue.use_instance_profile" = "true",
    "aws.glue.iam_role_arn" = "arn:aws:iam::081976408565:role/test_glue_role",
    "aws.glue.region" = "us-west-2",
    "aws.s3.use_instance_profile" = "true",
    "aws.s3.iam_role_arn" = "arn:aws:iam::081976408565:role/test_s3_role",
    "aws.s3.region" = "us-west-2"
);

If you choose IAM user-based credential

If you use Hive metastore in your Delta Lake cluster, run a command like below:

CREATE EXTERNAL CATALOG deltalake_catalog_hms
PROPERTIES
(
    "type" = "deltalake",
    "hive.metastore.type" = "hive",
    "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083",
    "aws.s3.use_instance_profile" = "false",
    "aws.s3.access_key" = "<iam_user_access_key>",
    "aws.s3.secret_key" = "<iam_user_access_key>",
    "aws.s3.region" = "us-west-2"
);

If you use AWS Glue in your Amazon EMR Delta Lake cluster, run a command like below:

CREATE EXTERNAL CATALOG deltalake_catalog_glue
PROPERTIES
(
    "type" = "deltalake",
    "hive.metastore.type" = "glue",
    "aws.glue.use_instance_profile" = "false",
    "aws.glue.access_key" = "<iam_user_access_key>",
    "aws.glue.secret_key" = "<iam_user_secret_key>",
    "aws.glue.region" = "us-west-2",
    "aws.s3.use_instance_profile" = "false",
    "aws.s3.access_key" = "<iam_user_access_key>",
    "aws.s3.secret_key" = "<iam_user_secret_key>",
    "aws.s3.region" = "us-west-2"
);

S3-compatible storage system

Use MinIO as an example. Run a command like below:

CREATE EXTERNAL CATALOG deltalake_catalog_hms
PROPERTIES
(
    "type" = "deltalake",
    "hive.metastore.type" = "hive",
    "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083",
    "aws.s3.enable_ssl" = "true",
    "aws.s3.enable_path_style_access" = "true",
    "aws.s3.endpoint" = "<s3_endpoint>",
    "aws.s3.access_key" = "<iam_user_access_key>",
    "aws.s3.secret_key" = "<iam_user_secret_key>"
);

Microsoft Azure Storage

Azure Blob Storage

If you choose the Shared Key authentication method, run a command like below:

CREATE EXTERNAL CATALOG deltalake_catalog_hms
PROPERTIES
(
    "type" = "deltalake",
    "hive.metastore.type" = "hive",
    "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083",
    "azure.blob.storage_account" = "<blob_storage_account_name>",
    "azure.blob.shared_key" = "<blob_storage_account_shared_key>"
);

If you choose the SAS Token authentication method, run a command like below:

CREATE EXTERNAL CATALOG deltalake_catalog_hms
PROPERTIES
(
    "type" = "deltalake",
    "hive.metastore.type" = "hive",
    "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083",
    "azure.blob.storage_account" = "<blob_storage_account_name>",
    "azure.blob.container" = "<blob_container_name>",
    "azure.blob.sas_token" = "<blob_storage_account_SAS_token>"
);

Azure Data Lake Storage Gen1

If you choose the Managed Service Identity authentication method, run a command like below:

CREATE EXTERNAL CATALOG deltalake_catalog_hms
PROPERTIES
(
    "type" = "deltalake",
    "hive.metastore.type" = "hive",
    "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083",
    "azure.adls1.use_managed_service_identity" = "true"    
);

If you choose the Service Principal authentication method, run a command like below:

CREATE EXTERNAL CATALOG deltalake_catalog_hms
PROPERTIES
(
    "type" = "deltalake",
    "hive.metastore.type" = "hive",
    "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083",
    "azure.adls1.oauth2_client_id" = "<application_client_id>",
    "azure.adls1.oauth2_credential" = "<application_client_credential>",
    "azure.adls1.oauth2_endpoint" = "<OAuth_2.0_authorization_endpoint_v2>"
);

Azure Data Lake Storage Gen2

If you choose the Managed Identity authentication method, run a command like below:

CREATE EXTERNAL CATALOG deltalake_catalog_hms
PROPERTIES
(
    "type" = "deltalake",
    "hive.metastore.type" = "hive",
    "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083",
    "azure.adls2.oauth2_use_managed_identity" = "true",
    "azure.adls2.oauth2_tenant_id" = "<service_principal_tenant_id>",
    "azure.adls2.oauth2_client_id" = "<service_client_id>"
);

If you choose the Shared Key authentication method, run a command like below:

CREATE EXTERNAL CATALOG deltalake_catalog_hms
PROPERTIES
(
    "type" = "deltalake",
    "hive.metastore.type" = "hive",
    "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083",
    "azure.adls2.storage_account" = "<storage_account_name>",
    "azure.adls2.shared_key" = "<shared_key>"     
);

If you choose the Service Principal authentication method, run a command like below:

CREATE EXTERNAL CATALOG deltalake_catalog_hms
PROPERTIES
(
    "type" = "deltalake",
    "hive.metastore.type" = "hive",
    "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083",
    "azure.adls2.oauth2_client_id" = "<service_client_id>",
    "azure.adls2.oauth2_client_secret" = "<service_principal_client_secret>",
    "azure.adls2.oauth2_client_endpoint" = "<service_principal_client_endpoint>"
);

Google GCS

If you choose the VM-based authentication method, run a command like below:

CREATE EXTERNAL CATALOG deltalake_catalog_hms
PROPERTIES
(
    "type" = "deltalake",
    "hive.metastore.type" = "hive",
    "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083",
    "gcp.gcs.use_compute_engine_service_account" = "true"    
);

If you choose the service account-based authentication method, run a command like below:

CREATE EXTERNAL CATALOG deltalake_catalog_hms
PROPERTIES
(
    "type" = "deltalake",
    "hive.metastore.type" = "hive",
    "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083",
    "gcp.gcs.service_account_email" = "<google_service_account_email>",
    "gcp.gcs.service_account_private_key_id" = "<google_service_private_key_id>",
    "gcp.gcs.service_account_private_key" = "<google_service_private_key>"    
);

If you choose the impersonation-based authentication method:

If you make a VM instance impersonate a service account, run a command like below:

CREATE EXTERNAL CATALOG deltalake_catalog_hms
PROPERTIES
(
    "type" = "deltalake",
    "hive.metastore.type" = "hive",
    "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083",
    "gcp.gcs.use_compute_engine_service_account" = "true",
    "gcp.gcs.impersonation_service_account" = "<assumed_google_service_account_email>"    
);

If you make a service account impersonate another service account, run a command like below:

CREATE EXTERNAL CATALOG deltalake_catalog_hms
PROPERTIES
(
    "type" = "deltalake",
    "hive.metastore.type" = "hive",
    "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083",
    "gcp.gcs.service_account_email" = "<google_service_account_email>",
    "gcp.gcs.service_account_private_key_id" = "<meta_google_service_account_email>",
    "gcp.gcs.service_account_private_key" = "<meta_google_service_account_email>",
    "gcp.gcs.impersonation_service_account" = "<data_google_service_account_email>"    
);

View Delta Lake catalogs

You can use SHOW CATALOGS to query all catalogs in the current StarRocks cluster:

SHOW CATALOGS;

You can also use SHOW CREATE CATALOG to query the creation statement of an external catalog. The following example queries the creation statement of a Delta Lake catalog named deltalake_catalog_glue:

SHOW CREATE CATALOG deltalake_catalog_glue;

Switch to a Delta Lake Catalog and a database in it

You can use one of the following methods to switch to a Delta Lake catalog and a database in it:

Use SET CATALOG to specify a Delta Lake catalog in the current session, and then use USE to specify an active database:

-- Switch to a specified catalog in the current session:
SET CATALOG <catalog_name>
-- Specify the active database in the current session:
USE <db_name>

Directly use USE to switch to a Delta Lake catalog and a database in it:
```
USE <catalog_name>.<db_name>
```

Drop a Delta Lake catalog

You can use DROP CATALOG to drop an external catalog.

The following example drops a Delta Lake catalog named deltalake_catalog_glue:

DROP Catalog deltalake_catalog_glue;

View the schema of a Delta Lake table

You can use one of the following syntaxes to view the schema of a Delta Lake table:

View schema

DESC[RIBE] <catalog_name>.<database_name>.<table_name>

View schema and location from the CREATE statement

SHOW CREATE TABLE <catalog_name>.<database_name>.<table_name>

Query a Delta Lake table

Use SHOW DATABASES to view the databases in your Delta Lake cluster:
```
SHOW DATABASES FROM <catalog_name>
```
Switch to a Delta Lake Catalog and a database in it.
Use SELECT to query the destination table in the specified database:
```
SELECT count(*) FROM <table_name> LIMIT 10
```

Load data from Delta Lake

Suppose you have an OLAP table named olap_tbl, you can transform and load data like below:

INSERT INTO default_catalog.olap_db.olap_tbl SELECT * FROM deltalake_table

Configure metadata cache and update strategy

From v3.3.3 onwards, Delta Lake Catalog supports Metadata Local Cache and Retrieval.

You can configure the Delta Lake metadata cache refresh through the following FE parameters:

Configuration item	Default	Description
enable_background_refresh_connector_metadata	`true`	Whether to enable the periodic Delta Lake metadata cache refresh. After it is enabled, StarRocks polls the metastore (Hive Metastore or AWS Glue) of your Delta Lake cluster, and refreshes the cached metadata of the frequently accessed Delta Lake catalogs to perceive data changes. `true` indicates to enable the Delta Lake metadata cache refresh, and `false` indicates to disable it.
background_refresh_metadata_interval_millis	`600000`	The interval between two consecutive Delta Lake metadata cache refreshes. Unit: millisecond.
background_refresh_metadata_time_secs_since_last_access_secs	`86400`	The expiration time of a Delta Lake metadata cache refresh task. For the Delta Lake catalog that has been accessed, if it has not been accessed for more than the specified time, StarRocks stops refreshing its cached metadata. For the Delta Lake catalog that has not been accessed, StarRocks will not refresh its cached metadata. Unit: second.

Appendix: Metadata Local Cache and Retrieval

Because the repeated decompression and parsing of metadata files can introduce unnecessary delays, StarRocks employs a new metadata cache strategy of caching deserialized memory objects. By storing these deserialized files in memory, the system can bypass the decompression and parsing stages for subsequent queries. This caching mechanism allows direct access to the required metadata, significantly reducing retrieval times. As a result, the system becomes more responsive and better suited to meet high query demands and materialized view rewriting needs.

You can configure this behavior through the Catalog property MetadataUpdateParams and relevant configuration items.

Parameter	Required	Description
azure.blob.storage_account	Yes	The username of your Blob Storage account.
azure.blob.shared_key	Yes	The shared key of your Blob Storage account.

Delta Lake catalog

Usage notes​

Integration preparations​

AWS IAM​

HDFS​

Kerberos authentication​

Create a Delta Lake catalog​

Syntax​

Parameters​

catalog_name​

comment​

type​

MetastoreParams​

Hive metastore​

AWS Glue​

StorageCredentialParams​

AWS S3​

S3-compatible storage system​

Microsoft Azure Storage​

Azure Blob Storage​

Azure Data Lake Storage Gen2​

Azure Data Lake Storage Gen1​

Google GCS​

MetadataUpdateParams​

Examples​

HDFS​

AWS S3​

If you choose instance profile-based credential​

If you choose assumed role-based credential​

If you choose IAM user-based credential​

S3-compatible storage system​

Microsoft Azure Storage​

Azure Blob Storage​

Azure Data Lake Storage Gen1​

Azure Data Lake Storage Gen2​

Google GCS​

View Delta Lake catalogs​

Switch to a Delta Lake Catalog and a database in it​

Drop a Delta Lake catalog​

View the schema of a Delta Lake table​

Query a Delta Lake table​

Load data from Delta Lake​

Configure metadata cache and update strategy​

Appendix: Metadata Local Cache and Retrieval​

What did you think of this doc?

Usage notes

Integration preparations

AWS IAM

HDFS

Kerberos authentication

Create a Delta Lake catalog

Syntax

Parameters

catalog_name

comment

type

MetastoreParams

Hive metastore

AWS Glue

StorageCredentialParams

AWS S3

S3-compatible storage system

Microsoft Azure Storage

Azure Blob Storage

Azure Data Lake Storage Gen2

Azure Data Lake Storage Gen1

Google GCS

MetadataUpdateParams

Examples

HDFS

AWS S3

If you choose instance profile-based credential

If you choose assumed role-based credential

If you choose IAM user-based credential

S3-compatible storage system

Microsoft Azure Storage

Azure Blob Storage

Azure Data Lake Storage Gen1

Azure Data Lake Storage Gen2

Google GCS

View Delta Lake catalogs

Switch to a Delta Lake Catalog and a database in it

Drop a Delta Lake catalog

View the schema of a Delta Lake table

Query a Delta Lake table

Load data from Delta Lake

Configure metadata cache and update strategy

Appendix: Metadata Local Cache and Retrieval