Version: Latest-3.3

query_dump interface

This topic describes how to use the query_dump interface to obtain the details of an SQL query and its related information.

If you encounter any of the following issues when executing SQL queries with StarRocks, you can use query_dump to obtain the SQL details and send the information to StarRocks technical support for troubleshooting:

Unknown Error is returned when you execute an SQL query or EXPLAIN.
An error message or exception is returned when you execute an SQL query.
Executing an SQL query is not as efficient as expected, or the execution plan can be optimized (for example, partitions can be pruned or Join order can be adjusted).

Function overview

The query_dump interface returns the information that FE relies on when executing the SQL, including:

Query statement
Table creation statement
Session variables
Number of BEs
Statistics information (Min, Max values in a column)
Exception information (abnormal stack)
Explain costs info

To ensure data privacy, we desensitize the meta information such as database names, table names, and column names. We also use the desensitized metadata to rewrite the query statements.

Meta information desensitization is enabled by default. If an exception occurs during the desensitization process, the original information is used. If desensitization needs to be bypassed, you can add "mock=false" in the HTTP URI.

Syntax

HTTP Post

 fe_host:fe_http_port/api/query_dump?db=${database}&mock=${value} post_data=${Query}

wget --user=${username} --password=${password} --post-file ${query_file} "http://${fe_host}:${fe_http_port}/api/query_dump?db=${database}&mock={value}" -O ${dump_file}

Parameter description:

query_file: the file containing the query
dump_file: the output file
db: the database where the SQL query is executed. The db parameter is optional if the query includes use db. Otherwise, it must be specified.
mock: whether to enable or disable desensitization

Examples

Disable desensitization

Command:

wget --user=root --password=123 --post-file query_file "http://127.0.0.1:8030/api/query_dump?db=tpch&mock=false" -O dump_file

Return data:

Data is returned in JSON format.

{
  "statement": "select\n    l_returnflag,\n    l_linestatus,\n    sum(l_quantity) as sum_qty,\n    sum(l_extendedprice) as sum_base_price,\n    sum(l_extendedprice * (1 - l_discount)) as sum_disc_price,\n    sum(l_extendedprice * (1 - l_discount) * (1 + l_tax)) as sum_charge,\n    avg(l_quantity) as avg_qty,\n    avg(l_extendedprice) as avg_price,\n    avg(l_discount) as avg_disc,\n    count(*) as count_order\nfrom\n    lineitem\nwhere\n    l_shipdate <= date '1998-12-01'\ngroup by\n    l_returnflag,\n    l_linestatus\norder by\n    l_returnflag,\n    l_linestatus ;\n\n",
  "table_meta": {
    "tpch.lineitem": "CREATE TABLE `lineitem` (\n  `L_ORDERKEY` int(11) NOT NULL COMMENT \"\",\n  `L_PARTKEY` int(11) NOT NULL COMMENT \"\",\n  `L_SUPPKEY` int(11) NOT NULL COMMENT \"\",\n  `L_LINENUMBER` int(11) NOT NULL COMMENT \"\",\n  `L_QUANTITY` double NOT NULL COMMENT \"\",\n  `L_EXTENDEDPRICE` double NOT NULL COMMENT \"\",\n  `L_DISCOUNT` double NOT NULL COMMENT \"\",\n  `L_TAX` double NOT NULL COMMENT \"\",\n  `L_RETURNFLAG` char(1) NOT NULL COMMENT \"\",\n  `L_LINESTATUS` char(1) NOT NULL COMMENT \"\",\n  `L_SHIPDATE` date NOT NULL COMMENT \"\",\n  `L_COMMITDATE` date NOT NULL COMMENT \"\",\n  `L_RECEIPTDATE` date NOT NULL COMMENT \"\",\n  `L_SHIPINSTRUCT` char(25) NOT NULL COMMENT \"\",\n  `L_SHIPMODE` char(10) NOT NULL COMMENT \"\",\n  `L_COMMENT` varchar(44) NOT NULL COMMENT \"\",\n  `PAD` char(1) NOT NULL COMMENT \"\"\n) ENGINE=OLAP \nDUPLICATE KEY(`L_ORDERKEY`)\nCOMMENT \"OLAP\"\nDISTRIBUTED BY HASH(`L_ORDERKEY`) BUCKETS 20 \nPROPERTIES (\n\"replication_num\" = \"1\",\n\"in_memory\" = \"false\",\n\"enable_persistent_index\" = \"false\",\n\"replicated_storage\" = \"true\",\n\"compression\" = \"LZ4\"\n);"
  },
  "table_row_count": {
    "tpch.lineitem": {
      "lineitem": 3
    }
  },
  "column_statistics": {
    "tpch.lineitem": {
      "L_TAX": "[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE",
      "L_SHIPDATE": "[1.6094304E9, 1.6094304E9, 0.0, 4.0, 1.0] ESTIMATE",
      "L_EXTENDEDPRICE": "[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE",
      "L_DISCOUNT": "[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE",
      "L_RETURNFLAG": "[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE",
      "L_LINESTATUS": "[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE",
      "L_QUANTITY": "[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE"
    }
  },
  "explain_info": "PLAN FRAGMENT 0(F02)\n  Output Exprs:9: L_RETURNFLAG | 10: L_LINESTATUS | 20: sum | 21: sum | 22: sum | 23: sum | 24: avg | 25: avg | 26: avg | 27: count\n  Input Partition: UNPARTITIONED\n  RESULT SINK\n\n  6:MERGING-EXCHANGE\n     distribution type: GATHER\n     cardinality: 1\n     column statistics: \n     * L_RETURNFLAG-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n     * L_LINESTATUS-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n     * sum-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n     * sum-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n     * sum-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n     * sum-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n     * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n     * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n     * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n     * count-->[0.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n\nPLAN FRAGMENT 1(F01)\n\n  Input Partition: HASH_PARTITIONED: 9: L_RETURNFLAG, 10: L_LINESTATUS\n  OutPut Partition: UNPARTITIONED\n  OutPut Exchange Id: 06\n\n  5:SORT\n  |  order by: [9, VARCHAR, false] ASC, [10, VARCHAR, false] ASC\n  |  offset: 0\n  |  cardinality: 1\n  |  column statistics: \n  |  * L_RETURNFLAG-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n  |  * L_LINESTATUS-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n  |  * sum-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n  |  * sum-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n  |  * sum-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n  |  * sum-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n  |  * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n  |  * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n  |  * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n  |  * count-->[0.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n  |  \n  4:AGGREGATE (merge finalize)\n  |  aggregate: sum[([20: sum, DOUBLE, true]); args: DOUBLE; result: DOUBLE; args nullable: true; result nullable: true], sum[([21: sum, DOUBLE, true]); args: DOUBLE; result: DOUBLE; args nullable: true; result nullable: true], sum[([22: sum, DOUBLE, true]); args: DOUBLE; result: DOUBLE; args nullable: true; result nullable: true], sum[([23: sum, DOUBLE, true]); args: DOUBLE; result: DOUBLE; args nullable: true; result nullable: true], avg[([24: avg, VARBINARY, true]); args: DOUBLE; result: DOUBLE; args nullable: true; result nullable: true], avg[([25: avg, VARBINARY, true]); args: DOUBLE; result: DOUBLE; args nullable: true; result nullable: true], avg[([26: avg, VARBINARY, true]); args: DOUBLE; result: DOUBLE; args nullable: true; result nullable: true], count[([27: count, BIGINT, false]); args: ; result: BIGINT; args nullable: true; result nullable: false]\n  |  group by: [9: L_RETURNFLAG, VARCHAR, false], [10: L_LINESTATUS, VARCHAR, false]\n  |  cardinality: 1\n  |  column statistics: \n  |  * L_RETURNFLAG-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n  |  * L_LINESTATUS-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n  |  * sum-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n  |  * sum-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n  |  * sum-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n  |  * sum-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n  |  * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n  |  * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n  |  * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n  |  * count-->[0.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n  |  \n  3:EXCHANGE\n     distribution type: SHUFFLE\n     partition exprs: [9: L_RETURNFLAG, VARCHAR, false], [10: L_LINESTATUS, VARCHAR, false]\n     cardinality: 1\n\nPLAN FRAGMENT 2(F00)\n\n  Input Partition: RANDOM\n  OutPut Partition: HASH_PARTITIONED: 9: L_RETURNFLAG, 10: L_LINESTATUS\n  OutPut Exchange Id: 03\n\n  2:AGGREGATE (update serialize)\n  |  STREAMING\n  |  aggregate: sum[([5: L_QUANTITY, DOUBLE, false]); args: DOUBLE; result: DOUBLE; args nullable: false; result nullable: true], sum[([6: L_EXTENDEDPRICE, DOUBLE, false]); args: DOUBLE; result: DOUBLE; args nullable: false; result nullable: true], sum[([18: expr, DOUBLE, false]); args: DOUBLE; result: DOUBLE; args nullable: false; result nullable: true], sum[([19: expr, DOUBLE, false]); args: DOUBLE; result: DOUBLE; args nullable: false; result nullable: true], avg[([5: L_QUANTITY, DOUBLE, false]); args: DOUBLE; result: VARBINARY; args nullable: false; result nullable: true], avg[([6: L_EXTENDEDPRICE, DOUBLE, false]); args: DOUBLE; result: VARBINARY; args nullable: false; result nullable: true], avg[([7: L_DISCOUNT, DOUBLE, false]); args: DOUBLE; result: VARBINARY; args nullable: false; result nullable: true], count[(*); args: ; result: BIGINT; args nullable: false; result nullable: false]\n  |  group by: [9: L_RETURNFLAG, VARCHAR, false], [10: L_LINESTATUS, VARCHAR, false]\n  |  cardinality: 1\n  |  column statistics: \n  |  * L_RETURNFLAG-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n  |  * L_LINESTATUS-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n  |  * sum-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n  |  * sum-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n  |  * sum-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n  |  * sum-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n  |  * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n  |  * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n  |  * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n  |  * count-->[0.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n  |  \n  1:Project\n  |  output columns:\n  |  5 <-> [5: L_QUANTITY, DOUBLE, false]\n  |  6 <-> [6: L_EXTENDEDPRICE, DOUBLE, false]\n  |  7 <-> [7: L_DISCOUNT, DOUBLE, false]\n  |  9 <-> [9: L_RETURNFLAG, CHAR, false]\n  |  10 <-> [10: L_LINESTATUS, CHAR, false]\n  |  18 <-> [29: multiply, DOUBLE, false]\n  |  19 <-> [29: multiply, DOUBLE, false] * 1.0 + [8: L_TAX, DOUBLE, false]\n  |  common expressions:\n  |  28 <-> 1.0 - [7: L_DISCOUNT, DOUBLE, false]\n  |  29 <-> [6: L_EXTENDEDPRICE, DOUBLE, false] * [28: subtract, DOUBLE, false]\n  |  cardinality: 1\n  |  column statistics: \n  |  * L_QUANTITY-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n  |  * L_EXTENDEDPRICE-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n  |  * L_DISCOUNT-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n  |  * L_RETURNFLAG-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n  |  * L_LINESTATUS-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n  |  * expr-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n  |  * expr-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n  |  \n  0:OlapScanNode\n     table: lineitem, rollup: lineitem\n     preAggregation: on\n     Predicates: [11: L_SHIPDATE, DATE, false] <= '1998-12-01'\n     partitionsRatio=1/1, tabletsRatio=20/20\n     tabletList=45030,45032,45034,45036,45038,45040,45042,45044,45046,45048 ...\n     actualRows=3, avgRowSize=54.0\n     cardinality: 1\n     column statistics: \n     * L_QUANTITY-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n     * L_EXTENDEDPRICE-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n     * L_DISCOUNT-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n     * L_TAX-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n     * L_RETURNFLAG-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n     * L_LINESTATUS-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n     * L_SHIPDATE-->[NaN, NaN, 0.0, 4.0, 1.0] ESTIMATE\n     * expr-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n     * expr-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n",
  "session_variables": "{\"partial_update_mode\":\"auto\",\"cbo_cte_reuse\":true,\"character_set_connection\":\"utf8\",\"cbo_use_correlated_join_estimate\":true,\"enable_insert_strict\":true,\"enable_connector_adaptive_io_tasks\":true,\"tx_isolation\":\"REPEATABLE-READ\",\"enable_hive_metadata_cache_with_insert\":false,\"cbo_cte_reuse_rate_v2\":1.15,\"character_set_results\":\"utf8\",\"enable_count_star_optimization\":true,\"query_excluding_mv_names\":\"\",\"enable_rewrite_simple_agg_to_meta_scan\":false,\"enable_adaptive_sink_dop\":true,\"consistent_hash_virtual_number\":32,\"enable_profile\":false,\"load_mem_limit\":0,\"sql_safe_updates\":0,\"runtime_filter_early_return_selectivity\":0.05,\"enable_local_shuffle_agg\":true,\"disable_function_fold_constants\":false,\"select_ratio_threshold\":0.15,\"query_delivery_timeout\":300,\"collation_database\":\"utf8_general_ci\",\"spill_mem_table_size\":104857600,\"cbo_use_lock_db\":false,\"new_planner_agg_stage\":0,\"use_compute_nodes\":-1,\"collation_connection\":\"utf8_general_ci\",\"resource_group\":\"\",\"profile_limit_fold\":true,\"spill_operator_max_bytes\":1048576000,\"cbo_max_reorder_node_use_dp\":10,\"enable_hive_column_stats\":true,\"enable_groupby_use_output_alias\":false,\"forward_to_leader\":false,\"count_distinct_column_buckets\":1024,\"query_cache_agg_cardinality_limit\":5000000,\"cboPushDownAggregateMode_v1\":-1,\"window_partition_mode\":1,\"enable_tablet_internal_parallel_v2\":true,\"interpolate_passthrough\":true,\"enable_incremental_mv\":false,\"SQL_AUTO_IS_NULL\":false,\"event_scheduler\":\"OFF\",\"max_pipeline_dop\":64,\"broadcast_right_table_scale_factor\":10,\"materialized_view_rewrite_mode\":\"DEFAULT\",\"enable_simplify_case_when\":true,\"runtime_join_filter_push_down_limit\":1024000,\"big_query_log_cpu_second_threshold\":480,\"div_precision_increment\":4,\"runtime_adaptive_dop_max_block_rows_per_driver_seq\":16384,\"log_rejected_record_num\":0,\"cbo_push_down_distinct_below_window\":true,\"sql_mode_v2\":32,\"prefer_cte_rewrite\":false,\"hdfs_backend_selector_scan_range_shuffle\":false,\"pipeline_profile_level\":1,\"parallel_fragment_exec_instance_num\":1,\"max_scan_key_num\":-1,\"net_read_timeout\":60,\"streaming_preaggregation_mode\":\"auto\",\"hive_partition_stats_sample_size\":3000,\"enable_mv_planner\":false,\"enable_collect_table_level_scan_stats\":true,\"profile_timeout\":2,\"cbo_push_down_aggregate\":\"global\",\"spill_encode_level\":7,\"enable_query_dump\":false,\"global_runtime_filter_build_max_size\":67108864,\"enable_rewrite_sum_by_associative_rule\":true,\"query_cache_hot_partition_num\":3,\"enable_prune_complex_types\":true,\"query_cache_type\":0,\"max_parallel_scan_instance_num\":-1,\"query_cache_entry_max_rows\":409600,\"enable_mv_optimizer_trace_log\":false,\"connector_io_tasks_per_scan_operator\":16,\"enable_materialized_view_union_rewrite\":true,\"sql_quote_show_create\":true,\"scan_or_to_union_threshold\":50000000,\"enable_exchange_pass_through\":true,\"runtime_profile_report_interval\":10,\"query_cache_entry_max_bytes\":4194304,\"enable_exchange_perf\":false,\"workgroup_id\":0,\"enable_rewrite_groupingsets_to_union_all\":false,\"transmission_compression_type\":\"NO_COMPRESSION\",\"interactive_timeout\":3600,\"use_page_cache\":true,\"big_query_log_scan_bytes_threshold\":10737418240,\"collation_server\":\"utf8_general_ci\",\"tablet_internal_parallel_mode\":\"auto\",\"enable_pipeline\":true,\"spill_mode\":\"auto\",\"enable_query_debug_trace\":false,\"enable_show_all_variables\":false,\"full_sort_max_buffered_bytes\":16777216,\"wait_timeout\":28800,\"transmission_encode_level\":7,\"query_including_mv_names\":\"\",\"transaction_isolation\":\"REPEATABLE-READ\",\"enable_global_runtime_filter\":true,\"enable_load_profile\":false,\"enable_plan_validation\":true,\"load_transmission_compression_type\":\"NO_COMPRESSION\",\"cbo_enable_low_cardinality_optimize\":true,\"scan_use_query_mem_ratio\":0.3,\"new_planner_optimize_timeout\":3000,\"enable_outer_join_reorder\":true,\"force_schedule_local\":false,\"hudi_mor_force_jni_reader\":false,\"cbo_enable_greedy_join_reorder\":true,\"range_pruner_max_predicate\":100,\"enable_rbo_table_prune\":false,\"spillable_operator_mask\":-1,\"rpc_http_min_size\":2147482624,\"cbo_debug_alive_backend_number\":0,\"global_runtime_filter_probe_min_size\":102400,\"scan_or_to_union_limit\":4,\"enable_cbo_table_prune\":false,\"enable_parallel_merge\":true,\"nested_mv_rewrite_max_level\":3,\"net_write_timeout\":60,\"cbo_prune_shuffle_column_rate\":0.1,\"spill_revocable_max_bytes\":0,\"hash_join_push_down_right_table\":true,\"pipeline_sink_dop\":0,\"broadcast_row_limit\":15000000,\"enable_populate_block_cache\":true,\"exec_mem_limit\":2147483648,\"enable_sort_aggregate\":false,\"query_cache_force_populate\":false,\"runtime_filter_on_exchange_node\":false,\"disable_join_reorder\":false,\"enable_rule_based_materialized_view_rewrite\":true,\"connector_scan_use_query_mem_ratio\":0.3,\"net_buffer_length\":16384,\"cbo_prune_subfield\":true,\"full_sort_max_buffered_rows\":1024000,\"query_timeout\":300,\"connector_io_tasks_slow_io_latency_ms\":50,\"cbo_max_reorder_node\":50,\"enable_distinct_column_bucketization\":false,\"enable_big_query_log\":true,\"analyze_mv\":\"sample\",\"runtime_filter_scan_wait_time\":20,\"enable_sync_materialized_view_rewrite\":true,\"prefer_compute_node\":false,\"enable_strict_type\":false,\"group_concat_max_len\":65535,\"parse_tokens_limit\":3500000,\"chunk_size\":4096,\"global_runtime_filter_probe_min_selectivity\":0.5,\"query_mem_limit\":0,\"enable_filter_unused_columns_in_scan_stage\":true,\"enable_scan_block_cache\":false,\"enable_materialized_view_single_table_view_delta_rewrite\":false,\"auto_increment_increment\":1,\"sql_dialect\":\"StarRocks\",\"big_query_log_scan_rows_threshold\":1000000000,\"character_set_client\":\"utf8\",\"autocommit\":true,\"enable_column_expr_predicate\":true,\"enable_runtime_adaptive_dop\":false,\"cbo_cte_max_limit\":10,\"storage_engine\":\"olap\",\"enable_optimizer_trace_log\":false,\"spill_operator_min_bytes\":52428800,\"cbo_enable_dp_join_reorder\":true,\"tx_visible_wait_timeout\":10,\"enable_materialized_view_view_delta_rewrite\":true,\"cbo_max_reorder_node_use_exhaustive\":4,\"enable_sql_digest\":false,\"spill_mem_table_num\":2,\"enable_spill\":false,\"pipeline_dop\":0,\"single_node_exec_plan\":false,\"full_sort_late_materialization_v2\":true,\"join_implementation_mode_v2\":\"auto\",\"sql_select_limit\":9223372036854775807,\"enable_materialized_view_rewrite\":true,\"statistic_collect_parallel\":1,\"hdfs_backend_selector_hash_algorithm\":\"consistent\",\"disable_colocate_join\":false,\"max_pushdown_conditions_per_column\":-1,\"default_table_compression\":\"lz4_frame\",\"runtime_adaptive_dop_max_output_amplification_factor\":0,\"innodb_read_only\":true,\"spill_mem_limit_threshold\":0.8,\"cbo_reorder_threshold_use_exhaustive\":6,\"enable_predicate_reorder\":false,\"enable_query_cache\":false,\"max_allowed_packet\":33554432,\"time_zone\":\"Asia/Shanghai\",\"enable_multicolumn_global_runtime_filter\":false,\"character_set_server\":\"utf8\",\"cbo_use_nth_exec_plan\":0,\"io_tasks_per_scan_operator\":4,\"parallel_exchange_instance_num\":-1,\"enable_shared_scan\":false,\"allow_default_partition\":false}",
  "be_number": 1,
  "be_core_stat": {
    "numOfHardwareCoresPerBe": "{\"10004\":104}",
    "cachedAvgNumOfHardwareCores": 104
  },
  "exception": [],
  "version": "main_querydump",
  "commit_version": "0c4d8c8d3e"
}

Enable desensitization (default)

Command:

wget --user=root --password=123 --post-file query_file "http://127.0.0.1:8030/api/query_dump?db=tpch -O dump_file

Return data:

The desensitized data is returned in JSON format.

{
  "statement": "SELECT tbl_mock_001.mock_012, tbl_mock_001.mock_007, sum(tbl_mock_001.mock_010) AS mock_019, sum(tbl_mock_001.mock_005) AS mock_020, sum(tbl_mock_001.mock_005 * (1 - tbl_mock_001.mock_004)) AS mock_021, sum((tbl_mock_001.mock_005 * (1 - tbl_mock_001.mock_004)) * (1 + tbl_mock_001.mock_017)) AS mock_022, avg(tbl_mock_001.mock_010) AS mock_023, avg(tbl_mock_001.mock_005) AS mock_024, avg(tbl_mock_001.mock_004) AS mock_025, count(*) AS mock_026\nFROM db_mock_000.tbl_mock_001\nWHERE tbl_mock_001.mock_013 <= '1998-12-01'\nGROUP BY tbl_mock_001.mock_012, tbl_mock_001.mock_007 ORDER BY tbl_mock_001.mock_012 ASC, tbl_mock_001.mock_007 ASC ",
  "table_meta": {
    "db_mock_000.tbl_mock_001": "CREATE TABLE db_mock_000.tbl_mock_001 (\nmock_008 int(11) NOT NULL ,\nmock_009 int(11) NOT NULL ,\nmock_016 int(11) NOT NULL ,\nmock_006 int(11) NOT NULL ,\nmock_010 double NOT NULL ,\nmock_005 double NOT NULL ,\nmock_004 double NOT NULL ,\nmock_017 double NOT NULL ,\nmock_012 char(1) NOT NULL ,\nmock_007 char(1) NOT NULL ,\nmock_013 date NOT NULL ,\nmock_003 date NOT NULL ,\nmock_011 date NOT NULL ,\nmock_014 char(25) NOT NULL ,\nmock_015 char(10) NOT NULL ,\nmock_002 varchar(44) NOT NULL ,\nmock_018 char(1) NOT NULL \n) ENGINE= OLAP \nDUPLICATE KEY(mock_008)\nDISTRIBUTED BY HASH(mock_008) BUCKETS 20 \nPROPERTIES (\n\"replication_num\" = \"1\"\n);"
  },
  "table_row_count": {
    "db_mock_000.tbl_mock_001": {
      "tbl_mock_001": 3
    }
  },
  "column_statistics": {
    "db_mock_000.tbl_mock_001": {
      "mock_017": "[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE",
      "mock_013": "[1.6094304E9, 1.6094304E9, 0.0, 4.0, 1.0] ESTIMATE",
      "mock_005": "[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE",
      "mock_004": "[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE",
      "mock_012": "[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE",
      "mock_007": "[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE",
      "mock_010": "[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE"
    }
  },
  "explain_info": "PLAN FRAGMENT 0(F02)\n  Output Exprs:9: mock_012 | 10: mock_007 | 20: sum | 21: sum | 22: sum | 23: sum | 24: avg | 25: avg | 26: avg | 27: count\n  Input Partition: UNPARTITIONED\n  RESULT SINK\n\n  6:MERGING-EXCHANGE\n     distribution type: GATHER\n     cardinality: 1\n     column statistics: \n     * mock_012-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n     * mock_007-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n     * sum-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n     * sum-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n     * sum-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n     * sum-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n     * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n     * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n     * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n     * count-->[0.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n\nPLAN FRAGMENT 1(F01)\n\n  Input Partition: HASH_PARTITIONED: 9: mock_012, 10: mock_007\n  OutPut Partition: UNPARTITIONED\n  OutPut Exchange id: 06\n\n  5:SORT\n  |  order by: [9, VARCHAR, false] ASC, [10, VARCHAR, false] ASC\n  |  offset: 0\n  |  cardinality: 1\n  |  column statistics: \n  |  * mock_012-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n  |  * mock_007-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n  |  * sum-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n  |  * sum-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n  |  * sum-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n  |  * sum-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n  |  * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n  |  * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n  |  * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n  |  * count-->[0.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n  |  \n  4:AGGREGATE (merge finalize)\n  |  aggregate: sum[([20: sum, DOUBLE, true]); args: DOUBLE; result: DOUBLE; args nullable: true; result nullable: true], sum[([21: sum, DOUBLE, true]); args: DOUBLE; result: DOUBLE; args nullable: true; result nullable: true], sum[([22: sum, DOUBLE, true]); args: DOUBLE; result: DOUBLE; args nullable: true; result nullable: true], sum[([23: sum, DOUBLE, true]); args: DOUBLE; result: DOUBLE; args nullable: true; result nullable: true], avg[([24: avg, VARBINARY, true]); args: DOUBLE; result: DOUBLE; args nullable: true; result nullable: true], avg[([25: avg, VARBINARY, true]); args: DOUBLE; result: DOUBLE; args nullable: true; result nullable: true], avg[([26: avg, VARBINARY, true]); args: DOUBLE; result: DOUBLE; args nullable: true; result nullable: true], count[([27: count, BIGINT, false]); args: ; result: BIGINT; args nullable: true; result nullable: false]\n  |  group by: [9: mock_012, VARCHAR, false], [10: mock_007, VARCHAR, false]\n  |  cardinality: 1\n  |  column statistics: \n  |  * mock_012-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n  |  * mock_007-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n  |  * sum-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n  |  * sum-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n  |  * sum-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n  |  * sum-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n  |  * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n  |  * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n  |  * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n  |  * count-->[0.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n  |  \n  3:EXCHANGE\n     distribution type: SHUFFLE\n     partition exprs: [9: mock_012, VARCHAR, false], [10: mock_007, VARCHAR, false]\n     cardinality: 1\n\nPLAN FRAGMENT 2(F00)\n\n  Input Partition: RANDOM\n  OutPut Partition: HASH_PARTITIONED: 9: mock_012, 10: mock_007\n  OutPut Exchange id: 03\n\n  2:AGGREGATE (update serialize)\n  |  STREAMING\n  |  aggregate: sum[([5: mock_010, DOUBLE, false]); args: DOUBLE; result: DOUBLE; args nullable: false; result nullable: true], sum[([6: mock_005, DOUBLE, false]); args: DOUBLE; result: DOUBLE; args nullable: false; result nullable: true], sum[([18: expr, DOUBLE, false]); args: DOUBLE; result: DOUBLE; args nullable: false; result nullable: true], sum[([19: expr, DOUBLE, false]); args: DOUBLE; result: DOUBLE; args nullable: false; result nullable: true], avg[([5: mock_010, DOUBLE, false]); args: DOUBLE; result: VARBINARY; args nullable: false; result nullable: true], avg[([6: mock_005, DOUBLE, false]); args: DOUBLE; result: VARBINARY; args nullable: false; result nullable: true], avg[([7: mock_004, DOUBLE, false]); args: DOUBLE; result: VARBINARY; args nullable: false; result nullable: true], count[(*); args: ; result: BIGINT; args nullable: false; result nullable: false]\n  |  group by: [9: mock_012, VARCHAR, false], [10: mock_007, VARCHAR, false]\n  |  cardinality: 1\n  |  column statistics: \n  |  * mock_012-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n  |  * mock_007-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n  |  * sum-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n  |  * sum-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n  |  * sum-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n  |  * sum-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n  |  * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n  |  * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n  |  * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n  |  * count-->[0.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n  |  \n  1:Project\n  |  output columns:\n  |  5 <-> [5: mock_010, DOUBLE, false]\n  |  6 <-> [6: mock_005, DOUBLE, false]\n  |  7 <-> [7: mock_004, DOUBLE, false]\n  |  9 <-> [9: mock_012, CHAR, false]\n  |  10 <-> [10: mock_007, CHAR, false]\n  |  18 <-> [29: multiply, DOUBLE, false]\n  |  19 <-> [29: multiply, DOUBLE, false] * 1.0 + [8: mock_017, DOUBLE, false]\n  |  common expressions:\n  |  28 <-> 1.0 - [7: mock_004, DOUBLE, false]\n  |  29 <-> [6: mock_005, DOUBLE, false] * [28: subtract, DOUBLE, false]\n  |  cardinality: 1\n  |  column statistics: \n  |  * mock_010-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n  |  * mock_005-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n  |  * mock_004-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n  |  * mock_012-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n  |  * mock_007-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n  |  * expr-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n  |  * expr-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n  |  \n  0:OlapScanNode\n     table: mock_001, rollup: mock_001\n     preAggregation: on\n     Predicates: [11: mock_013, DATE, false] <= '1998-12-01'\n     partitionsRatio=1/1, tabletsRatio=20/20\n     tabletList=45030,45032,45034,45036,45038,45040,45042,45044,45046,45048 ...\n     actualRows=3, avgRowSize=54.0\n     cardinality: 1\n     column statistics: \n     * mock_010-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n     * mock_005-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n     * mock_004-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n     * mock_017-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n     * mock_012-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n     * mock_007-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n     * mock_013-->[NaN, NaN, 0.0, 4.0, 1.0] ESTIMATE\n     * expr-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n     * expr-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n",
  "session_variables": "{\"partial_update_mode\":\"auto\",\"cbo_cte_reuse\":true,\"character_set_connection\":\"utf8\",\"cbo_use_correlated_join_estimate\":true,\"enable_insert_strict\":true,\"enable_connector_adaptive_io_tasks\":true,\"tx_isolation\":\"REPEATABLE-READ\",\"enable_hive_metadata_cache_with_insert\":false,\"cbo_cte_reuse_rate_v2\":1.15,\"character_set_results\":\"utf8\",\"enable_count_star_optimization\":true,\"query_excluding_mv_names\":\"\",\"enable_rewrite_simple_agg_to_meta_scan\":false,\"enable_adaptive_sink_dop\":true,\"consistent_hash_virtual_number\":32,\"enable_profile\":false,\"load_mem_limit\":0,\"sql_safe_updates\":0,\"runtime_filter_early_return_selectivity\":0.05,\"enable_local_shuffle_agg\":true,\"disable_function_fold_constants\":false,\"select_ratio_threshold\":0.15,\"query_delivery_timeout\":300,\"collation_database\":\"utf8_general_ci\",\"spill_mem_table_size\":104857600,\"cbo_use_lock_db\":false,\"new_planner_agg_stage\":0,\"use_compute_nodes\":-1,\"collation_connection\":\"utf8_general_ci\",\"resource_group\":\"\",\"profile_limit_fold\":true,\"spill_operator_max_bytes\":1048576000,\"cbo_max_reorder_node_use_dp\":10,\"enable_hive_column_stats\":true,\"enable_groupby_use_output_alias\":false,\"forward_to_leader\":false,\"count_distinct_column_buckets\":1024,\"query_cache_agg_cardinality_limit\":5000000,\"cboPushDownAggregateMode_v1\":-1,\"window_partition_mode\":1,\"enable_tablet_internal_parallel_v2\":true,\"interpolate_passthrough\":true,\"enable_incremental_mv\":false,\"SQL_AUTO_IS_NULL\":false,\"event_scheduler\":\"OFF\",\"max_pipeline_dop\":64,\"broadcast_right_table_scale_factor\":10,\"materialized_view_rewrite_mode\":\"DEFAULT\",\"enable_simplify_case_when\":true,\"runtime_join_filter_push_down_limit\":1024000,\"big_query_log_cpu_second_threshold\":480,\"div_precision_increment\":4,\"runtime_adaptive_dop_max_block_rows_per_driver_seq\":16384,\"log_rejected_record_num\":0,\"cbo_push_down_distinct_below_window\":true,\"sql_mode_v2\":32,\"prefer_cte_rewrite\":false,\"hdfs_backend_selector_scan_range_shuffle\":false,\"pipeline_profile_level\":1,\"parallel_fragment_exec_instance_num\":1,\"max_scan_key_num\":-1,\"net_read_timeout\":60,\"streaming_preaggregation_mode\":\"auto\",\"hive_partition_stats_sample_size\":3000,\"enable_mv_planner\":false,\"enable_collect_table_level_scan_stats\":true,\"profile_timeout\":2,\"cbo_push_down_aggregate\":\"global\",\"spill_encode_level\":7,\"enable_query_dump\":false,\"global_runtime_filter_build_max_size\":67108864,\"enable_rewrite_sum_by_associative_rule\":true,\"query_cache_hot_partition_num\":3,\"enable_prune_complex_types\":true,\"query_cache_type\":0,\"max_parallel_scan_instance_num\":-1,\"query_cache_entry_max_rows\":409600,\"enable_mv_optimizer_trace_log\":false,\"connector_io_tasks_per_scan_operator\":16,\"enable_materialized_view_union_rewrite\":true,\"sql_quote_show_create\":true,\"scan_or_to_union_threshold\":50000000,\"enable_exchange_pass_through\":true,\"runtime_profile_report_interval\":10,\"query_cache_entry_max_bytes\":4194304,\"enable_exchange_perf\":false,\"workgroup_id\":0,\"enable_rewrite_groupingsets_to_union_all\":false,\"transmission_compression_type\":\"NO_COMPRESSION\",\"interactive_timeout\":3600,\"use_page_cache\":true,\"big_query_log_scan_bytes_threshold\":10737418240,\"collation_server\":\"utf8_general_ci\",\"tablet_internal_parallel_mode\":\"auto\",\"enable_pipeline\":true,\"spill_mode\":\"auto\",\"enable_query_debug_trace\":false,\"enable_show_all_variables\":false,\"full_sort_max_buffered_bytes\":16777216,\"wait_timeout\":28800,\"transmission_encode_level\":7,\"query_including_mv_names\":\"\",\"transaction_isolation\":\"REPEATABLE-READ\",\"enable_global_runtime_filter\":true,\"enable_load_profile\":false,\"enable_plan_validation\":true,\"load_transmission_compression_type\":\"NO_COMPRESSION\",\"cbo_enable_low_cardinality_optimize\":true,\"scan_use_query_mem_ratio\":0.3,\"new_planner_optimize_timeout\":3000,\"enable_outer_join_reorder\":true,\"force_schedule_local\":false,\"hudi_mor_force_jni_reader\":false,\"cbo_enable_greedy_join_reorder\":true,\"range_pruner_max_predicate\":100,\"enable_rbo_table_prune\":false,\"spillable_operator_mask\":-1,\"rpc_http_min_size\":2147482624,\"cbo_debug_alive_backend_number\":0,\"global_runtime_filter_probe_min_size\":102400,\"scan_or_to_union_limit\":4,\"enable_cbo_table_prune\":false,\"enable_parallel_merge\":true,\"nested_mv_rewrite_max_level\":3,\"net_write_timeout\":60,\"cbo_prune_shuffle_column_rate\":0.1,\"spill_revocable_max_bytes\":0,\"hash_join_push_down_right_table\":true,\"pipeline_sink_dop\":0,\"broadcast_row_limit\":15000000,\"enable_populate_block_cache\":true,\"exec_mem_limit\":2147483648,\"enable_sort_aggregate\":false,\"query_cache_force_populate\":false,\"runtime_filter_on_exchange_node\":false,\"disable_join_reorder\":false,\"enable_rule_based_materialized_view_rewrite\":true,\"connector_scan_use_query_mem_ratio\":0.3,\"net_buffer_length\":16384,\"cbo_prune_subfield\":true,\"full_sort_max_buffered_rows\":1024000,\"query_timeout\":300,\"connector_io_tasks_slow_io_latency_ms\":50,\"cbo_max_reorder_node\":50,\"enable_distinct_column_bucketization\":false,\"enable_big_query_log\":true,\"analyze_mv\":\"sample\",\"runtime_filter_scan_wait_time\":20,\"enable_sync_materialized_view_rewrite\":true,\"prefer_compute_node\":false,\"enable_strict_type\":false,\"group_concat_max_len\":65535,\"parse_tokens_limit\":3500000,\"chunk_size\":4096,\"global_runtime_filter_probe_min_selectivity\":0.5,\"query_mem_limit\":0,\"enable_filter_unused_columns_in_scan_stage\":true,\"enable_scan_block_cache\":false,\"enable_materialized_view_single_table_view_delta_rewrite\":false,\"auto_increment_increment\":1,\"sql_dialect\":\"StarRocks\",\"big_query_log_scan_rows_threshold\":1000000000,\"character_set_client\":\"utf8\",\"autocommit\":true,\"enable_column_expr_predicate\":true,\"enable_runtime_adaptive_dop\":false,\"cbo_cte_max_limit\":10,\"storage_engine\":\"olap\",\"enable_optimizer_trace_log\":false,\"spill_operator_min_bytes\":52428800,\"cbo_enable_dp_join_reorder\":true,\"tx_visible_wait_timeout\":10,\"enable_materialized_view_view_delta_rewrite\":true,\"cbo_max_reorder_node_use_exhaustive\":4,\"enable_sql_digest\":false,\"spill_mem_table_num\":2,\"enable_spill\":false,\"pipeline_dop\":0,\"single_node_exec_plan\":false,\"full_sort_late_materialization_v2\":true,\"join_implementation_mode_v2\":\"auto\",\"sql_select_limit\":9223372036854775807,\"enable_materialized_view_rewrite\":true,\"statistic_collect_parallel\":1,\"hdfs_backend_selector_hash_algorithm\":\"consistent\",\"disable_colocate_join\":false,\"max_pushdown_conditions_per_column\":-1,\"default_table_compression\":\"lz4_frame\",\"runtime_adaptive_dop_max_output_amplification_factor\":0,\"innodb_read_only\":true,\"spill_mem_limit_threshold\":0.8,\"cbo_reorder_threshold_use_exhaustive\":6,\"enable_predicate_reorder\":false,\"enable_query_cache\":false,\"max_allowed_packet\":33554432,\"time_zone\":\"Asia/Shanghai\",\"enable_multicolumn_global_runtime_filter\":false,\"character_set_server\":\"utf8\",\"cbo_use_nth_exec_plan\":0,\"io_tasks_per_scan_operator\":4,\"parallel_exchange_instance_num\":-1,\"enable_shared_scan\":false,\"allow_default_partition\":false}",
  "be_number": 1,
  "be_core_stat": {
    "numOfHardwareCoresPerBe": "{\"10004\":104}",
    "cachedAvgNumOfHardwareCores": 104
  },
  "exception": [],
  "version": "main_querydump",
  "commit_version": "0c4d8c8d3e"
}

query_dump interface

Function overview​

Syntax​

Examples​

Disable desensitization​

Enable desensitization (default)​

What did you think of this doc?

Function overview

Syntax

Examples

Disable desensitization

Enable desensitization (default)