Skip to main content
Version: Next

Dremio

Certified

Important Capabilities​

CapabilityStatusNotes
Asset Containers✅Enabled by default
Data Profiling✅Optionally enabled via configuration
Descriptions✅Enabled by default
Detect Deleted Entities✅Optionally enabled via stateful_ingestion.remove_stale_metadata
Domains✅Supported via the domain config field
Extract Ownership✅Enabled by default
Platform Instance✅Enabled by default
Table-Level Lineage✅Enabled by default

This plugin extracts the following:

  • Metadata for databases, schemas, views and tables
  • Column types associated with each table
  • Table, row, and column statistics via optional SQL profiling
  • Lineage information for views and datasets

CLI based Ingestion​

Install the Plugin​

The dremio source works out of the box with acryl-datahub.

Starter Recipe​

Check out the following recipe to get started with ingestion! See below for full configuration options.

For general pointers on writing and running a recipe, see our main recipe guide.

source:
  type: dremio
  config:
    # Coordinates
    hostname: localhost
    port: 9047
    tls: true

    # Credentials
    authentication_method: password
    username: user
    password: pass

    include_query_lineage: True

    source_mappings:
      - platform: s3
        platform_name: samples

    schema_pattern:
      allow:
        - ".*"

sink:
  # sink configs

Config Details​

Note that a . is used to denote nested fields in the YAML recipe.

FieldDescription
authentication_method
string
Authentication method: 'password' or 'PAT' (Personal Access Token)
Default: password
disable_certificate_verification
boolean
Disable TLS certificate verification
Default: False
domain
string
Domain for all source objects.
dremio_cloud_region
Enum
One of: "US", "EMEA"
Default: US
hostname
string
Hostname or IP Address of the Dremio server
include_copy_lineage
boolean
Whether to include copy lineage
Default: True
include_query_lineage
boolean
Whether to include query-based lineage information.
Default: False
include_table_rename_lineage
boolean
Whether to include table rename lineage
Default: True
is_dremio_cloud
boolean
Whether this is a Dremio Cloud instance
Default: False
max_workers
integer
Number of worker threads to use for parallel processing
Default: 20
password
string
Dremio password or Personal Access Token
path_to_certificates
string
Path to SSL certificates
Default: /vercel/path0/metadata-ingestion/venv/lib/python3....
platform_instance
string
The instance of the platform that all assets produced by this recipe belong to. This should be unique within the platform. See https://datahubproject.io/docs/platform-instances/ for more details.
port
integer
Port of the Dremio REST API
Default: 9047
tls
boolean
Whether the Dremio REST API port is encrypted
Default: True
username
string
Dremio username
env
string
Environment to use in namespace when constructing URNs.
Default: PROD
dataset_pattern
AllowDenyPattern
Regex patterns for schemas to filter
Default: {'allow': ['.*'], 'deny': [], 'ignoreCase': True}
dataset_pattern.ignoreCase
boolean
Whether to ignore case sensitivity during pattern matching.
Default: True
dataset_pattern.allow
array
List of regex patterns to include in ingestion
Default: ['.*']
dataset_pattern.allow.string
string
dataset_pattern.deny
array
List of regex patterns to exclude from ingestion.
Default: []
dataset_pattern.deny.string
string
profile_pattern
AllowDenyPattern
Regex patterns for tables to profile
Default: {'allow': ['.*'], 'deny': [], 'ignoreCase': True}
profile_pattern.ignoreCase
boolean
Whether to ignore case sensitivity during pattern matching.
Default: True
profile_pattern.allow
array
List of regex patterns to include in ingestion
Default: ['.*']
profile_pattern.allow.string
string
profile_pattern.deny
array
List of regex patterns to exclude from ingestion.
Default: []
profile_pattern.deny.string
string
schema_pattern
AllowDenyPattern
Regex patterns for schemas to filter
Default: {'allow': ['.*'], 'deny': [], 'ignoreCase': True}
schema_pattern.ignoreCase
boolean
Whether to ignore case sensitivity during pattern matching.
Default: True
schema_pattern.allow
array
List of regex patterns to include in ingestion
Default: ['.*']
schema_pattern.allow.string
string
schema_pattern.deny
array
List of regex patterns to exclude from ingestion.
Default: []
schema_pattern.deny.string
string
source_mappings
array
Mappings from Dremio sources to DataHub platforms and datasets.
source_mappings.DremioSourceMapping
DremioSourceMapping
source_mappings.DremioSourceMapping.dremio_source_type
string
Source connection made by Dremio (e.g. S3, Snowflake)
source_mappings.DremioSourceMapping.platform
string
Source connection made by Dremio (e.g. S3, Snowflake)
source_mappings.DremioSourceMapping.platform_instance
string
Platform instance of source connection in Datahub
source_mappings.DremioSourceMapping.platform_name
string
Alias of platform in Dremio connection
source_mappings.DremioSourceMapping.env
string
ENV in Datahub of source connection made by Dremio (e.g. PROD)
Default: PROD
profiling
ProfileConfig
Configuration for profiling
Default: {'enabled': False, 'operation_config': {'lower_fre...
profiling.catch_exceptions
boolean
Default: True
profiling.column_count
boolean
Default: True
profiling.enabled
boolean
Whether profiling should be done.
Default: False
profiling.field_sample_values_limit
integer
Upper limit for number of sample values to collect for all columns.
Default: 20
profiling.include_field_distinct_count
boolean
Whether to profile for the number of distinct values for each column.
Default: True
profiling.include_field_distinct_value_frequencies
boolean
Whether to profile for distinct value frequencies.
Default: False
profiling.include_field_histogram
boolean
Whether to profile for the histogram for numeric fields.
Default: False
profiling.include_field_max_value
boolean
Whether to profile for the max value of numeric columns.
Default: True
profiling.include_field_mean_value
boolean
Whether to profile for the mean value of numeric columns.
Default: True
profiling.include_field_median_value
boolean
Median causes a number of issues in Dremio.
Default: False
profiling.include_field_min_value
boolean
Whether to profile for the min value of numeric columns.
Default: True
profiling.include_field_null_count
boolean
Whether to profile for the number of nulls for each column.
Default: True
profiling.include_field_quantiles
boolean
Whether to profile for the quantiles of numeric columns.
Default: False
profiling.include_field_sample_values
boolean
Whether to profile for the sample values for all columns.
Default: True
profiling.include_field_stddev_value
boolean
Whether to profile for the standard deviation of numeric columns.
Default: True
profiling.limit
integer
Max number of documents to profile. By default, profiles all documents.
profiling.max_number_of_fields_to_profile
integer
A positive integer that specifies the maximum number of columns to profile for any table. None implies all columns. The cost of profiling goes up significantly as the number of columns to profile goes up.
profiling.max_workers
integer
Number of worker threads to use for profiling. Set to 1 to disable.
Default: 20
profiling.offset
integer
Offset in documents to profile. By default, uses no offset.
profiling.partition_datetime
string(date-time)
If specified, profile only the partition which matches this datetime. If not specified, profile the latest partition. Only Bigquery supports this.
profiling.partition_profiling_enabled
boolean
Partition profiling disabled for Dremio.
Default: False
profiling.profile_external_tables
boolean
Whether to profile external tables. Only Snowflake and Redshift supports this.
Default: False
profiling.profile_if_updated_since_days
number
Profile table only if it has been updated since these many number of days. If set to null, no constraint of last modified time for tables to profile. Supported only in snowflake and BigQuery.
profiling.profile_table_level_only
boolean
Whether to perform profiling at table-level only, or include column-level profiling as well.
Default: False
profiling.profile_table_row_count_estimate_only
boolean
Use an approximate query for row count. This will be much faster but slightly less accurate. Only supported for Postgres and MySQL.
Default: False
profiling.profile_table_row_limit
integer
Profile tables only if their row count is less then specified count. If set to null, no limit on the row count of tables to profile. Supported only in snowflake and BigQuery
Default: 5000000
profiling.profile_table_size_limit
integer
Profile tables only if their size is less then specified GBs. If set to null, no limit on the size of tables to profile. Supported only in snowflake and BigQuery
Default: 5
profiling.query_combiner_enabled
boolean
This feature is still experimental and can be disabled if it causes issues. Reduces the total number of queries issued and speeds up profiling by dynamically combining SQL queries where possible.
Default: True
profiling.query_timeout
integer
Time before cancelling Dremio profiling query
Default: 300
profiling.report_dropped_profiles
boolean
Whether to report datasets or dataset columns which were not profiled. Set to True for debugging purposes.
Default: False
profiling.row_count
boolean
Default: True
profiling.sample_size
integer
Number of rows to be sampled from table for column level profiling.Applicable only if use_sampling is set to True.
Default: 10000
profiling.sample_values
boolean
Default: True
profiling.turn_off_expensive_profiling_metrics
boolean
Whether to turn off expensive profiling or not. This turns off profiling for quantiles, distinct_value_frequencies, histogram & sample_values. This also limits maximum number of fields being profiled to 10.
Default: False
profiling.use_sampling
boolean
Whether to profile column level stats on sample of table. Only BigQuery and Snowflake support this. If enabled, profiling is done on rows sampled from table. Sampling is not done for smaller tables.
Default: True
profiling.operation_config
OperationConfig
Experimental feature. To specify operation configs.
profiling.operation_config.lower_freq_profile_enabled
boolean
Whether to do profiling at lower freq or not. This does not do any scheduling just adds additional checks to when not to run profiling.
Default: False
profiling.operation_config.profile_date_of_month
integer
Number between 1 to 31 for date of month (both inclusive). If not specified, defaults to Nothing and this field does not take affect.
profiling.operation_config.profile_day_of_week
integer
Number between 0 to 6 for day of week (both inclusive). 0 is Monday and 6 is Sunday. If not specified, defaults to Nothing and this field does not take affect.
profiling.tags_to_ignore_sampling
array
Fixed list of tags to ignore sampling. If not specified, tables will be sampled based on use_sampling.
profiling.tags_to_ignore_sampling.string
string
stateful_ingestion
StatefulIngestionConfig
Stateful Ingestion Config
stateful_ingestion.enabled
boolean
Whether or not to enable stateful ingest. Default: True if a pipeline_name is set and either a datahub-rest sink or datahub_api is specified, otherwise False
Default: False

Code Coordinates​

  • Class Name: datahub.ingestion.source.dremio.dremio_source.DremioSource
  • Browse on GitHub

Questions

If you've got any questions on configuring ingestion for Dremio, feel free to ping us on our Slack.