BigQuery Data Transfer Config

BigQuery Data Transfer Config is a Google Cloud resource that automates data movement into BigQuery from various sources such as Google Ads, YouTube, Cloud Storage, or external SaaS applications. It defines the configuration for scheduled or on-demand data transfers, including source type, destination dataset, refresh frequency, and credentials. This resource helps streamline data ingestion, ensuring that BigQuery tables stay up to date without manual intervention.

gcp.bigquerydatatransfer_transfer_config

Fields

TitleIDTypeData TypeDescription
_keycorestring
ancestorscorearray<string>
data_refresh_window_dayscoreint64The number of days to look back to automatically refresh the data. For example, if `data_refresh_window_days = 10`, then every day BigQuery reingests data for [today-10, today-1], rather than ingesting data for just [today-1]. Only valid if the data source supports the feature. Set the value to 0 to use the default value.
data_source_idcorestringData source ID. This cannot be changed once data transfer is created. The full list of available data source IDs can be returned through an API call: https://cloud.google.com/bigquery-transfer/docs/reference/datatransfer/rest/v1/projects.locations.dataSources/list
datadog_display_namecorestring
dataset_regioncorestringOutput only. Region in which BigQuery dataset is located.
destination_dataset_idcorestringThe BigQuery target dataset id.
disabledcoreboolIs this config disabled. When set to true, no runs will be scheduled for this transfer config.
email_preferencescorejsonEmail notifications will be sent according to these preferences to the email address of the user who owns this transfer config.
encryption_configurationcorejsonThe encryption configuration part. Currently, it is only used for the optional KMS key name. The BigQuery service account of your project must be granted permissions to use the key. Read methods will return the key name applied in effect. Write methods will apply the key if it is present, or otherwise try to apply project default keys if it is absent.
errorcorejsonOutput only. Error code with detailed information about reason of the latest config failure.
gcp_display_namecorestringUser specified display name for the data transfer.
labelscorearray<string>
namecorestringIdentifier. The resource name of the transfer config. Transfer config names have the form either `projects/{project_id}/locations/{region}/transferConfigs/{config_id}` or `projects/{project_id}/transferConfigs/{config_id}`, where `config_id` is usually a UUID, even though it is not guaranteed or required. The name is ignored when creating a transfer config.
next_run_timecoretimestampOutput only. Next time when data transfer will run.
notification_pubsub_topiccorestringPub/Sub topic where notifications will be sent after transfer runs associated with this transfer config finish. The format for specifying a pubsub topic is: `projects/{project_id}/topics/{topic_id}`
organization_idcorestring
owner_infocorejsonOutput only. Information about the user whose credentials are used to transfer data. Populated only for `transferConfigs.get` requests. In case the user information is not available, this field will not be populated.
parentcorestring
project_idcorestring
project_numbercorestring
resource_namecorestring
schedulecorestringData transfer schedule. If the data source does not support a custom schedule, this should be empty. If it is empty, the default value for the data source will be used. The specified times are in UTC. Examples of valid format: `1st,3rd monday of month 15:30`, `every wed,fri of jan,jun 13:15`, and `first sunday of quarter 00:00`. See more explanation about the format here: https://cloud.google.com/appengine/docs/flexible/python/scheduling-jobs-with-cron-yaml#the_schedule_format NOTE: The minimum interval time between recurring transfers depends on the data source; refer to the documentation for your data source.
schedule_optionscorejsonOptions customizing the data transfer schedule.
schedule_options_v2corejsonOptions customizing different types of data transfer schedule. This field replaces "schedule" and "schedule_options" fields. ScheduleOptionsV2 cannot be used together with ScheduleOptions/Schedule.
statecorestringOutput only. State of the most recently updated transfer run.
tagscorehstore
update_timecoretimestampOutput only. Data transfer modification time. Ignored by server on input.
user_idcoreint64Deprecated. Unique ID of the user on whose behalf transfer is done.