Google Cloud Storage

5min
Google Cloud Storage also known as GCP Storage is the Google Cloud object storage or blob storage concept for storing files and other objects in the cloud.
DLH.io provides this connector as a direct way to work with data and files both as source and as target (i.e.: backup) conduits.  
GCP Storage is mainly used for synchronizing data into BigQuery but can also be used for other general synchronization data flows and pipelines. 
GCP Storage Pre-Requisities:
Name of your GCP Project
Name of your GCP Storage Bucket
Service Account Key JSON
Setup Instructions 
DataLakeHouse.io securely connects to your Google Cloud Storage bucket. Using the form in the DataLakeHouse.io portal please complete the following basic steps. 
Enter a Name or Alias for this connection, in the Name/Alias field, that is unique from other connectors

Enter a Target Schema Prefix, which will be the prefix for the schema at the target you will sync your data files into

Enter a Bucket name, where your files are stored
Typically just the name of the bucket. No http or gs prefixes required.

Select your 'Region'

Enter any other optional details in the available fields (See the setup video if you need help or contact support)
Folder Path, is a path on the root bucket from where desired files will be retrieved
File Pattern, is a regular expression (RegEx) used to isolated only certain files to be retrieved
File Type, allows for a pre-determined type of file extension to be retreived

Enter your Service Account Key, which should be a JSON string
Paste the entire Service Account Key (JSON).
If using this bucket as process storage for BigQuery, please Add this Service Account email, dlh-global-bq-data-sync-svc@stg-datalakehouse.iam.gserviceaccount.com, as a principal user on your GCP project and assign it the Storage Admin role.

Click the Save & Test button. Once your credentials are accepted you should be able to see a successful connection.
FAQs & Troubleshooting
This issue is due to how dlh.io needs to retrieve file information from your bucket.  If you see an error message containing this warning or error, it means that you need to update your permissions for the Service Account and/or the dlh.io user shown in the connection to have either the Storage Admin role granted or create a custom role in your GCP project with this permission.  See this Serverfault.com answer for some general direction, if needed.
..
DLH.io runs a test when you create this GCP Storage source connector which lists the files in the bucket and some other steps to ensure that DLH.io can act as the conduit to work with your bucket. If any portion of the test fails a notification o the permission issue should appear in the logs, alerts, or in the page when the test is conducted.



No. Not currently as of 01/2023, it is only available for BigQuery processing.
Control Each Column Data Type
﻿SQL Transformations﻿ allow logic to be executed against a target connection based on a scheduled frequency or triggered event of new data on tables updated via DataLakeHouse.io (DLH.io). This especially helps when you want to control the data type set in your Target Connection since all columns are set as VARCHAR(16777216). 
﻿
﻿
﻿
Updated 19 Jun 2024
Did this page help you?
What is the file size limit for flat file uploads?
Google Drive
Google Cloud Storage

Setup Instructions

FAQs & Troubleshooting

Why am I getting a storage.buckets.get error?

Can we use the GCP Storage Option for non-BigQuery DWs?

Control Each Column Data Type
