Databricks

databricks a composible data lake and lakehouse architecture platform that runs on all major cloud vendor systems to enable ml, ai, data warehousing, and data science capabilities from develop through to production lifecycles databricks prerequisites be sure to our ip grantlist / whitelist docid\ ernsp9vcy4af88jk0uexc the dlh io ip addresses for any cloud vendor or databricks server or network access required have created a databricks personal access token following their documentation, https //docs databricks com/en/dev tools/auth/pat html https //docs databricks com/en/dev tools/auth/pat html or oauth 2 0 machine to machine (m2m) access using the documentation, https //docs databricks com/en/dev tools/auth/oauth m2m html https //docs databricks com/en/dev tools/auth/oauth m2m html once a type of authorization is selected in dlh io connection configuration, it cannot be changed for that connector so, choose the method desired appropriately before proceeding for m2m oauth be sure to capture your client id, client secret, and workspace url for your databricks target if using a user's personal access token, the following permissions must be granted create modify read metadata select usage if (optional) able to specify a catalog in the connection form that existing catalog should allow the following permissions for the service principal/account create schema create table modify select use catalog use schema have a databricks sql warehouse or compute (cluster/all purpose) created if not yet created, follow databrick's documentation for creating a sql warehouse, https //docs databricks com/en/compute/sql warehouse/create html https //docs databricks com/en/compute/sql warehouse/create html dlh io does not currently support direct unity catalog target destination, only sql warehouse target loading for permissions for oauth assign at a minimum the can use permission create a specific new user in databricks for dlh io (recommended) / m2m configuration, or confirm an existing user is available with necessary permissions to create a new user in databricks log in to your databricks account open the workspace desired create a new user and assign the appropriate permissions create a pat for the user following instructions here, https //docs databricks com/en/dev tools/auth/pat html#databricks personal access tokens for workspace users https //docs databricks com/en/dev tools/auth/pat html#databricks personal access tokens for workspace users , because personal access tokens are user specific if using a type of "service account" user then you will need to log in as this user and follow the instructions developer > access tokens > click "manage"> generate new token > click "generate" set permissions in the default (or selected (optional)) catalog update the security permissions for the service provider/account or pat user to have permissions to the catalog click on the catalog, then click permissions > click the grant button complete the form and finalize for the service provider/account or user by clicking the grant button instructions (basic connection setup) per the prerequisites, capture all the necessary details about your databricks account and prepare to use them for creating your dlh io databricks connection in dlh io from the header or sidebar chose targets search for, or directly locate, and select databricks from the list of connectors a new target connection form will appear for you to enter details about your account/workspace that will be used for the data integration process on the dlh io connection form enter your connection information enter in the name/alias field, the name you'll use within dlh io to differentiate this connection from others enter in the server hostname field, the name of the endpoint server name use the endpoint as provided in your workspace, such as the server hostname which is similar to for example, ddd asdfasw3 as00 cloud databricks com, that is unique to your workspace (optional) enter in the catalog field, the name of the existing unity catalog you wish to load data into by default dlh io will create a new catalog and load your respective source data into it, thus it is important to provide authentication credentials with the necessary permissions this option to specify an existing unity catalog for business critical plans only at this time the unity catalog feature must be enabled in the workspace for this option to work correctly enter in the port field, where this account is accessible and the firewall/network rules allow dlh io (see prerequisites for ip grantlisting) for this connector the default port is 443, which is standard but we have it listed here for future proofing only change this if suggested by support or you have an advanced implemented or knowledge of your account's requirements enter in the http path field, enter the http path provided in your workspace connection area for either your compute cluster or the sql warehouse in the case of a compute cluster (or legacy) you will enter for example sql/protocolv1/o/\<orgid>/\<clusterid> or sql/1 0/endpoints/\<clusterid>, and for a sql warehouse option enter for example or sql/1 0/warehouses/\<warehouseid> enter in the user or client id field, the username of the user account or service account you created (see prerequsites) that will authenticate your connection in dlh io if using a personal access token or username/password (community edition only) authentication option, then this field will be the user's email address usually if using the oauth authentication type (i e m2m, etc ) then this field will be seen as the client id field select auth type required from the dropdown use the personal access token (pat) option for legacy account environment or quick testing as development if not having administrative permissions to create the oauth credentials databricks has mentioned that in future this option may be deprecated use the oauth 2 0 option (recommended) for current account environment versions and newer configurations as personal access tokens may be deprecated in the near future use the username/password option only for the databricks community edition this option may be removed at anytime from dlh io, so it is recommended to use one of the other authentication options based on your auth type selection, the correct credential field will appear enter in the password field, your personal access token (pat) or username/password option, for the user for which you are connecting to your account if selecting oauth 2 0 then the client id and client secret fields will appear an enter the appropriate credentials and information in the correct fields click on save & test to save the connection and test that we can connect if updating the form click save & test or just test clicking on save & test will again save any changes such as the password change, etc any test of the connection will attempt to connect to your database/account with the credentials and info provided a message of success or failure will be shown if success you'll be prompted with the schema objects objects of the database and will need to complete the final steps for configuration shown below if failure happens with the test connection, the connection is still saved but you will need to correct the failure based on the failure reason information provided in the message issue handling if any issues occur with the authorization simply return to the sources page in datalakehouse io, edit the source details and click the save & test or authorize your account or re authorize account button to confirm connectivity if any issues persist please contact our support team via the datalakehouse io support portal https //datalakehouse zendesk com creating an oauth m2m credential (id + secret) before you can use oauth to authenticate to databricks, you must first create an oauth secret, which can be used to generate oauth access tokens a service principal can have up to five oauth secrets account admins and workspace admins can create an oauth secret for a service principal the secret will only be revealed once during creation the client id is the same as the service principal’s application id dlh io will only use your credentials for access the workspace, focused on creating only the necessary objects required for the data synchronization process, otherwise read only schema or data references, not to conduct operations at the account level to enable the service principal to use clusters or sql warehouses, you must give the service principal access to them see compute permissions https //docs databricks com/en/compute/clusters manage html#cluster level permissions or manage a sql warehouse https //docs databricks com/en/compute/sql warehouse/create html#manage for a sql warehouse, provide the can use permission at a minimum to understand the security acl, visit this link, https //docs databricks com/en/security/auth/access control/index html#sql warehouses https //docs databricks com/en/security/auth/access control/index html#sql warehouses for a compute / cluster add minimum permission of can restart the can restart permission is necessary if your cluster terminates (auto shutsdown, etc ) and you need for the user or service provider account to start up the computer when dlh io sync bridges initiate the synchronization process if your compute is up most of the time and you wish to have a slightly lower security permission you can use the can attach to permision, however, dlh io will not be able to re/start the compute if it is down or terminated for any reason failure to provide the correct permission level may result in the following error permission denied you do not have permission to autostart or permission denied user does not have use schema on schema