website logo
⌘K
Getting Started 🚀
What is DataLakeHouse.io?
Our Business-Value Focus
Learn the Basic Concepts
Connectors
Operations Applications
Toast
Asana
Aloha POS
BILL
Bloom Growth
Bullhorn
Ceridian Dayforce
ConnectWise
Facebook Ads
Food Delivery Service Connector
Google Analytics 4
Harvest
Hubspot
Jira
MailChimp
McLeod Transportation
NetSuite (Oracle NetSuite)
Optimum HRIS
Oracle EBS
Oracle PeopleSoft
QuickBooks Online
Salesforce
Shopify
Square
Square Marketplace
Stripe
TriNet
Verizon Wireless Business
Workday HCM
Xero
Zoom
Databases
Files & Object Storage
SSH Tunnel Setup for Hosted Database Systems
Databases FAQ
SQL Transformations
Terraform: Reverse Terraforming
DBT Cloud Transformations
Sync Bridge (Data Pipelines)
Create a Sync Bridge
Manually Run a Sync Bridge
Deleting a Sync Bridge
Historical Re-sync
Analytics
Access Analytics
Snowflake Usage Analytics
Data Catalog
Create the Catalog
Populate the Catalog
Access the Catalog
Data Warehouse Clouds
Snowflake
Open Source DW Models
Alerts & Notifications
Slack Notifications
Untitled doc
Logs & Monitoring
Security
Callback Links
Service Level Agreement (SLA)
Release Notes
May 2023
April 2023
Q3 2022
Q4 2022
Community Overview
Contributor Agreements
Code Contribution Guide
About
License
Viewpoint
Docs powered by archbee 

What is ELT?

2min

The move from building ETL pipelines (where much of the transformation is carried out in tools like Spark or Informatica before the data is loaded into Snowflake) to ELT pipelines (where the transformation is carried out within Snowflake itself). The reasons are that (1) SQL is easier for business users to write (2) Snowflake scales better and is less expensive than alternative data processing technologies.

The problem with doing all the transformation code in SQL, though, is that it can become hard to maintain. How often have you come back to a project after a few months and been faced with a bunch of views, tables, user-defined functions, and scripts and scratched your head in confusion?

That’s why it’s very useful to have a environment that supports best practices in terms of transformation code — the same sort of best practices you want to apply to any code: documentation, reusability, readability, assertions, unit testing, source code control, and so on.

dbt and DataForm are our main tools for incorporating transformation within DataLakeHouse forming the modern data stack engine

ELT brings a software engineering approach to data modeling and pipelines making data transformations more accessible and reliable:

  1. Collaborate and create data pipelines—Develop data workflows in SQL and collaborate with others via Git. Include data documentation that is automatically visible to others.
  2. Deploy data pipelines—Keep logical data up-to-date by scheduling data workflows which incrementally update downstream datasets, reducing cost and latency.
  3. Ensure data quality—Define data quality checks in SQL and automatically receive alerts when those checks fail. View logs, version history and dependency graphs to understand changes in data.



Updated 27 Apr 2023
Did this page help you?
Yes
No
PREVIOUS
Learn the Basic Concepts
NEXT
What is dbt?
Docs powered by archbee