website logo
⌘K
Getting Started 🚀
What is DataLakeHouse.io?
Our Business-Value Focus
Learn the Basic Concepts
Connectors
Operations Applications
Toast
Asana
Aloha POS
BILL
Bloom Growth
Bullhorn
Ceridian Dayforce
ConnectWise
Facebook Ads
Food Delivery Service Connector
Google Analytics 4
Harvest
Hubspot
Jira
MailChimp
McLeod Transportation
NetSuite (Oracle NetSuite)
Optimum HRIS
Oracle EBS
Oracle PeopleSoft
QuickBooks Online
Salesforce
Shopify
Square
Square Marketplace
Stripe
TriNet
Verizon Wireless Business
Workday HCM
Xero
Zoom
Databases
Files & Object Storage
SSH Tunnel Setup for Hosted Database Systems
Databases FAQ
SQL Transformations
Terraform: Reverse Terraforming
DBT Cloud Transformations
Sync Bridge (Data Pipelines)
Create a Sync Bridge
Manually Run a Sync Bridge
Deleting a Sync Bridge
Historical Re-sync
Analytics
Access Analytics
Snowflake Usage Analytics
Data Catalog
Create the Catalog
Populate the Catalog
Access the Catalog
Data Warehouse Clouds
Snowflake
Open Source DW Models
Alerts & Notifications
Slack Notifications
Untitled doc
Logs & Monitoring
Security
Callback Links
Service Level Agreement (SLA)
Release Notes
May 2023
April 2023
Q3 2022
Q4 2022
Community Overview
Contributor Agreements
Code Contribution Guide
About
License
Viewpoint
Docs powered by archbee 

Q4 2022

4min

Our July release gives more options, more features, and improved speed of synchronization for our popular connectors:

PostgreSQL : Custom Logical Replication Slots

Previously DLH.io allowed for only one general replication slot to be created per any database and it had to be a specific name, datalakehouseio_replication_slot. While we still recommend this name for your slot when creating your database, we now allow for any custom name of a replication slot. You will simply add the name of the slot when you create your source connection to PostgreSQL. You will not be able to change it later for obvious reasons for that source connection but this added flexibility is a great edition for teams with DBAs with strong opinions on naming conventions.

We still recommend that if you need to create multiple replication slots for example to capture data changes only on specific schemas and not the entire database, that you use our recommended naming convention, datalakehouseio_replication_slot, and use a short suffix to differentiate the slots, for example, datalakehouseio_replication_slot_schema1, or datalakehouseio_replication_slot_mytbl.

Ultimately this enables slots to not be read or cleared out for a situation where a database has multiple schemas and each schema should be synchronized on a different frequency with different parameters. By creating multiple source connections to the same database with different replication slots, you will deselect the schemas for each connection you wish not to synchronize and the replication slot will be respected for that synchronization of that source. We've seen the most success for this use case with Aiven.io PostgreSQL customers, Hooray!!!

PostgreSQL : Replication Slot Auto-ReCreate

The PostgresSQL database is notorious for dropping the replication slot upon a database update/refresh and we've seen this happen with a number of customers who do not have large replicated instances and other means of maintaining the state of their slots. DLH.io will now auto-recreate the replication slot for you in the database you use in your DLH.io Postgres Source Connection if it detects that it was removed or no longer available. Permissions to do so will obviously need to be part of the privileges granted to the user credentials you use in the source connection.

New Options for Logical Replication
New Options for Logical Replication


To learn more about other ways the Postgres could retain your slot states you could reference some documentation from PostgreSQL:

  • https://www.postgresql.org/docs/current/logical-replication-subscription.html
  • https://www.postgresql.org/docs/current/logical-replication-restrictions.html
  • https://wiki.postgresql.org/wiki/Logical_replication_and_physical_standby_failover





Updated 15 May 2023
Did this page help you?
Yes
No
PREVIOUS
Q3 2022
NEXT
Community Overview
Docs powered by archbee 
TABLE OF CONTENTS
PostgreSQL : Custom Logical Replication Slots
PostgreSQL : Replication Slot Auto-ReCreate