Build your first pipeline
Move data from source to destination on a schedule.
A pipeline moves data from one place to another. You pick a source connector and a target connector, choose which tables or datasets to sync, set how each one is loaded, and (optionally) put it on a schedule. Databasin handles the reading, the batching, the retries, and the monitoring.
This guide takes you through your first one, end to end.
Before you start
You need two connectors already saved — one for the source and one for the target. If you only have one so far, pause and build another. Common pairings:
- Prod Postgres → Lakehouse (classic analytics sync)
- Salesforce → Lakehouse (SaaS into your warehouse)
- S3 CSV drops → Lakehouse (ingesting vendor files)
Step 1 — Start from "Create integration"
Pipelines live in the Integrations hub. Open Integrations and click Create integration (top right). The wizard opens on "What do you want to do?" with three cards:
- Sync data — copy data into your lakehouse on a schedule. This is the pipeline flow — pick this one.
- Automate a task — a multi-step workflow (a different tool; see Pipelines vs. Automations).
- Set up a live connection (Beta) — query the source directly instead of copying it. See Live connections.
Step 2 — "What are we connecting?"
Sync data drops you straight onto the guided pipeline screen. The first section asks for three things together:
- Name — how you'll recognize this pipeline (e.g. "Marketing Sync").
- Source — where the data is read from.
- Destination — where the data lands.
Both pickers list the connectors already in your project, and both let you create a new connection inline without leaving the wizard.
Step 3 — "How should your data be organized?"
Next you pick the shape of the landing zone. Two cards:
| Pattern | What it does | Good for |
|---|---|---|
| Datalake | Medallion architecture — bronze, silver, and gold layers, with the transformations between them handled for you. | Analytics and ML. Lakehouse / Trino targets. |
| Data Warehouse | Direct loading — data lands straight into a catalog, schemas mirrored from the source. | Fastest to query, familiar SQL workflows. Any warehouse target. |
Datalake is only selectable when the target supports it; otherwise the card is disabled. Pick Datalake and you name the namespace your tables land under; pick Data Warehouse and you name the catalog.
If the destination is your Databasin lakehouse and you're going to point Databasin One or dashboards at this data, take Datalake. Data Warehouse is the right call when you're loading into an existing warehouse whose structure you already own.
Step 4 — "What to ingest" (native connectors)
If your source is a Databasin Native connector, you get a section that a generic source doesn't — two checkboxes, both on by default:
- Include all certified routes — ingest every route the connector exposes. Elevated-permission routes are off by default. Uncheck to pick routes by hand on the next screen instead.
- Include built-in semantic model — auto-publish the connector's bundled Gold views so they show up in Databasin One after the first run. Uncheck to skip the semantic layer.
Expand the preview below them and you'll see exactly what you're agreeing to: the certified routes (with a count, and an "Elevated · off by default" badge on the ones excluded) and the Gold views that will be queryable in Databasin One.
Some routes need a value before they can run — a board id, an account id. Those appear under Required configuration on the same screen.
Leaving both boxes checked is how a native connector goes from "connected" to "Databasin One can answer questions about it" in one pass. See Native connectors in pipelines.
For a non-native source, you pick what to sync directly instead: narrow to a catalog / schema, toggle the tables you want, and keep or trim columns.
Step 5 — Set the ingestion mode
For each table, you choose how it's loaded. There are five modes:
| Mode | What it does |
|---|---|
| Snapshot | Full refresh — replaces the target table on every run. Best for small reference tables. |
| Delta | Incremental — adds new and changed rows each run using a watermark column. |
| Historical | Append-only — each run adds new rows; existing rows are never touched. |
| CDC | Change data capture — reads inserts, updates, and deletes from the source's log. |
| Stored Procedure | Calls a stored procedure on the source to produce the extract. |
Snapshot and Delta cover most jobs. For the full picture of when to reach for each, see Ingestion modes.
The pipeline wizard lands your data faithfully — there's no rename / cast / filter / join stage inside it. To reshape, model, or join data, that's an Automations job. Databasin even points you there: "Use Databasin Automations to model and transform data."
Step 6 — "When should it run?"
The last section sets the schedule and who hears about it:
- Manual only — runs when you click Run now or an automation triggers it.
- Presets — common cadences like hourly, daily, or weekly, one click each.
- Custom cron — a cron expression when the presets don't fit.
- Email notifications — addresses that get the run result. You can add these later too.
Advanced options on the same screen cover table naming (the default uses the source table name as-is) and the compute the run uses.
See Scheduling and triggers for the details. Save, and your pipeline is live.
Step 7 — Run it and watch
Open the pipeline and click Run now. The detail page shows run status, run history, and per-run details; if a run fails you'll get a notification on the bell in the top bar. Monitoring and alerts covers what to watch.
A Delta pipeline merges on the keys you configured, so re-running a completed run won't create duplicates. A Snapshot pipeline replaces the whole table each run, so it's dup-safe by construction.