Pushrail Docs
Open app
Guides · Destinations

Data warehouses

Land events directly in BigQuery, Snowflake, or ClickHouse with schema-aware loaders.

Data warehouses

Warehouse destinations write events as rows in a target table. Pushrail handles the per-warehouse loader semantics (streaming inserts on BigQuery, Snowpipe on Snowflake, native HTTP inserts on ClickHouse) and presents the same configuration surface across all three. The result is queryable in your warehouse within minutes of ingest, with no glue code on your side.

When to use

Warehouse destinations are the right choice when the consumer is analytics: a dashboard, a notebook, a BI tool. They handle high event volume cheaply and let your data team query events with the same tools they use for everything else.

If the consumer is a service that needs sub-second access, prefer a database destination or a webhook. Warehouses are optimized for "scan a lot of rows" rather than "look up one row."

Auth options

  • BigQuery, grant our service account bigquery.dataEditor on the dataset with an IAM binding (recommended), or upload a service-account JSON key. (Google sign-in / OAuth is no longer offered for new destinations; existing OAuth connections keep working.)
  • Snowflake, key-pair or username/password. Key-pair is recommended for production; password is fine for evaluation.
  • ClickHouse, username/password against the HTTP interface, or a TLS client certificate for ClickHouse Cloud.

In all cases the credential is encrypted at rest, masked in the dashboard, and rotatable with a 24-hour grace window.

Config reference

BigQuery

Required: projectId, datasetId, tableId, auth. The target table must already exist; Pushrail does not create or evolve schemas (intentionally, schema changes are your DDL workflow, not ours).

{
  "type": "bigquery",
  "projectId": "acme-prod",
  "datasetId": "pushrail",
  "tableId": "events",
  "auth": { "type": "service_account_json", "credentials": "{{secret}}" }
}

Pushrail uses BigQuery's storage write API for low-latency inserts. Failed inserts retry with backoff; rows that fail validation against the target schema are surfaced in the delivery log with the BigQuery error message.

Snowflake

Required: account, warehouse, database, schema, table, auth.

{
  "type": "snowflake",
  "account": "acme-prod.us-east-1",
  "warehouse": "PUSHRAIL_WH",
  "database": "ANALYTICS",
  "schema": "PUSHRAIL",
  "table": "EVENTS",
  "auth": { "type": "key_pair", "username": "PUSHRAIL_LOADER", "privateKey": "{{secret}}" }
}

Loads use Snowpipe streaming for near-real-time visibility. The configured warehouse only spins up when there are events to load and auto-suspends after the configured idle window, cost is proportional to event volume, not destination existence.

ClickHouse

Required: host, database, table, auth. Optional: port, secure (TLS, default true).

{
  "type": "clickhouse",
  "host": "acme.clickhouse.cloud",
  "database": "events",
  "table": "pushrail_events",
  "secure": true,
  "auth": { "type": "password", "username": "pushrail", "password": "{{secret}}" }
}

Pushrail buffers events client-side and inserts in batches of up to 10,000 rows or 30 seconds, whichever comes first.

Common patterns

Flatten the payload with a transform: warehouses query best on flat schemas. Use a declarative transform to lift payload.orderId into a top-level order_id column rather than a JSON_VALUE(payload, '$.orderId') lookup in every query.

One table per major event family: orders_events, subscriptions_events, etc. Route with routing rules so each table only gets the event types it expects. This keeps schemas tight and queries fast.

Late-arriving data: the warehouse table should partition on occurredAt, not on the load time. Replays land at load time but carry their original occurredAt, so partition-by-event-time gives you the right semantics for backfills.

Verifying delivery

The delivery log shows the row count loaded, the load latency, and any per-row errors the warehouse returned. The destination card surfaces success rate, average rows per load, and load lag (the gap between ingest and warehouse availability).

A spike in load lag usually means the warehouse compute is saturated, for Snowflake, increase the warehouse size or add a multi-cluster warehouse; for BigQuery, the storage write API has no compute knob, so contact us if the lag is unacceptable.