Object storage

Object-storage destinations land events as files in a bucket. Pushrail buffers events into batches, writes each batch as newline-delimited JSON (one event per line), and uploads to a key that includes a partition prefix you choose. This is the right destination for a data lake, an archive, or any downstream that expects file-based ingest.

When to use

Pick object storage when the consumer is a batch system: a warehouse loader, a Spark job, a daily report, a long-term archive. The latency is minutes-to-hours, not seconds, events flush on a size or time threshold, whichever comes first.

If the consumer is interactive (a dashboard, a real-time UI) or needs per-event delivery semantics, prefer a warehouse destination (for queryable storage) or a webhook (for low-latency push).

Auth options

Both supported types use cloud-native credential schemes.

• S3, IAM access key + secret, assume-role from a Pushrail-managed cross-account role, or a one-click CloudFormation "automated" setup that creates the role for you (enter your AWS account ID, launch the stack, and Pushrail detects the role). Use a role-based option in production; rotate access keys never.
• GCS, grant our service account access with an IAM binding (recommended, nothing tied to a personal account), or upload a service-account JSON key. Keys are stored encrypted and only a fingerprint is exposed thereafter. (Google sign-in / OAuth is no longer offered for new destinations; existing OAuth connections keep working.)

Credentials need write-only access to the configured prefix, Pushrail never lists, reads, or deletes existing objects. The least-privilege IAM policy is s3:PutObject on arn:aws:s3:::your-bucket/your-prefix/* (S3) or roles/storage.objectCreator on the bucket (GCS).

Config reference

S3

Required: bucket, region, prefix, auth. Optional: partitionFormat, flushIntervalSec, maxBatchSize, compression.

{
  "type": "s3",
  "bucket": "acme-pushrail-events",
  "region": "us-east-1",
  "prefix": "events/",
  "partitionFormat": "{eventType}/dt={yyyy}-{MM}-{dd}/h={HH}/",
  "flushIntervalSec": 300,
  "maxBatchSize": 10000,
  "compression": "gzip",
  "auth": { "type": "iam_role", "roleArn": "arn:aws:iam::123:role/pushrail-write" }
}

The default partition format produces Hive-style prefixes that BigQuery, Athena, and Snowflake can discover automatically.

GCS

Same shape as S3, minus the region (GCS bucket region is part of the bucket config). Required: bucket, prefix, auth.

{
  "type": "gcs",
  "bucket": "acme-pushrail-events",
  "prefix": "events/",
  "partitionFormat": "{eventType}/dt={yyyy}-{MM}-{dd}/",
  "compression": "gzip",
  "auth": { "type": "service_account_json", "credentials": "{{secret}}" }
}

Common patterns

Partition by event type and day: the default partition format already does this. Downstream Athena/BigQuery/Snowflake jobs can prune to a single day's data without scanning the whole prefix.

One bucket per environment: keep Development, Staging, and Production buckets separate. The destination config in each environment points at its own bucket, never share buckets across environments.

Compression: gzip cuts storage cost by 5-10× on JSON. Enable it unless your downstream loader can't handle gzipped NDJSON (most can).

Verifying delivery

A delivery to an object-storage destination is "this batch of events was written to this key." The delivery log shows the bucket, the full object key, the byte size, the event count in the batch, and the upload latency. A failed delivery shows the cloud-provider error (permission denied, throttled, etc.) and the retry schedule.

If you don't see the file in the bucket, check IAM first, the most common failure mode is a misconfigured role or a prefix the credentials don't have write access to.