Pushrail Docs
Open app
Guides · Destinations

Queues & streams

Publish events to Kafka, Kinesis, Pub/Sub, SNS, and SQS for consumer-paced delivery.

Queues & streams

Queue and stream destinations hand events off to a managed messaging system. Consumers read at their own pace, fan out across partitions, and replay history natively. This is the destination type for high-throughput pipelines and any consumer that needs ordering or back-pressure semantics that an HTTP webhook cannot provide.

When to use

Choose a queue or stream when the receiver is a service that wants control over consumption: a Flink job, a Lambda fanned out from SQS, a Kafka Streams app, a Pub/Sub push subscription. The producer (Pushrail) commits the event into the queue and moves on; the consumer scales independently.

For a single HTTP endpoint that processes events synchronously, a webhook is simpler. For analytics, a warehouse is cheaper. Queues shine for "this is a real pipeline with real consumers."

Auth options

  • Kafka, SASL/PLAIN, SASL/SCRAM, mTLS, or AWS MSK IAM.
  • Kinesis, IAM access key + secret, or cross-account role assumption.
  • Pub/Sub, grant our service account roles/pubsub.publisher on the topic with an IAM binding (recommended), or upload a service-account JSON key. (Google sign-in / OAuth is no longer offered for new destinations; existing OAuth connections keep working.)
  • SNS, IAM with sns:Publish on the topic ARN.
  • SQS, IAM with sqs:SendMessage on the queue ARN.

For all AWS-backed types, prefer cross-account role assumption over long-lived access keys.

Config reference

Kafka

Required: bootstrapServers, topic, auth. Optional: partitioner, compression (none, gzip, snappy, lz4, zstd), acks (all default).

{
  "type": "kafka",
  "bootstrapServers": "broker1.acme.example:9092,broker2.acme.example:9092",
  "topic": "pushrail.events",
  "partitioner": "customer_id",
  "compression": "lz4",
  "acks": "all",
  "auth": { "type": "sasl_scram", "username": "pushrail", "password": "{{secret}}" }
}

The customer_id partitioner hashes customerExternalId to a partition so all events for one customer stay in order on a single partition. Use round_robin if you don't need per-customer ordering.

Kinesis

Required: streamName, region, auth. Optional: partitionKeyField (default customerExternalId).

{
  "type": "kinesis",
  "streamName": "pushrail-events",
  "region": "us-east-1",
  "partitionKeyField": "customerExternalId",
  "auth": { "type": "iam_role", "roleArn": "arn:aws:iam::123:role/pushrail-kinesis" }
}

Pub/Sub

Required: projectId, topicId, auth. Optional: orderingKeyField (when set, enables ordered delivery on Pub/Sub).

{
  "type": "pubsub",
  "projectId": "acme-prod",
  "topicId": "pushrail-events",
  "orderingKeyField": "customerExternalId",
  "auth": { "type": "service_account_json", "credentials": "{{secret}}" }
}

SNS

Required: topicArn, region, auth. The message attribute eventType is set automatically so SNS subscription filter policies can route on it.

{
  "type": "sns",
  "topicArn": "arn:aws:sns:us-east-1:123:pushrail-events",
  "region": "us-east-1",
  "auth": { "type": "iam_role", "roleArn": "arn:aws:iam::123:role/pushrail-sns" }
}

SQS

Required: queueUrl, region, auth. Optional: messageGroupIdField (required for FIFO queues).

{
  "type": "sqs",
  "queueUrl": "https://sqs.us-east-1.amazonaws.com/123/pushrail-events",
  "region": "us-east-1",
  "messageGroupIdField": "customerExternalId",
  "auth": { "type": "iam_role", "roleArn": "arn:aws:iam::123:role/pushrail-sqs" }
}

Common patterns

Partition by customer for ordering: all stream types support a per-customer partition key. Set partitionKeyField (Kinesis), orderingKeyField (Pub/Sub), messageGroupIdField (SQS FIFO), or partitioner: customer_id (Kafka) so events for one customer arrive in order on the consumer side.

Use a transform to set message attributes: SNS and Kafka headers can carry the eventType and customerExternalId outside the body so consumers can filter without parsing JSON.

Tune compression: for Kafka and Kinesis, lz4 or zstd compression typically cuts wire bytes by 4-6× on JSON payloads with negligible CPU impact.

Verifying delivery

A delivery to a queue is "this event was acknowledged by the broker." The delivery log records the broker-side message id (Kafka offset, Kinesis sequence number, Pub/Sub message id, etc), the partition/shard it landed on, and the publish latency.

A successful publish does not prove the consumer processed the event, that's the consumer's responsibility. Use the broker's native consumer-lag metrics to monitor consumption; Pushrail only owns the producer side.