Skip to main content
ETL Pipelines integration is available for the Scale plan.
With ETL Pipelines, Junction streams all Junction events continuously to a supported destination over direct authenticated TLS connections. Compared to Webhooks, this simplifies your operational complexity by removing the need to operate public unauthenticated HTTPS endpoints for receiving incoming Webhooks.

Scalable

Offload the pressure on your public HTTP services.

Secure

Events are published through authenticated channels over TLS.

Compressed

Data events larger than 1 KiB are compressed before publication.

Ordered

Publication order can be preserved for select destination types.

​
Destinations

The following destinations are supported:

Google Cloud Pub/Sub

RabbitMQ

Azure Event Hubs

​
Event Schema

Webhooks, ETL Pipelines and our API endpoints all share the same JSON object schema. Webhooks and ETL Pipelines use identical JSON event payload structure. Check out our Event Catalog.

​
Features

​
Data compression

Junction does not compress payload blobs that are smaller than 1 KiB. Please refer to the destination-specific documentation below on how to detect a compressed blob versus an uncompressed blob.

​
Static outbound IPs

Junction establish connections to your ETL Pipeline destination through a static pool of IPs. Contact Junction support to obtain the up-to-date static IP list.

​
Double writing for migration

To ensure smooth switchover from Webhooks to your new ETL Pipelines destination, Junction supports double writing events to the combination of:
  • all your Webhook endpoints; and
  • your ETL Pipelines destination.
This is not a proper fan-out feature, and is only intended to aid customers evaluating and one-off migrating between destinations.There can only be one preferred destination, and the secondary destination(s) do not enjoy the same reliability guarantee as the preferred one.

​
Pushed Historical Data

When you use ETL Pipelines, Junction can push all the historical data as daily.data.* data events to your ETL Pipelines destination. The data-less historical.data.*.created event would still be emitted to denote the completion of historical pull.

​
Configuring ETL Pipelines

You can change the ETL Pipelines configuration on your Teams at any time through the Set Team ETL Pipelines endpoint of the Org Management API. Your new configuration is typically active as soon as the endpoint has acknowledged your request. Refer to the destination-specific documentation for configuration examples.
Feel free to reach out Junction support if you need assisted setup, or if you need a Junction Management Key to access the Org Management API.