> ## Documentation Index
> Fetch the complete documentation index at: https://docs.junction.com/llms.txt
> Use this file to discover all available pages before exploring further.

# ETL Pipelines

> Use ETL Pipelines to stream Junction events to supported destinations over authenticated TLS as a scalable alternative to webhooks.

<Info>
  ETL Pipelines integration is available for [the Scale plan](https://tryvital.io/pricing).
</Info>

With ETL Pipelines, Junction streams all [Junction events](/webhooks/event-structure) continuously to a supported destination
over direct authenticated TLS connections. Compared to [Webhooks](/webhooks/introduction), this simplifies your operational
complexity by removing the need to operate public unauthenticated HTTPS endpoints for receiving incoming Webhooks.

<CardGroup cols={2}>
  <Card title="Scalable" icon="arrow-up-right-dots">
    Offload the pressure on your public HTTP services.
  </Card>

  <Card title="Secure" icon="shield">
    Events are published through authenticated channels over TLS.
  </Card>

  <Card title="Compressed" icon="gift">
    Data events larger than 1 KiB are compressed before publication.
  </Card>

  <Card title="Ordered" icon="traffic-cone">
    Publication order can be preserved for select destination types.
  </Card>
</CardGroup>

## Destinations

The following destinations are supported:

<Card horizontal href="/webhooks/etl-pipelines/google-cloud-pubsub" title="Google Cloud Pub/Sub" icon="google" />

<Card horizontal href="/webhooks/etl-pipelines/rabbitmq" title="RabbitMQ" icon="rabbit" />

<Card horizontal href="/webhooks/etl-pipelines/azure-event-hubs" title="Azure Event Hubs" icon="microsoft" />

## Event Schema

Webhooks, ETL Pipelines and our API endpoints all share the same JSON object schema. Webhooks and ETL Pipelines use
identical JSON event payload structure.

Check out our [Event Catalog](/event-catalog).

## Features

### Data compression

Junction does not compress payload blobs that are smaller than 1 KiB. Please refer to the destination-specific
documentation on how to detect a compressed blob versus an uncompressed blob.

### Static outbound IPs

Junction establishes connections to your ETL Pipeline destination through a static pool of IPs. Contact Junction support to
obtain the up-to-date static IP list.

### Double writing for migration

To ensure smooth switchover from Webhooks to your new ETL Pipelines destination, Junction supports double writing events
to the combination of:

* all your Webhook endpoints; and
* your ETL Pipelines destination.

<Info>
  This is not a proper fan-out feature, and is only intended to aid customers in evaluating and one-off migrating
  between destinations.

  There can only be one *preferred* destination, and the secondary destination(s) do not enjoy the same reliability
  guarantee as the *preferred* one.
</Info>

### Pushed Historical Data

When you use ETL Pipelines, Junction can *push* all the historical data as `daily.data.*` data events to your ETL Pipelines
destination.
The data-less `historical.data.*.created` event would still be emitted to denote the completion of historical pull.

## Configuring ETL Pipelines

You can change the ETL Pipelines configuration on your Teams at any time through the
[Set Team ETL Pipelines](/api-reference/org-management/team-etl-pipeline/upsert-team-etl-pipelines) endpoint
of the Org Management API.

Your new configuration is typically active as soon as the endpoint has acknowledged your request.

Refer to the destination-specific documentation for configuration examples.

<Info>
  Feel free to reach out to Junction support if you need assisted setup, or if you need a Junction Management Key to access the Org Management API.
</Info>