Stream Backup

RabbitMQ Streams (x-queue-type: stream) are append-only log structures that support non-destructive reads natively. rabbitmq-backup uses the RabbitMQ Stream Protocol for efficient, offset-based backup of stream-type queues.

How It Differs from Classic Queue Backup

Feature	Classic/Quorum Queues	Streams
Protocol	AMQP 0-9-1	RabbitMQ Stream Protocol
Port	5672	5552
Read method	Consume + requeue	Offset-based read (inherently non-destructive)
Resume support	SQLite checkpoint	Stream offset tracking
Ordering	No guarantee after requeue	Guaranteed (append-only log)

Prerequisites

Enable the Stream Plugin

The RabbitMQ broker must have the Stream plugin enabled:

rabbitmq-plugins enable rabbitmq_stream

For the Management UI integration:

rabbitmq-plugins enable rabbitmq_stream_management

Verify the plugin is active:

rabbitmq-plugins list | grep stream

Expected output:

[E*] rabbitmq_stream                  4.0.0
[E*] rabbitmq_stream_management       4.0.0

Verify Stream Port

The Stream Protocol listens on port 5552 by default. Verify it is accessible:

# Test connectivity
nc -zv rabbitmq.example.com 5552

Step 1: Create a Stream Queue

If you do not already have stream queues, create one:

# Via rabbitmqadmin
rabbitmqadmin declare queue name=events-stream \
  durable=true \
  arguments='{"x-queue-type":"stream"}'

# Via Management API
curl -u guest:guest -X PUT \
  http://localhost:15672/api/queues/%2f/events-stream \
  -H "content-type: application/json" \
  -d '{"durable":true,"arguments":{"x-queue-type":"stream"}}'

Step 2: Configure the Backup

Enable stream support and filter for stream-type queues:

backup-streams.yaml

mode: backup
backup_id: "stream-backup-001"

source:
  amqp_url: "amqp://guest:guest@localhost:5672/%2f"
  management_url: "http://localhost:15672"
  management_username: guest
  management_password: guest
  stream_port: 5552

  queues:
    include:
      - "events-*"
    types:
      - stream

storage:
  backend: filesystem
  path: /var/lib/rabbitmq-backup/data

backup:
  compression: zstd
  compression_level: 3
  max_concurrent_queues: 4
  include_definitions: true
  stop_at_current_depth: true
  stream_enabled: true

Key settings:

• stream_port: The Stream Protocol port (default: 5552)
• queues.types: Filter to only stream queue types
• stream_enabled: Must be true to activate the Stream Protocol client

Step 3: Run the Backup

rabbitmq-backup backup --config backup-streams.yaml

With debug logging to see offset progress:

rabbitmq-backup backup -v --config backup-streams.yaml

Expected output:

INFO  Starting backup stream-backup-001
INFO  Connected to RabbitMQ at localhost:5672
INFO  Stream protocol connected on port 5552
INFO  Discovered 2 stream queues matching criteria
INFO  Backing up stream events-orders (offset 0 -> 15,000)
INFO  Backing up stream events-payments (offset 0 -> 8,200)
INFO  Segment segment-0001.zst written (15,000 records)
INFO  Segment segment-0001.zst written (8,200 records)
INFO  Backup stream-backup-001 completed: 23,200 messages

Step 4: Resumable Backups

Stream backups track completion and stream offsets in SQLite. On subsequent runs with the same backup_id, completed stream segments are verified and reused; partial stream backups resume from the next stored offset:

offset_storage:
  backend: sqlite
  db_path: ./stream-offsets.db
  s3_key: stream-backup-001/state/offsets.db
  sync_interval_secs: 30

Run the backup again with the same backup_id:

rabbitmq-backup backup --config backup-streams.yaml

INFO  Skipping already-completed stream //events-orders from checkpoint
INFO  Resuming stream //events-payments from offset 8201

If the previous run completed and all segment files still exist, the manifest is rebuilt from existing segment metadata without rewriting the segment files.

Mixed Queue Types

You can back up both classic/quorum queues and streams in a single run. The tool automatically selects the correct protocol per queue type:

source:
  queues:
    include:
      - "*"
    types:
      - classic
      - quorum
      - stream

backup:
  stream_enabled: true
  requeue_strategy: cancel  # applies to classic/quorum only

• Classic and quorum queues use AMQP 0-9-1 with the configured requeue_strategy
• Stream queues use the Stream Protocol with offset-based reads

Stream-Specific Considerations

Large Streams

Streams can contain millions of messages. For the initial backup of a large stream:

• Increase segment_max_bytes to reduce the number of segment files
• Run with stop_at_current_depth: true to avoid chasing the tail of an active stream

backup:
  segment_max_bytes: 536870912  # 512 MB
  stop_at_current_depth: true
  stream_enabled: true

Stream Retention

Streams have their own retention policies (time-based, size-based). Back up streams before their retention window expires if you need historical data beyond the broker's retention.

TLS for Streams

When TLS is enabled, the Stream Protocol connects on the TLS stream port (default: 5551):

source:
  stream_port: 5551
  tls:
    enabled: true
    ca_cert: /certs/ca.pem

Restore Considerations

Stream-backed-up messages are restored via AMQP 0-9-1 basic.publish (not the Stream Protocol). The restore publishes messages to the target exchange/queue as normal AMQP messages. If the target queue is a stream, messages are appended to the stream.

restore-streams.yaml

mode: restore
backup_id: "stream-backup-001"

target:
  amqp_url: "amqp://guest:guest@localhost:5672/%2f"
  management_url: "http://localhost:15672"
  management_username: guest
  management_password: guest

storage:
  backend: filesystem
  path: /var/lib/rabbitmq-backup/data

restore:
  publish_mode: direct-to-queue
  publisher_confirms: true
  create_missing_queues: true

Set create_missing_queues: true when restoring stream backups into missing target queues and you want the tool to create target queues from manifest queue types before publishing. This option requires target.management_url, target.management_username, and target.management_password.