Skip to main content

Backup to S3

This guide walks through configuring rabbitmq-backup to back up RabbitMQ messages and definitions to S3, then listing and restoring from those backups.

Prerequisites​

  • A running RabbitMQ broker with the Management Plugin enabled
  • An S3 bucket (or S3-compatible service like MinIO) with write access
  • rabbitmq-backup installed (install guide)

Step 1: Create the Backup Configuration​

backup-s3.yaml
mode: backup
backup_id: "s3-backup-001"

source:
  amqp_url: "amqp://guest:guest@localhost:5672/%2f"
  management_url: "http://localhost:15672"
  management_username: guest
  management_password: guest
  queues:
    include:
      - "*"
    exclude:
      - "*-dead-letter"
    vhosts:
      - "/"

storage:
  backend: s3
  bucket: rabbitmq-backups
  region: us-east-1
  prefix: prod/

backup:
  compression: zstd
  compression_level: 3
  prefetch_count: 100
  requeue_strategy: cancel
  max_concurrent_queues: 4
  include_definitions: true
  stop_at_current_depth: true

offset_storage:
  backend: sqlite
  db_path: ./offsets.db
  s3_key: state/offsets.db
  sync_interval_secs: 30

metrics:
  enabled: true
  port: 8080

Set your credentials:

export AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
export AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

Step 2: Run the Backup​

rabbitmq-backup backup --config backup-s3.yaml

Add -v for debug output to see per-queue progress:

rabbitmq-backup backup -v --config backup-s3.yaml

Expected output:

INFO  Starting backup s3-backup-001
INFO  Connected to RabbitMQ at localhost:5672
INFO  Discovered 5 queues matching selection criteria
INFO  Backing up queue orders-queue (1,234 messages)
INFO  Backing up queue payments-queue (567 messages)
INFO  Segment segment-0001.zst written (1,234 records, 2.3 MB)
INFO  Segment segment-0001.zst written (567 records, 1.1 MB)
INFO  Definitions exported (3 vhosts, 12 queues, 8 exchanges)
INFO  Backup s3-backup-001 completed: 1,801 messages in 2 segments

Step 3: List Backups​

rabbitmq-backup list --path s3://rabbitmq-backups

Output:

Backups in s3://rabbitmq-backups:
  s3-backup-001    2025-01-15T02:00:00Z    1,801 messages    2 queues
  s3-backup-002    2025-01-16T02:00:00Z    2,105 messages    2 queues

Step 4: Describe a Backup​

Get detailed information about a specific backup:

rabbitmq-backup describe --path s3://rabbitmq-backups --backup-id s3-backup-001

For machine-readable output:

rabbitmq-backup describe --path s3://rabbitmq-backups --backup-id s3-backup-001 --format json

Step 5: Validate Backup Integrity​

Run a checksum validation:

rabbitmq-backup validate --path s3://rabbitmq-backups --backup-id s3-backup-001

For deep validation (verifies every segment's checksum):

rabbitmq-backup validate --path s3://rabbitmq-backups --backup-id s3-backup-001 --deep

Step 6: Restore from S3​

Create a restore configuration:

restore-s3.yaml
mode: restore
backup_id: "s3-backup-001"

target:
  amqp_url: "amqp://guest:guest@localhost:5672/%2f"
  management_url: "http://localhost:15672"
  management_username: guest
  management_password: guest

storage:
  backend: s3
  bucket: rabbitmq-backups
  region: us-east-1
  prefix: prod/

restore:
  restore_definitions: true
  publish_mode: exchange
  publisher_confirms: true
  max_concurrent_queues: 4
  produce_batch_size: 100
  dry_run: false

Run the restore:

rabbitmq-backup restore --config restore-s3.yaml

Dry Run​

Validate the restore without publishing any messages:

# Set dry_run: true in the config, or run with verbose logging to inspect
rabbitmq-backup restore -v --config restore-s3.yaml

Using MinIO (Local Development)​

For local development with MinIO:

# Start MinIO
docker run -d --name minio \
  -p 9000:9000 -p 9001:9001 \
  -e MINIO_ROOT_USER=minioadmin \
  -e MINIO_ROOT_PASSWORD=minioadmin \
  minio/minio server /data --console-address ":9001"

# Create the bucket
docker exec minio mc alias set local http://localhost:9000 minioadmin minioadmin
docker exec minio mc mb local/rabbitmq-backups

Adjust the storage config:

storage:
  backend: s3
  bucket: rabbitmq-backups
  region: us-east-1
  endpoint: http://localhost:9000
  access_key: minioadmin
  secret_key: minioadmin
  path_style: true
  allow_http: true

Next Steps​