Compliance & Audit

Organizations in regulated industries (finance, healthcare, government) often need to retain message data for audit purposes, prove data integrity, and maintain a chain of custody for messages. rabbitmq-backup provides the tools to meet these requirements.

Message Retention

The Problem

RabbitMQ is designed for message delivery, not long-term storage. Once a consumer acknowledges a message, it is gone. Compliance regulations may require retaining messages for months or years.

The Solution

Schedule regular non-destructive backups. Messages remain in RabbitMQ for consumers while copies are stored in durable object storage with configurable lifecycle policies.

compliance-backup.yaml

mode: backup
backup_id: "compliance-2025-q1"

source:
  amqp_url: "amqp://audit_user:${RABBITMQ_PASSWORD}@rabbitmq:5672/%2f"
  management_url: "http://rabbitmq:15672"
  management_username: audit_user
  management_password: "${RABBITMQ_PASSWORD}"
  queues:
    include:
      - "transactions-*"
      - "audit-*"
      - "orders-*"

storage:
  backend: s3
  bucket: compliance-message-archive
  region: us-east-1
  prefix: rabbitmq/2025/q1/

backup:
  compression: zstd
  compression_level: 6  # Higher compression for long-term storage
  include_definitions: true
  stop_at_current_depth: true

Storage Lifecycle for Retention

Configure S3 lifecycle rules to match your retention policy:

aws s3api put-bucket-lifecycle-configuration \
  --bucket compliance-message-archive \
  --lifecycle-configuration '{
    "Rules": [
      {
        "ID": "compliance-retention",
        "Status": "Enabled",
        "Filter": { "Prefix": "rabbitmq/" },
        "Transitions": [
          { "Days": 30, "StorageClass": "STANDARD_IA" },
          { "Days": 90, "StorageClass": "GLACIER" },
          { "Days": 365, "StorageClass": "DEEP_ARCHIVE" }
        ],
        "Expiration": { "Days": 2555 }
      }
    ]
  }'

This example retains messages for 7 years with cost-effective tiering.

Audit Trail

Backup Manifest as Audit Record

Every backup produces a manifest.json containing:

• Backup ID and timestamps (created, completed)
• Source cluster name and RabbitMQ version
• Per-queue message counts and time ranges
• Per-segment checksums (SHA-256)
• Total messages, bytes, and segment counts

{
  "backup_id": "compliance-2025-q1",
  "created_at": 1735689600000,
  "completed_at": 1735690800000,
  "source_cluster": "prod-rabbitmq-cluster",
  "rabbitmq_version": "4.0.0",
  "backup_tool_version": "0.1.0",
  "total_messages": 1500000,
  "total_bytes": 524288000,
  "total_segments": 42,
  "queues": [
    {
      "name": "transactions-queue",
      "vhost": "/",
      "queue_type": "quorum",
      "message_count": 1200000,
      "first_message_timestamp": 1735603200000,
      "last_message_timestamp": 1735689599000,
      "segments": [
        {
          "key": "queues/%2f/transactions-queue/segment-0001.zst",
          "sequence": 1,
          "record_count": 50000,
          "size_bytes": 12582912,
          "checksum": "sha256:a1b2c3d4e5f6..."
        }
      ]
    }
  ]
}

Message-Level Metadata

Each backed-up message preserves all AMQP properties for audit:

• message_id -- unique message identifier
• timestamp -- original publish timestamp
• user_id -- publishing user
• app_id -- publishing application
• correlation_id -- request correlation
• headers -- all custom headers
• backed_up_at -- when the backup captured this message

Integrity Verification

Validate Command

Use the validate command to verify that backup data has not been tampered with:

# Quick validation (manifest and structure)
rabbitmq-backup validate \
  --path s3://compliance-message-archive \
  --backup-id compliance-2025-q1

# Deep validation (verify every segment checksum)
rabbitmq-backup validate \
  --path s3://compliance-message-archive \
  --backup-id compliance-2025-q1 \
  --deep

Deep validation:

1. Downloads each segment
2. Decompresses it
3. Recomputes the SHA-256 checksum
4. Compares against the checksum stored in the manifest

A passing deep validation proves the data is identical to what was originally backed up.

Automated Integrity Checks

Schedule regular validation runs:

# Weekly deep validation
0 6 * * 0 rabbitmq-backup validate \
  --path s3://compliance-message-archive \
  --backup-id compliance-2025-q1 \
  --deep >> /var/log/rabbitmq-backup-audit.log 2>&1

S3 Object Lock (WORM)

For regulatory write-once-read-many (WORM) requirements, enable S3 Object Lock:

# Enable object lock on bucket (must be set at creation)
aws s3api create-bucket \
  --bucket compliance-message-archive \
  --region us-east-1 \
  --object-lock-enabled-for-bucket

# Set default retention
aws s3api put-object-lock-configuration \
  --bucket compliance-message-archive \
  --object-lock-configuration '{
    "ObjectLockEnabled": "Enabled",
    "Rule": {
      "DefaultRetention": {
        "Mode": "COMPLIANCE",
        "Years": 7
      }
    }
  }'

With COMPLIANCE mode, no one -- including the root account -- can delete or overwrite backup data until the retention period expires.

Compliance Reporting

List All Backups

Generate a report of all backups in storage:

rabbitmq-backup list --path s3://compliance-message-archive

Describe Specific Backup

Get detailed information for audit documentation:

rabbitmq-backup describe \
  --path s3://compliance-message-archive \
  --backup-id compliance-2025-q1 \
  --format json > audit-report-2025-q1.json

Message Recovery for Auditors

If auditors need to inspect specific messages, restore with a time window to a separate isolated queue:

audit-restore.yaml

mode: restore
backup_id: "compliance-2025-q1"

target:
  amqp_url: "amqp://auditor:${AUDITOR_PASSWORD}@audit-rabbitmq:5672/audit"

storage:
  backend: s3
  bucket: compliance-message-archive
  region: us-east-1

restore:
  time_window_start: 1735603200000  # Specific date range
  time_window_end:   1735689600000
  queue_mapping:
    transactions-queue: audit-transactions-review
  restore_definitions: false
  publish_mode: direct-to-queue

Compliance Checklist

Requirement	How rabbitmq-backup Addresses It
Message retention	Scheduled backups to durable storage with lifecycle policies
Data integrity	SHA-256 checksums per segment, validate --deep command
Tamper protection	S3 Object Lock (WORM), bucket versioning
Audit trail	Manifest with timestamps, counts, and checksums
Data recovery	Point-in-time restore to isolated queues
Chain of custody	Manifest tracks source cluster, tool version, timestamps
Access control	IAM policies, dedicated backup user with minimal permissions