Version: v3

Backup Strategies

This guide covers backup strategies for self-hosted Langfuse deployments. Follow one of the deployment guides to get started.

Proper backup strategies are essential for protecting your Langfuse data and ensuring business continuity. This guide covers backup approaches for all components of your self-hosted Langfuse deployment.

ClickHouse

ClickHouse stores your observability data including traces, observations, and scores. Backup strategies vary depending on whether you use a managed service or self-hosted deployment.

ClickHouse Cloud (Managed Service)

Automatic Backups: ClickHouse Cloud automatically manages backups for you with:

Continuous incremental backups
Point-in-time recovery capabilities
Cross-region replication options
Enterprise-grade durability guarantees

No Action Required: If you’re using ClickHouse Cloud, backups are handled automatically. Refer to the ClickHouse Cloud documentation for backup retention policies and recovery procedures.

Self-Hosted ClickHouse

For self-hosted ClickHouse instances, you need to implement your own backup strategy.

Kubernetes Deployments

1. Volume Snapshots (Recommended)

Most cloud providers support volume snapshots for persistent volumes. Ensure that you’re also adding snapshots for the clickhouse zookeeper volumes.

# Create a VolumeSnapshot for each ClickHouse replica
kubectl apply -f - <<EOF
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: clickhouse-backup-$(date +%Y%m%d-%H%M%S)
  namespace: langfuse
spec:
  source:
    persistentVolumeClaimName: data-langfuse-clickhouse-0
  volumeSnapshotClassName: csi-hostpath-snapclass
EOF

2. Velero for Complete Cluster Backups

Velero provides comprehensive Kubernetes backup solutions:

# Install Velero
velero install --provider aws --plugins velero/velero-plugin-for-aws:v1.8.0 \
  --bucket langfuse-backups --secret-file ./credentials-velero
 
# Create a backup schedule
velero schedule create langfuse-daily \
  --schedule="0 2 * * *" \
  --include-namespaces langfuse \
  --ttl 720h0m0s

3. ClickHouse Native Backups

Use ClickHouse’s built-in backup functionality. Follow the ClickHouse backup guide for all details.

-- Create a backup
BACKUP DATABASE default TO S3('s3://backup-bucket/clickhouse-backup-{timestamp}', 'access_key', 'secret_key');
 
-- Restore from backup
RESTORE DATABASE default FROM S3('s3://backup-bucket/clickhouse-backup-{timestamp}', 'access_key', 'secret_key');

Docker Deployments

For Docker-based deployments, implement regular volume backups:

#!/bin/bash
# Backup script for ClickHouse Docker volumes
 
BACKUP_DIR="/backups/clickhouse"
TIMESTAMP=$(date +%Y%m%d_%H%M%S)
CONTAINER_NAME="clickhouse-server"
 
# Create backup directory
mkdir -p "$BACKUP_DIR"
 
# Stop ClickHouse temporarily for consistent backup
docker stop "$CONTAINER_NAME"
 
# Create tar archive of data volume
docker run --rm \
  -v clickhouse_data:/source:ro \
  -v "$BACKUP_DIR":/backup \
  alpine tar czf "/backup/clickhouse-backup-$TIMESTAMP.tar.gz" -C /source .
 
# Restart ClickHouse
docker start "$CONTAINER_NAME"
 
# Clean up old backups (keep last 7 days)
find "$BACKUP_DIR" -name "clickhouse-backup-*.tar.gz" -mtime +7 -delete

Postgres

Postgres stores critical transactional data including users, organizations, projects, and API keys. We strongly recommend using managed database services for production deployments.

Managed Database Services (Recommended)

Cloud Provider Services: Use managed PostgreSQL services which provide automatic backups.

Self-Hosted Postgres Backups

If you must self-host Postgres, implement comprehensive backup strategies.

Kubernetes Generic Backup Solution

1. pg_dump with CronJob

apiVersion: batch/v1
kind: CronJob
metadata:
  name: postgres-backup
  namespace: langfuse
spec:
  schedule: "0 2 * * *"  # Daily at 2 AM
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: postgres-backup
            image: postgres:15
            env:
            - name: PGPASSWORD
              valueFrom:
                secretKeyRef:
                  name: postgres-credentials
                  key: password
            command:
            - /bin/bash
            - -c
            - |
              TIMESTAMP=$(date +%Y%m%d_%H%M%S)
              pg_dump -h postgres-service -U langfuse -d langfuse > /backup/langfuse-backup-$TIMESTAMP.sql
 
              # Upload to S3 (optional)
              aws s3 cp /backup/langfuse-backup-$TIMESTAMP.sql s3://langfuse-backups/postgres/
 
              # Clean up local files older than 3 days
              find /backup -name "langfuse-backup-*.sql" -mtime +3 -delete
            volumeMounts:
            - name: backup-storage
              mountPath: /backup
          volumes:
          - name: backup-storage
            persistentVolumeClaim:
              claimName: postgres-backup-pvc
          restartPolicy: OnFailure

2. Volume Snapshots

# Create snapshot of Postgres PVC
kubectl apply -f - <<EOF
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: postgres-backup-$(date +%Y%m%d-%H%M%S)
  namespace: langfuse
spec:
  source:
    persistentVolumeClaimName: postgres-data-pvc
  volumeSnapshotClassName: csi-hostpath-snapclass
EOF

Docker Deployments

#!/bin/bash
# Postgres backup script for Docker
 
BACKUP_DIR="/backups/postgres"
TIMESTAMP=$(date +%Y%m%d_%H%M%S)
CONTAINER_NAME="postgres"
DB_NAME="langfuse"
DB_USER="langfuse"
 
mkdir -p "$BACKUP_DIR"
 
# Create SQL dump
docker exec "$CONTAINER_NAME" pg_dump -U "$DB_USER" "$DB_NAME" > "$BACKUP_DIR/langfuse-backup-$TIMESTAMP.sql"
 
# Compress backup
gzip "$BACKUP_DIR/langfuse-backup-$TIMESTAMP.sql"
 
# Upload to cloud storage (optional)
aws s3 cp "$BACKUP_DIR/langfuse-backup-$TIMESTAMP.sql.gz" s3://langfuse-backups/postgres/
 
# Clean up old backups
find "$BACKUP_DIR" -name "langfuse-backup-*.sql.gz" -mtime +7 -delete

MinIO

MinIO is Obsolete with Cloud Storage: If you’re using cloud storage services like AWS S3, Azure Blob Storage, or Google Cloud Storage, MinIO is not needed and backup strategies should focus on your cloud storage provider’s native backup features.

MinIO is only relevant for self-hosted deployments that don’t use cloud storage services. For most production deployments, we recommend using managed cloud storage instead.

When MinIO is Used

MinIO is typically used in:

Air-gapped environments
On-premises deployments without cloud access
Development environments
Specific compliance requirements

MinIO Backup Strategies

Cloud Storage Replication (Recommended)

Configure MinIO to replicate to cloud storage:

# Configure MinIO client
mc alias set myminio http://localhost:9000 minio miniosecret
mc alias set s3backup https://s3.amazonaws.com ACCESS_KEY SECRET_KEY
 
# Set up bucket replication
mc replicate add myminio/langfuse --remote-bucket s3backup/langfuse-backup

Kubernetes MinIO Backups

1. Volume Snapshots

# Snapshot MinIO data volumes
kubectl apply -f - <<EOF
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: minio-backup-$(date +%Y%m%d-%H%M%S)
  namespace: langfuse
spec:
  source:
    persistentVolumeClaimName: data-minio-0
  volumeSnapshotClassName: csi-hostpath-snapclass
EOF

2. Scheduled Sync to External Storage

apiVersion: batch/v1
kind: CronJob
metadata:
  name: minio-backup-sync
  namespace: langfuse
spec:
  schedule: "0 3 * * *"  # Daily at 3 AM
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: minio-backup
            image: minio/mc:latest
            command:
            - /bin/sh
            - -c
            - |
              mc alias set source http://minio:9000 $MINIO_ACCESS_KEY $MINIO_SECRET_KEY
              mc alias set backup s3://backup-bucket $AWS_ACCESS_KEY $AWS_SECRET_KEY
              mc mirror source/langfuse backup/langfuse-backup/$(date +%Y%m%d)
            env:
            - name: MINIO_ACCESS_KEY
              valueFrom:
                secretKeyRef:
                  name: minio-credentials
                  key: access-key
            - name: MINIO_SECRET_KEY
              valueFrom:
                secretKeyRef:
                  name: minio-credentials
                  key: secret-key
            - name: AWS_ACCESS_KEY
              valueFrom:
                secretKeyRef:
                  name: aws-credentials
                  key: access-key
            - name: AWS_SECRET_KEY
              valueFrom:
                secretKeyRef:
                  name: aws-credentials
                  key: secret-key
          restartPolicy: OnFailure

Sizing & Scaling Local/VM (Docker Compose)

Was this page helpful?

Support