Export to blob storage

Overview

Schedule exports to Amazon S3, S3-compatible storage, Google Cloud Storage, or Azure Blob Storage every 20 minutes, hourly, daily, or weekly.

Configure the export

Create the integration

1. Open Project Settings > Integrations > Blob Storage.
2. Select your provider and enter its bucket, path, and credentials. Self-hosters can find per-vendor bucket and credential setup on the blob storage deployment page.
3. Choose the file format, schedule, export mode, and field groups.
4. Enable the integration and save. The first run starts shortly afterward.

The integration settings page shows a status badge:

Parquet notes

• Observation files omit input_price, output_price, and total_price. Use cost_details and total_cost; trace and score fields are unaffected.
• On self-hosted ClickHouse older than 25.11, upgrade ClickHouse or use CSV, JSON, or JSONL because incomplete Parquet output may not be reported.

Use Validate before saving to check that Langfuse can access the target with the supplied credentials. The integration then writes all output beneath the configured prefix without changing other objects in the bucket.

Langfuse writes exports to your bucket using this structure:

{prefix}{project-id}/
├── observations_v2/
│   └── {timestamp}.{parquet|json|jsonl|csv}[.gz]
├── scores/
│   └── {timestamp}.{parquet|json|jsonl|csv}[.gz]
└── manifests/
    └── {timestamp}.json

Field selection

Observation columns are configurable; score columns are fixed.

Configure through the API

Use the public endpoint to read or update the integration:

GET /api/public/integrations/blob-storage
PUT /api/public/integrations/blob-storage

On PUT, exportFieldGroups must include core; omit the property to keep the current selection. compressed applies only to CSV, JSON, and JSONL.

See the REST API reference for authentication, provider settings, and the complete schema.

Process exports

Use each run's manifest as the unit of work.

Know when an export is complete

A run is complete only when its manifest exists at {prefix}{project-id}/manifests/{timestamp}.json. Langfuse writes it after every data file uploads successfully.

If an upload fails, no manifest is written. The run is retried and any data objects under the same keys are overwritten.

When a manifest appears:

1. Read files[].
2. Fetch every full object key in files[].key. Do not derive filenames or list table directories.
3. Deduplicate records by ID because adjacent export windows share an inclusive boundary.

For Parquet, files[].rowCount is null; inspect the file metadata when you need a count. Ignore unknown manifest fields so additive changes do not break your parser.

Trigger your pipeline

Subscribe to object-created events filtered to {prefix}{project-id}/manifests/, then fetch the manifest and process each files[].key.

• Amazon S3: S3 Event Notifications or EventBridge
• Google Cloud Storage: Pub/Sub notifications for OBJECT_FINALIZE
• Azure Blob Storage: Event Grid for Microsoft.Storage.BlobCreated
• MinIO: bucket notifications
• Backblaze B2: event notifications

If your S3-compatible provider has no object-created events, poll the manifest prefix and checkpoint the latest processed key. Manifest names sort by export timestamp.

Make manifest processing idempotent. Provider events can be delivered more than once, and a catch-up can create several manifests close together.

Upgrade a legacy export

What changes in the exported data

The legacy source writes observation data to observations/ and trace context to a separate traces/ file, which consumers join on trace_id. The enriched source writes a single observations_v2/ file in which each observation row already carries its trace context.

Field groups select columns in both observation files. Three groups contain fewer fields in the legacy observations/ file; the rest are identical:

The traces/ file itself has a fixed schema that field groups do not apply to. Its trace-level input, output, metadata, timestamp, and version have no direct equivalent in observations_v2/, where identically named fields hold observation-level data.

For every column in each file, see the legacy field reference.

Switch to enriched observations

Most integrations can switch directly once their consumer supports the enriched layout:

On self-hosted Langfuse v4, first switch the server to dual write mode and complete the historic data migration.

Enriched export sources require the deployment to write the v4 events table, so they are unavailable only in legacy write mode. Legacy export sources are unavailable in events_only, which is the default. Switch the integration before the server's events_only cutover.

1. Prepare the new consumer. Process observations_v2/ and scores/ from each manifest's files[].key. Use the enriched field reference.
2. Switch the export. Select Enriched observations (recommended).
3. Verify and retire legacy. Process the next completed export with the new consumer. After it succeeds, retire the consumer that reads traces/ and observations/.

Cutover checks

• All deployments: traces/ and observations/ stop receiving files after the switch, including empty files. Use manifests/ as the run-completion signal.
• Langfuse Cloud: Ignore DEPRECATION_NOTICE.txt when processing exports; it is not part of a run.
• Self-hosted Langfuse v4: dual is the only write mode in which both layouts are selectable, so the optional dual-layout validation above requires it.

Export field reference

Field names and types for each exported file. Types match JSON and JSONL output. Timestamps use YYYY-MM-DD HH:MM:SS.ffffff in UTC. Each schema is collapsed on its own below—open the file you need.

For paths, filenames, and the manifest workflow, see process exports.

FAQ

• How do blob storage export timing and retries work?
• I changed my blob storage config — why didn't it re-export, and why do I still see an error?
• Why do I see empty files in my blob storage export?