ClickHouse
ClickHouse is the main storage solution within Langfuse for our Trace, Observation, and Score entities. It is optimized for high write throughput and fast analytical queries. This guide covers how to configure ClickHouse within Langfuse and what to keep in mind when bringing your own ClickHouse.
Configuration
Langfuse accepts the following environment variables to fine-tune your ClickHouse usage. They need to be provided for the Langfuse Web and Langfuse Worker containers.
Variable | Required / Default | Description |
---|---|---|
CLICKHOUSE_MIGRATION_URL | Required | Migration URL (TCP protocol) for the ClickHouse instance. Pattern: clickhouse://<hostname>:(9000/9440) |
CLICKHOUSE_MIGRATION_SSL | false | Set to true to establish an SSL connection to ClickHouse for the database migration. |
CLICKHOUSE_URL | Required | Hostname of the ClickHouse instance. Pattern: http(s)://<hostname>:(8123/8443) |
CLICKHOUSE_USER | Required | Username of the ClickHouse database. Needs SELECT, ALTER, INSERT, CREATE, DELETE grants. |
CLICKHOUSE_PASSWORD | Required | Password of the ClickHouse user. |
CLICKHOUSE_CLUSTER_ENABLED | true | Whether to run ClickHouse commands ON CLUSTER . Set to false for single-container setups. |
LANGFUSE_AUTO_CLICKHOUSE_MIGRATION_DISABLED | false | Whether to disable automatic ClickHouse migrations. |
Please note, that Langfuse uses the default
schema and default
cluster.
Get in touch with us on Discord or contact the maintainers at [email protected] if this limitation is blocking.
Langfuse only supports ClickHouse versions >= 24.3.
Deployment Options
This section covers different deployment options and provides example environment variables.
ClickHouse Cloud
ClickHouse Cloud is a scalable and fully managed deployment option for ClickHouse. You can provision it directly from ClickHouse or through one of the cloud provider marketplaces:
ClickHouse Cloud clusters will be provisioned outside your cloud environment and your VPC, but Clickhouse offers private links for AWS, GCP, and Azure.
Example Configuration
Set the following environment variables to connect to your ClickHouse instance:
CLICKHOUSE_URL=https://<identifier>.<region>.aws.clickhouse.cloud:8443
CLICKHOUSE_MIGRATION_URL=clickhouse://<identifier>.<region>.aws.clickhouse.cloud:9440
CLICKHOUSE_USER=default
CLICKHOUSE_PASSWORD=changeme
CLICKHOUSE_MIGRATION_SSL=true
Troubleshooting
- ‘error: driver: bad connection in line 0’ during migration: If you see the previous error message during startup of your web container, ensure that the
CLICKHOUSE_MIGRATION_SSL
flag is set and that Langfuse Web can access your ClickHouse environment. Review the IP whitelisting if applicable and whether the instance has access to the Private Link.
ClickHouse on Kubernetes (Helm)
The Bitnami ClickHouse Helm Chart provides a production ready deployment of ClickHouse using a given Kubernetes cluster. We use it as a dependency for Langfuse K8s. See Langfuse on Kubernetes (Helm) for more details on how to deploy Langfuse on Kubernetes.
Example Configuration
For a minimum production setup, we recommend to use the following values.yaml overwrites when deploying the Clickhouse Helm chart:
clickhouse:
deploy: true
shards: 1 # Fixed: Langfuse does not support multi-shard clusters
replicaCount: 3
resourcesPreset: large # or more
auth:
username: default
password: changeme
- shards: Shards are used for horizontally scaling ClickHouse. A single ClickHouse shard can handle multiple Terabytes of data. Today, Langfuse does not support a multi-shard cluster, i.e. this value must be set to 1. Please get in touch with us if you hit scaling limits of a single shard cluster.
- replicaCount: The number of replicas for each shard. ClickHouse counts the all instances towards the number of replicas, i.e. a replica count of 1 means no redundancy at all. We recommend a minimum of 3 replicas for production setups. The number of replicas cannot be increased at runtime without manual intervention or downtime.
- resourcesPreset: ClickHouse is CPU and memory intensive for analytical and highly concurrent requests. We recommend at least the
large
resourcesPreset and more for larger deployments. - auth: The username and password for the ClickHouse database. Overwrite those values according to your preferences, or mount them from a secret.
- disk: The ClickHouse Helm chart uses the default storage class to create volumes for each replica. Ensure that the storage class has
allowVolumeExpansion = true
as observability workloads tend to be very disk heavy. For cloud providers like AWS, GCP, and Azure this should be the default.
Langfuse assumes that certain parameters are set in the ClickHouse configurations. To perform our database migrations, the following values must be provided:
<!--
Substitutions for parameters of replicated tables.
Optional. If you don't use replicated tables, you could omit that.
See https://clickhouse.com/docs/en/engines/table-engines/mergetree-family/replication/#creating-replicated-tables
-->
<!--
<macros>
<shard>01</shard>
<replica>example01-01-1</replica>
</macros>
-->
<!--
<default_replica_path>/clickhouse/tables/{database}/{table}</default_replica_path>
<default_replica_name>{replica}</default_replica_name>
-->
macros
and default_replica_*
configuration should be covered by the Helm chart without any further configuration.
Set the following environment variables to connect to your ClickHouse instance assuming that Langfuse runs within the same Cluster and Namespace:
CLICKHOUSE_URL=http://<chart-name>-clickhouse:8123
CLICKHOUSE_MIGRATION_URL=clickhouse://<chart-name>-clickhouse:9000
CLICKHOUSE_USER=default
CLICKHOUSE_PASSWORD=changeme
Docker
You can run ClickHouse in a single Docker container for development purposes. As there is no redundancy, this is not recommended for production workloads.
Example Configuration
Start the container with
docker run --name clickhouse-server \
-e CLICKHOUSE_DB=default \
-e CLICKHOUSE_USER=clickhouse \
-e CLICKHOUSE_PASSWORD=clickhouse \
-d --ulimit nofile=262144:262144 \
-p 8123:8123 \
-p 9000:9000 \
-p 9009:9009 \
clickhouse/clickhouse-server
Set the following environment variables to connect to your ClickHouse instance:
CLICKHOUSE_URL=http://localhost:8123
CLICKHOUSE_MIGRATION_URL=clickhouse://localhost:9000
CLICKHOUSE_USER=clickhouse
CLICKHOUSE_PASSWORD=clickhouse
CLICKHOUSE_CLUSTER_ENABLED=false
Backups
ClickHouse Cloud manages backups automatically for you. For self-hosted ClickHouse instances, you need to create backups on your own. We recommend following the ClickHouse backup guide for more details. In addition, the ClickHouse state can be restored based on the data in S3. While this is computationally expensive and may take a long time, it is an alternative safety measure to prevent data loss.