DocsSelf-hostv3 (Preview)ComponentsClickhouse

ClickHouse

ClickHouse is the main storage solution within Langfuse for our Trace, Observation, and Score entities. It is optimized for high write throughput and fast analytical queries. This guide covers how to configure ClickHouse within Langfuse and what to keep in mind when bringing your own ClickHouse.

Configuration

Langfuse accepts the following environment variables to fine-tune your ClickHouse usage. They need to be provided for the Langfuse Web and Langfuse Worker containers.

VariableRequired / DefaultDescription
CLICKHOUSE_MIGRATION_URLRequiredMigration URL (TCP protocol) for the ClickHouse instance. Pattern: clickhouse://<hostname>:(9000/9440)
CLICKHOUSE_MIGRATION_SSLfalseSet to true to establish an SSL connection to ClickHouse for the database migration.
CLICKHOUSE_URLRequiredHostname of the ClickHouse instance. Pattern: http(s)://<hostname>:(8123/8443)
CLICKHOUSE_USERRequiredUsername of the ClickHouse database. Needs SELECT, ALTER, INSERT, CREATE, DELETE grants.
CLICKHOUSE_PASSWORDRequiredPassword of the ClickHouse user.
CLICKHOUSE_CLUSTER_ENABLEDtrueWhether to run ClickHouse commands ON CLUSTER. Set to false for single-container setups.
LANGFUSE_AUTO_CLICKHOUSE_MIGRATION_DISABLEDfalseWhether to disable automatic ClickHouse migrations.

Please note, that Langfuse uses the default schema and default cluster. Get in touch with us on Discord or contact the maintainers at [email protected] if this limitation is blocking.

Langfuse only supports ClickHouse versions >= 24.3.

Deployment Options

This section covers different deployment options and provides example environment variables.

ClickHouse Cloud

ClickHouse Cloud is a scalable and fully managed deployment option for ClickHouse. You can provision it directly from ClickHouse or through one of the cloud provider marketplaces:

ClickHouse Cloud clusters will be provisioned outside your cloud environment and your VPC, but Clickhouse offers private links for AWS, GCP, and Azure.

Example Configuration

Set the following environment variables to connect to your ClickHouse instance:

CLICKHOUSE_URL=https://<identifier>.<region>.aws.clickhouse.cloud:8443
CLICKHOUSE_MIGRATION_URL=clickhouse://<identifier>.<region>.aws.clickhouse.cloud:9440
CLICKHOUSE_USER=default
CLICKHOUSE_PASSWORD=changeme
CLICKHOUSE_MIGRATION_SSL=true

Troubleshooting

  • ‘error: driver: bad connection in line 0’ during migration: If you see the previous error message during startup of your web container, ensure that the CLICKHOUSE_MIGRATION_SSL flag is set and that Langfuse Web can access your ClickHouse environment. Review the IP whitelisting if applicable and whether the instance has access to the Private Link.

ClickHouse on Kubernetes (Helm)

The Bitnami ClickHouse Helm Chart provides a production ready deployment of ClickHouse using a given Kubernetes cluster. We use it as a dependency for Langfuse K8s. See Langfuse on Kubernetes (Helm) for more details on how to deploy Langfuse on Kubernetes.

Example Configuration

For a minimum production setup, we recommend to use the following values.yaml overwrites when deploying the Clickhouse Helm chart:

clickhouse:
  deploy: true
  shards: 1 # Fixed: Langfuse does not support multi-shard clusters
  replicaCount: 3
  resourcesPreset: large # or more
  auth:
    username: default
    password: changeme
  • shards: Shards are used for horizontally scaling ClickHouse. A single ClickHouse shard can handle multiple Terabytes of data. Today, Langfuse does not support a multi-shard cluster, i.e. this value must be set to 1. Please get in touch with us if you hit scaling limits of a single shard cluster.
  • replicaCount: The number of replicas for each shard. ClickHouse counts the all instances towards the number of replicas, i.e. a replica count of 1 means no redundancy at all. We recommend a minimum of 3 replicas for production setups. The number of replicas cannot be increased at runtime without manual intervention or downtime.
  • resourcesPreset: ClickHouse is CPU and memory intensive for analytical and highly concurrent requests. We recommend at least the large resourcesPreset and more for larger deployments.
  • auth: The username and password for the ClickHouse database. Overwrite those values according to your preferences, or mount them from a secret.
  • disk: The ClickHouse Helm chart uses the default storage class to create volumes for each replica. Ensure that the storage class has allowVolumeExpansion = true as observability workloads tend to be very disk heavy. For cloud providers like AWS, GCP, and Azure this should be the default.

Langfuse assumes that certain parameters are set in the ClickHouse configurations. To perform our database migrations, the following values must be provided:

<!--
    Substitutions for parameters of replicated tables.
     Optional. If you don't use replicated tables, you could omit that.
     See https://clickhouse.com/docs/en/engines/table-engines/mergetree-family/replication/#creating-replicated-tables
-->
<!--
    <macros>
        <shard>01</shard>
        <replica>example01-01-1</replica>
    </macros>
-->
<!--
    <default_replica_path>/clickhouse/tables/{database}/{table}</default_replica_path>
    <default_replica_name>{replica}</default_replica_name>
-->

macros and default_replica_* configuration should be covered by the Helm chart without any further configuration.

Set the following environment variables to connect to your ClickHouse instance assuming that Langfuse runs within the same Cluster and Namespace:

CLICKHOUSE_URL=http://<chart-name>-clickhouse:8123
CLICKHOUSE_MIGRATION_URL=clickhouse://<chart-name>-clickhouse:9000
CLICKHOUSE_USER=default
CLICKHOUSE_PASSWORD=changeme

Docker

You can run ClickHouse in a single Docker container for development purposes. As there is no redundancy, this is not recommended for production workloads.

Example Configuration

Start the container with

docker run --name clickhouse-server \
  -e CLICKHOUSE_DB=default \
  -e CLICKHOUSE_USER=clickhouse \
  -e CLICKHOUSE_PASSWORD=clickhouse \
  -d --ulimit nofile=262144:262144 \
  -p 8123:8123 \
  -p 9000:9000 \
  -p 9009:9009 \
  clickhouse/clickhouse-server

Set the following environment variables to connect to your ClickHouse instance:

CLICKHOUSE_URL=http://localhost:8123
CLICKHOUSE_MIGRATION_URL=clickhouse://localhost:9000
CLICKHOUSE_USER=clickhouse
CLICKHOUSE_PASSWORD=clickhouse
CLICKHOUSE_CLUSTER_ENABLED=false

Backups

ClickHouse Cloud manages backups automatically for you. For self-hosted ClickHouse instances, you need to create backups on your own. We recommend following the ClickHouse backup guide for more details. In addition, the ClickHouse state can be restored based on the data in S3. While this is computationally expensive and may take a long time, it is an alternative safety measure to prevent data loss.

Was this page useful?

Questions? We're here to help

Subscribe to updates