ClickHouse

ClickHouse is the main OLAP storage solution within Langfuse for our Trace, Observation, and Score entities. It is optimized for high write throughput and fast analytical queries. This guide covers how to configure ClickHouse within Langfuse and what to keep in mind when (optionally) bringing your own ClickHouse.

Langfuse supports ClickHouse versions >= 24.3.

Configuration

Langfuse accepts the following environment variables to fine-tune your ClickHouse usage. They need to be provided for the Langfuse Web and Langfuse Worker containers.

Please note, that Langfuse uses the default schema and default cluster. Get in touch with us on Discord or contact the maintainers at [email protected] if this limitation is blocking.

Deployment Options

This section covers different deployment options and provides example environment variables.

ClickHouse Cloud

ClickHouse Cloud is a scalable and fully managed deployment option for ClickHouse. You can provision it directly from ClickHouse or through one of the cloud provider marketplaces:

• AWS Marketplace
• Google Cloud Marketplace
• Azure Marketplace

ClickHouse Cloud clusters will be provisioned outside your cloud environment and your VPC, but Clickhouse offers private links for AWS, GCP, and Azure.

Example Configuration

Set the following environment variables to connect to your ClickHouse instance:

CLICKHOUSE_URL=https://<identifier>.<region>.aws.clickhouse.cloud:8443
CLICKHOUSE_MIGRATION_URL=clickhouse://<identifier>.<region>.aws.clickhouse.cloud:9440
CLICKHOUSE_USER=default
CLICKHOUSE_PASSWORD=changeme
CLICKHOUSE_MIGRATION_SSL=true

Troubleshooting

• ‘error: driver: bad connection in line 0’ during migration: If you see the previous error message during startup of your web container, ensure that the CLICKHOUSE_MIGRATION_SSL flag is set and that Langfuse Web can access your ClickHouse environment. Review the IP whitelisting if applicable and whether the instance has access to the Private Link.

ClickHouse on Kubernetes (Helm)

The Bitnami ClickHouse Helm Chart provides a production ready deployment of ClickHouse using a given Kubernetes cluster. We use it as a dependency for Langfuse K8s. See Langfuse on Kubernetes (Helm) for more details on how to deploy Langfuse on Kubernetes.

Example Configuration

For a minimum production setup, we recommend to use the following values.yaml overwrites when deploying the Clickhouse Helm chart:

clickhouse:
  deploy: true
  shards: 1 # Fixed: Langfuse does not support multi-shard clusters
  replicaCount: 3
  resourcesPreset: large # or more
  auth:
    username: default
    password: changeme

• shards: Shards are used for horizontally scaling ClickHouse. A single ClickHouse shard can handle multiple Terabytes of data. Today, Langfuse does not support a multi-shard cluster, i.e. this value must be set to 1. Please get in touch with us if you hit scaling limits of a single shard cluster.
• replicaCount: The number of replicas for each shard. ClickHouse counts the all instances towards the number of replicas, i.e. a replica count of 1 means no redundancy at all. We recommend a minimum of 3 replicas for production setups. The number of replicas cannot be increased at runtime without manual intervention or downtime.
• resourcesPreset: ClickHouse is CPU and memory intensive for analytical and highly concurrent requests. We recommend at least the large resourcesPreset and more for larger deployments.
• auth: The username and password for the ClickHouse database. Overwrite those values according to your preferences, or mount them from a secret.
• disk: The ClickHouse Helm chart uses the default storage class to create volumes for each replica. Ensure that the storage class has allowVolumeExpansion = true as observability workloads tend to be very disk heavy. For cloud providers like AWS, GCP, and Azure this should be the default.

Langfuse assumes that certain parameters are set in the ClickHouse configurations. To perform our database migrations, the following values must be provided:

<!--
    Substitutions for parameters of replicated tables.
     Optional. If you don't use replicated tables, you could omit that.
     See https://clickhouse.com/docs/en/engines/table-engines/mergetree-family/replication/#creating-replicated-tables
-->
<!--
    <macros>
        <shard>01</shard>
        <replica>example01-01-1</replica>
    </macros>
-->
<!--
    <default_replica_path>/clickhouse/tables/{database}/{table}</default_replica_path>
    <default_replica_name>{replica}</default_replica_name>
-->

macros and default_replica_* configuration should be covered by the Helm chart without any further configuration.

Set the following environment variables to connect to your ClickHouse instance assuming that Langfuse runs within the same Cluster and Namespace:

CLICKHOUSE_URL=http://<chart-name>-clickhouse:8123
CLICKHOUSE_MIGRATION_URL=clickhouse://<chart-name>-clickhouse:9000
CLICKHOUSE_USER=default
CLICKHOUSE_PASSWORD=changeme

Docker

You can run ClickHouse in a single Docker container for development purposes. As there is no redundancy, this is not recommended for production workloads.

Example Configuration

Start the container with

docker run --name clickhouse-server \
  -e CLICKHOUSE_DB=default \
  -e CLICKHOUSE_USER=clickhouse \
  -e CLICKHOUSE_PASSWORD=clickhouse \
  -d --ulimit nofile=262144:262144 \
  -p 8123:8123 \
  -p 9000:9000 \
  -p 9009:9009 \
  clickhouse/clickhouse-server

Set the following environment variables to connect to your ClickHouse instance:

CLICKHOUSE_URL=http://localhost:8123
CLICKHOUSE_MIGRATION_URL=clickhouse://localhost:9000
CLICKHOUSE_USER=clickhouse
CLICKHOUSE_PASSWORD=clickhouse
CLICKHOUSE_CLUSTER_ENABLED=false

Backups

ClickHouse Cloud manages backups automatically for you. For self-hosted ClickHouse instances, you need to create backups on your own. We recommend following the ClickHouse backup guide for more details. In addition, the ClickHouse state can be restored based on the data in S3. While this is computationally expensive and may take a long time, it is an alternative safety measure to prevent data loss.