GitHub Integration for Langfuse Prompts

There are two methods to integrate Langfuse prompts with GitHub:

GitHub Actions Webhook - Trigger CI/CD workflows when prompts change. This does not require additional infrastructure.
Sync Langfuse Prompts to a repository - Store prompts in a specific file in your repository. This involves a webhook server that listens for prompt version changes and commits them to the repository.

Trigger GitHub Actions

Trigger GitHub Actions workflows when Langfuse prompts change using repository_dispatch events.

1. Create GitHub Workflow

.github/workflows/langfuse-ci.yml:

name: Langfuse Prompt CI
on:
  repository_dispatch:
    types: [langfuse-prompt-update]
  workflow_dispatch:
 
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Run tests
        run: |
          echo "Testing prompt: ${{ github.event.client_payload.prompt.name }} v${{ github.event.client_payload.prompt.version }}"
          # Add your test commands
          # npm test
          # python -m pytest
          
  deploy:
    needs: test
    runs-on: ubuntu-latest
    if: contains(github.event.client_payload.prompt.labels, 'production')
    steps:
      - uses: actions/checkout@v4
      - name: Deploy to production
        run: |
          echo "Deploying ${{ github.event.client_payload.prompt.name }} v${{ github.event.client_payload.prompt.version }}"
          # Your deployment commands

Accessing webhook data: Use github.event.client_payload.* to access prompt data:

# Example: Access webhook data in your workflow
- name: Process prompt data
  run: |
    echo "Action: ${{ github.event.client_payload.action }}"
    echo "Prompt: ${{ github.event.client_payload.prompt.name }}"
    echo "Version: ${{ github.event.client_payload.prompt.version }}"
    echo "Labels: ${{ github.event.client_payload.prompt.labels }}"
    
- name: Deploy only production prompts
  if: contains(github.event.client_payload.prompt.labels, 'production')
  run: echo "Deploying production prompt"

2. Create GitHub Token for Actions

Steps:

GitHub Settings > Developer settings > Personal access tokens
Generate new token (classic or fine-grained)
Select scope (see table below)

Token Type	Required Permissions
Personal Access Token (classic)	`repo` scope (public repos) or `public_repo` scope (private repos)
Fine-grained PAT or GitHub App	`read` and `write` to `actions`

3. Configure Langfuse Webhook for Actions

Go to Prompts > Webhooks in your Langfuse project
Click Create Webhook
Set endpoint URL: https://api.github.com/repos/{owner}/{repo}/dispatches
Add headers:
- Accept: application/vnd.github+json
- Authorization: Bearer {your_github_token}

Important: Store your GitHub token securely in the Authorization header. Langfuse encrypts webhook headers and stores them securely.

4. Test GitHub Actions Integration

Update a prompt in Langfuse with the production label
Check GitHub Actions tab for triggered workflow
Verify that both test and deploy jobs run successfully

Sync Langfuse Prompts to a repository

Automatically sync prompt changes from Langfuse to GitHub using Prompt Version Webhooks. This enables version control for prompts and can trigger CI/CD workflows.

Overview of the Sync Workflow

Whenever you save a new prompt version in Langfuse, it’s automatically committed to your GitHub repository. With this setup, you can also trigger CI/CD workflows when prompts change.

Prerequisites for Sync

Langfuse Project: Prompt setup with Project Owner access
GitHub Repository: Public or private repo to store prompts
GitHub PAT: Personal Access Token with minimum required permissions (see Step 2 for details)
Python 3.9+ (for the example below, can be any language) with FastAPI, Uvicorn, httpx, Pydantic
Public HTTPS endpoint for your webhook server (Render, Fly.io, Heroku, etc.)

Step 1: Configure a Prompt Webhook in Langfuse

Go to Prompts > Webhooks in your Langfuse project
Click Create Webhook
(optional) filter events: filter by which prompt version events to receive webhooks (default: created, updated, deleted)
Set endpoint URL: https://<your-domain>/webhook/prompt
Save and copy the Signing Secret

Note: Your endpoint must return 2xx status codes. Langfuse retries failed webhooks with exponential backoff.

Sample Webhook Payload

Sample webhook payload:

{
  "id": "550e8400-e29b-41d4-a716-446655440000",
  "timestamp": "2024-07-10T10:30:00Z",
  "type": "prompt-version",
  "action": "created",
  "prompt": {
    "id": "prompt_abc123",
    "name": "movie-critic",
    "version": 3,
    "projectId": "xyz789",
    "labels": ["production", "latest"],
    "prompt": "As a {{criticLevel}} movie critic, rate {{movie}} out of 10.",
    "type": "text",
    "config": { "...": "..." },
    "commitMessage": "Improved critic persona",
    "tags": ["entertainment"],
    "createdAt": "2024-07-10T10:30:00Z",
    "updatedAt": "2024-07-10T10:30:00Z"
  }
}

Step 2: Prepare Your GitHub Repo and Token for Sync

Create a .env file with your GitHub credentials:

GITHUB_TOKEN=<your_github_pat_here>
GITHUB_REPO_OWNER=<github_username_or_org>
GITHUB_REPO_NAME=<repo_name>
# (Optional) GITHUB_FILE_PATH=langfuse_prompt.json
# (Optional) GITHUB_BRANCH=main
# (Optional) REQUIRED_LABEL=production

Replace placeholders with your actual values. The server will commit prompts to langfuse_prompt.json on the main branch by default. If REQUIRED_LABEL is set, only prompts with that specific label will be synced to GitHub.

GitHub PAT Permissions for Sync

For the webhook to work, your GitHub Personal Access Token needs minimal permissions:

Permission Type	Required Permissions
Required Permissions	Contents: Read and write, Metadata: Read-only
Legacy Token Scopes	For public repositories: `public_repo` scope, For private repositories: `repo` scope

Step 3: Implement the FastAPI Webhook Server

Create main.py with this FastAPI server:

from typing import Any, Dict
from uuid import UUID
import json
import base64
 
import httpx
from pydantic import BaseModel, Field
from pydantic_settings import BaseSettings, SettingsConfigDict
from fastapi import FastAPI, HTTPException, Body
 
class GitHubSettings(BaseSettings):
    """GitHub repository configuration."""
    GITHUB_TOKEN: str
    GITHUB_REPO_OWNER: str
    GITHUB_REPO_NAME: str
    GITHUB_FILE_PATH: str = "langfuse_prompt.json"
    GITHUB_BRANCH: str = "main"
    REQUIRED_LABEL: str = ""  # Optional: only sync prompts with this label
 
    model_config = SettingsConfigDict(
        env_file=".env",
        env_file_encoding="utf-8",
        case_sensitive=True
    )
 
config = GitHubSettings()
 
class LangfuseEvent(BaseModel):
    """Langfuse webhook event structure."""
    id: UUID = Field(description="Event identifier")
    timestamp: str = Field(description="Event timestamp")
    type: str = Field(description="Event type")
    action: str = Field(description="Performed action")
    prompt: Dict[str, Any] = Field(description="Prompt content")
 
async def sync(event: LangfuseEvent) -> Dict[str, Any]:
    """Synchronize prompt data to GitHub repository."""
    # Check if prompt has required label (if specified)
    if config.REQUIRED_LABEL:
        prompt_labels = event.prompt.get("labels", [])
        if config.REQUIRED_LABEL not in prompt_labels:
            return {"skipped": f"Prompt does not have required label '{config.REQUIRED_LABEL}'"}
    
    api_endpoint = f"https://api.github.com/repos/{config.GITHUB_REPO_OWNER}/{config.GITHUB_REPO_NAME}/contents/{config.GITHUB_FILE_PATH}"
 
    request_headers = {
        "Authorization": f"Bearer {config.GITHUB_TOKEN}",
        "Accept": "application/vnd.github.v3+json"
    }
 
    content_json = json.dumps(event.prompt, indent=2)
    encoded_content = base64.b64encode(content_json.encode("utf-8")).decode("utf-8")
 
    name = event.prompt.get("name", "unnamed")
    version = event.prompt.get("version", "unknown")
    message = f"{event.action}: {name} v{version}"
 
    payload = {
        "message": message,
        "content": encoded_content,
        "branch": config.GITHUB_BRANCH
    }
 
    async with httpx.AsyncClient() as http_client:
        try:
            existing = await http_client.get(api_endpoint, headers=request_headers, params={"ref": config.GITHUB_BRANCH})
            if existing.status_code == 200:
                payload["sha"] = existing.json().get("sha")
        except Exception:
            pass
 
        try:
            response = await http_client.put(api_endpoint, headers=request_headers, json=payload)
            response.raise_for_status()
            return response.json()
        except Exception as e:
            raise HTTPException(status_code=500, detail=f"Repository sync failed: {str(e)}")
 
app = FastAPI(title="Langfuse GitHub Sync", version="1.0")
 
@app.post("/webhook/prompt", status_code=201)
async def receive_webhook(event: LangfuseEvent = Body(...)):
    """Process Langfuse webhook and sync to GitHub."""
    result = await sync(event)
    return {
        "status": "synced",
        "commit_info": result.get("commit", {}),
        "file_info": result.get("content", {})
    }
 
@app.get("/status")
async def health_status():
    """Service health check."""
    return {"healthy": True}

The server validates webhook payloads, retrieves existing file SHAs if needed, and commits prompt changes to GitHub with descriptive commit messages.

Dependencies

Install dependencies:

pip install fastapi uvicorn pydantic-settings httpx

Running Locally

Run locally:

uvicorn main:app --reload --port 8000

Test the health endpoint at http://localhost:8000/health. Use ngrok or similar to expose localhost for webhook testing.

Step 4: Deploy and Connect the Server

Deploy: Use Render, Fly.io, Heroku, or similar. Set environment variables and ensure HTTPS is enabled.
Update Webhook: In Langfuse, edit your webhook and set the URL to https://your-domain.com/webhook/prompt.
Test: Update a prompt in Langfuse and verify a new commit appears in your GitHub repository.

Security Considerations

Verify signatures: Use the signing secret and x-langfuse-signature header to validate requests
Limit PAT scope: Use fine-grained tokens restricted to specific repositories
Handle retries: The implementation is idempotent - duplicate events won’t create conflicting commits

Folders Guaranteed Availability

Was this page helpful?

Support