--- source: ⚠️ Jupyter Notebook title: Automate Code Optimization with Weco and Langfuse sidebarTitle: Weco logo: /images/integrations/weco_icon.svg description: Learn how to integrate Weco with Langfuse to automatically optimize LLM application code using Langfuse datasets and evaluators. Iteratively improve prompts, logic, and performance. category: Integrations --- # Automate Code Optimization with Weco and Langfuse This notebook provides a step-by-step guide on integrating **Weco** with **Langfuse** to automatically optimize LLM application code using Langfuse datasets, code evaluators, and managed LLM-as-a-Judge evaluators. > **What is Weco?** [Weco](https://www.weco.ai/) ([GitHub](https://github.com/WecoAI/weco-cli)) is a code optimization platform. Given a source file and an evaluation function, Weco's optimizer iteratively edits the code, re-evaluates, and keeps the version that scores best. It works with any measurable metric β€” accuracy, latency, cost, or a custom composite score. > **What is Langfuse?** [Langfuse](https://langfuse.com) is an open-source AI engineering platform. It offers tracing and monitoring capabilities for LLM applications. Langfuse helps developers debug, analyze, and optimize their AI systems by providing detailed insights and integrating with a wide array of tools and frameworks through native integrations, OpenTelemetry, and dedicated SDKs. ## How It Works Weco connects to Langfuse as an evaluation backend. On each optimization step, Weco: 1. Edits your source code (e.g. prompts, parsing logic) 2. Runs your **target function** against every item in a Langfuse **dataset** 3. Collects scores from local **code evaluators** and/or **managed evaluators** (LLM-as-a-Judge) configured in the Langfuse UI 4. Combines scores into a single **metric** and keeps the best-performing version Each iteration creates a new **experiment run** in Langfuse so you can compare all variants side-by-side in the Langfuse dashboard. ## Getting Started Let's walk through a practical example of using Weco with Langfuse to optimize a simple QA function. ### Step 1: Install Dependencies For this example we'll install the weco client in an virtual environment. For **global installation instructions and usage with agent skills please refer to [Weco's docs](https://docs.weco.ai/cli/installation)**. ```python !pip install "weco[langfuse]" langfuse openai -q ``` Authenticate with Weco: ```python !weco login ``` ### Step 2: Configure Langfuse SDK Set up your Langfuse API keys. You can get these keys by signing up for a free [Langfuse Cloud](https://cloud.langfuse.com/) account or by [self-hosting Langfuse](https://langfuse.com/self-hosting). These environment variables are essential for the Langfuse client to authenticate and send data to your Langfuse project. ```python import os # Get keys for your project from the project settings page: https://cloud.langfuse.com os.environ.setdefault("LANGFUSE_PUBLIC_KEY", "pk-lf-***"); os.environ.setdefault("LANGFUSE_SECRET_KEY", "sk-lf-***"); os.environ.setdefault("LANGFUSE_BASE_URL", "https://us.cloud.langfuse.com"); # πŸ‡ΊπŸ‡Έ US region # Other Langfuse data regions: πŸ‡ͺπŸ‡Ί EU https://cloud.langfuse.com, πŸ‡―πŸ‡΅ Japan https://jp.cloud.langfuse.com, βš•οΈ HIPAA https://hipaa.cloud.langfuse.com # Your OpenAI key os.environ.setdefault("OPENAI_API_KEY", "sk-proj-***"); ``` With the environment variables set, we can now initialize the Langfuse client. `get_client()` initializes the Langfuse client using the credentials provided in the environment variables. ```python from langfuse import get_client langfuse = get_client() # Verify connection if langfuse.auth_check(): print("Langfuse client is authenticated and ready!") else: print("Authentication failed. Please check your credentials and host.") ``` ### Step 3: Create a Dataset in Langfuse Weco evaluates your code against a Langfuse dataset. Each dataset item has an `input` dict and an optional `expected_output` dict. Let's create a small QA dataset. ```python dataset = langfuse.create_dataset(name="weco-demo-qa") qa_pairs = [ { "input": {"question": "What is the capital of France?"}, "expected_output": {"expected_answer": "Paris"}, }, { "input": {"question": "What is the largest planet in our solar system?"}, "expected_output": {"expected_answer": "Jupiter"}, }, { "input": {"question": "Who wrote Romeo and Juliet?"}, "expected_output": {"expected_answer": "William Shakespeare"}, }, { "input": {"question": "What is the boiling point of water in Celsius?"}, "expected_output": {"expected_answer": "100 degrees Celsius"}, }, { "input": {"question": "What year did the Berlin Wall fall?"}, "expected_output": {"expected_answer": "1989"}, }, ] for pair in qa_pairs: langfuse.create_dataset_item( dataset_name="weco-demo-qa", input=pair["input"], expected_output=pair["expected_output"], ) langfuse.flush() print(f"Created dataset 'weco-demo-qa' with {len(qa_pairs)} items.") ``` ### Step 4: Write the Target Function The target function is the code Weco will optimize. It receives an `inputs` dict from the dataset and returns a dict of outputs. Langfuse calls this function once per dataset item during each evaluation run. Save this as `agent.py` in your working directory: ```python %%writefile agent.py from openai import OpenAI client = OpenAI() SYSTEM_PROMPT = "You are a helpful assistant. Answer the question concisely in one sentence." def answer_question(inputs: dict) -> dict: question = inputs.get("question", "") response = client.chat.completions.create( model="gpt-4o-mini", messages=[ {"role": "system", "content": SYSTEM_PROMPT}, {"role": "user", "content": question}, ], temperature=0.0, ) return {"answer": response.choices[0].message.content} ``` Weco will iteratively edit `agent.py` β€” modifying `SYSTEM_PROMPT`, response parsing, or other logic β€” to improve the evaluation metric. ### Step 5: Write an Evaluator and Metric Function Code evaluators are Python functions that score each target function output. They receive keyword arguments and return a Langfuse `Evaluation` object. The metric function combines all evaluator scores into a single number that Weco optimizes. Save this as `evaluators.py`: ```python %%writefile evaluators.py from langfuse import Evaluation def answer_quality(*, input, output, expected_output=None, **kwargs): """Check that the answer is non-empty and reasonably concise.""" answer = (output or {}).get("answer", "") if not answer: return Evaluation(name="answer_quality", value=0.0, comment="Empty answer") word_count = len(answer.split()) score = 1.0 if word_count <= 50 else max(0.0, 1.0 - (word_count - 50) / 100) return Evaluation( name="answer_quality", value=score, comment=f"{word_count} words", ) def qa_metric(scores: dict) -> float: """Combine evaluator scores into a single optimization target. Multiplies answer_quality by Correctness (a managed evaluator configured in the Langfuse UI) so that only correct, concise answers score well. """ return scores.get("answer_quality", 0.0) * scores.get("Correctness", 0.0) ``` ### Step 6: Configure a Managed Evaluator in Langfuse Managed evaluators are LLM-as-a-Judge evaluators that run server-side on experiment traces. Set one up in the Langfuse UI: 1. Go to your project in [Langfuse](https://cloud.langfuse.com/) 2. Navigate to **Evaluation** β†’ **LL-as-a-Judge** 3. Click **+ Set up evaluator** Create a **Correctness** evaluator: - **Name**: `Correctness` - **Score**: 0 or 1 (binary factual accuracy) - **Variable mappings**: - `{{input}}` β†’ `$.input.question` - `{{output}}` β†’ `$.output.answer` - `{{expected_output}}` β†’ `$.expected_output.expected_answer` > **Important:** Evaluator names are case-sensitive. The name in the Langfuse UI (e.g. `Correctness`) must exactly match the name passed to `--langfuse-managed-evaluators` and the key used in your metric function (`scores.get("Correctness")`). > Managed evaluators run asynchronously after each experiment. Weco automatically polls for their scores (up to 15 minutes by default). Adjust the timeout with `--langfuse-managed-evaluator-timeout`. ### Step 7: Run the Optimization With the dataset, target function, evaluators, and metric function in place, run Weco from the command line. Weco uses Langfuse as the evaluation backend, running your target function against the dataset on each iteration and tracking progress as experiment runs in Langfuse. ```python !weco run --source agent.py \ --eval-backend langfuse \ --langfuse-dataset weco-demo-qa \ --langfuse-target agent:answer_question \ --langfuse-evaluators evaluators:answer_quality \ --langfuse-managed-evaluators Correctness \ --langfuse-metric-function evaluators:qa_metric \ --metric qa_metric --goal maximize --steps 5 \ --output plain # For Jupyter-friendly formatting ``` ### Step 8: View Results Each optimization step creates a new experiment run in Langfuse. Navigate to **Datasets** β†’ **weco-demo-qa** β†’ **Runs** to compare inputs, outputs, and evaluator scores across all variants side-by-side. You can also **track iteration-by-iteration progress in the [Weco dashboard](https://dashboard.weco.ai), which shows metric scores and the exact code changes Weco made at each step**. When the run completes, you'll be prompted to apply the best-performing version to your source file. ## Full Example: End-to-End QA Optimization A complete working example with a richer dataset, multiple evaluators, and holdout validation is available in the Weco CLI repository: ```bash git clone https://github.com/WecoAI/weco-cli.git cd weco-cli/examples/langfuse-zeph-hr-qa ``` This example optimizes an HR QA agent against a dataset of policy questions, using both local code evaluators and managed LLM-as-a-Judge evaluators to measure correctness and helpfulness. **See the [full tutorial in the Weco docs](https://docs.weco.ai/docs/integrations/langfuse) for a detailed walkthrough including dataset setup, evaluator configuration, and holdout validation**. ## Troubleshooting **No experiment runs appear in Langfuse** - Verify that `LANGFUSE_SECRET_KEY`, `LANGFUSE_PUBLIC_KEY`, and `LANGFUSE_BASE_URL` are exported in the same shell session where you run `weco`. - Check that the dataset name passed to `--langfuse-dataset` matches an existing dataset in your Langfuse project. **Managed evaluator scores never arrive** - Confirm the evaluator name passed to `--langfuse-managed-evaluators` exactly matches the name in the Langfuse UI (case-sensitive). - Verify variable mappings in the evaluator setup by using the live preview on a historical trace. - Increase the polling timeout with `--langfuse-managed-evaluator-timeout` if evaluators are slow. **Metric stays at 0** - Print intermediate evaluator outputs to check score scales and key names. - Confirm that the keys used in your metric function (`scores.get("Correctness")`) match the evaluator names exactly. **Auth errors** - Re-check API keys and confirm the base URL matches your region (EU vs US). - Verify project-level key permissions in Langfuse project settings. ## Learn More - [Weco documentation](https://docs.weco.ai/) β€” full CLI reference and advanced configuration. - [Weco CLI GitHub](https://github.com/WecoAI/weco-cli) β€” source code and example projects. - [Langfuse Datasets documentation](https://langfuse.com/docs/datasets/overview) β€” creating and managing datasets. - [Langfuse Evaluation documentation](https://langfuse.com/docs/scores/overview) β€” configuring evaluators and scores. --- ## Agent Instructions This page is part of the [Langfuse](https://langfuse.com) documentation, published as plain Markdown for AI agents. Every page is available as Markdown by appending `.md` to its URL, or by sending an `Accept: text/markdown` header. This page: `https://langfuse.com/integrations/other/weco.md`. ### Querying these docs If the answer is not on this page, query the documentation instead of guessing: - **Semantic search** across all Langfuse docs, returning an answer with the relevant pages and excerpts. Ask a specific, self-contained question: ```bash curl -sG "https://langfuse.com/api/search-docs" --data-urlencode "query=How do I trace a LangGraph agent?" ``` - **Index of every page**: , with per-section indexes [llms-docs.txt](https://langfuse.com/llms-docs.txt), [llms-integrations.txt](https://langfuse.com/llms-integrations.txt), and [llms-self-hosting.txt](https://langfuse.com/llms-self-hosting.txt). ### Before writing Langfuse code - **Install the [Langfuse Agent Skill](https://langfuse.com/docs/api-and-data-platform/features/agent-skill).** It encodes Langfuse's own best practices for instrumentation, prompt management, and evaluation, and materially improves results. - **Read [What does a good trace look like?](https://langfuse.com/docs/observability/best-practices.md)** before instrumenting an application. - **Verify endpoints, parameters, and response fields** against the [API reference](https://api.reference.langfuse.com) instead of inferring them from code examples. - **Use the [Langfuse CLI](https://langfuse.com/docs/api-and-data-platform/features/cli)** (`npx langfuse-cli api `) to read or write traces, prompts, datasets, and scores from the terminal. Found an error in these docs? Please open an issue at .