Firecrawl Integration 🔥🤝🪢

In this guide, we’ll show you how to integrate Langfuse with Firecrawl to trace your web scraping operations. By leveraging Langfuse’s tracing capabilities, you can automatically capture details such as inputs, outputs, and execution times of your Firecrawl functions.

What is Firecrawl? Firecrawl is an API service that crawls websites—automatically processing the target URL and its accessible subpages—and converts the retrieved content into clean, LLM-ready markdown. It offers robust scraping, crawling, and extraction capabilities that transform web content (including HTML, metadata, and more) into structured data for various applications.

Get Started

First, install the necessary Python packages:

%pip install langfuse firecrawl-py

Next, configure your environment with your Langfuse API keys. You can obtain your keys from your Langfuse dashboard.

import os
# Get keys for your project from the project settings page https://cloud.langfuse.com
 
os.environ["LANGFUSE_PUBLIC_KEY"] = "pk-lf-..."
os.environ["LANGFUSE_SECRET_KEY"] = "sk-lf-..." 
os.environ["LANGFUSE_HOST"] = "https://cloud.langfuse.com" # 🇪🇺 EU region
# os.environ["LANGFUSE_HOST"] = "https://us.cloud.langfuse.com" # 🇺🇸 US region

Initialize the Firecrawl application with your Firecrawl API key.

from firecrawl import FirecrawlApp
 
app = FirecrawlApp(api_key="fc-...")

To monitor your Firecrawl scraper, we use the Langfuse @observe() decorator. In this example, the @observe() decorator captures the inputs, outputs, and execution time of the scrape_website() function. All trace data is automatically sent to Langfuse, allowing you to monitor your Firecrawl operations in real time.

from langfuse.decorators import observe
 
@observe()
def scrape_website(url):
  scrape_status = app.scrape_url(
    url,
    params={'formats': ['markdown', 'html']}
  )
  return scrape_status
 
scrape_website("https://langfuse.com")

See Traces in Langfuse

After executing the traced function, log in to your Langfuse Dashboard to view detailed trace logs.

Public link to example trace in Langfuse

Learn More

• Firecrawl Documentation: https://docs.firecrawl.dev/introduction
• Langfuse @observe() Decorator: https://langfuse.com/docs/sdk/python/decorators