Docs
Model Usage & Cost

Model Usage & Cost

Across Langfuse, usage and cost are tracked for LLM generations:

  • Usage: token/character counts
  • Cost: USD cost of the generation

Both usage and cost can be either

  • ingested via API, SDKs or integrations
  • or inferred based on the model parameter of the generation. Langfuse comes with a list of predefined popular models and their tokenizers including OpenAI, Anthropic, and Google models. You can also add your own custom model definitions or request official support for new models via GitHub.

Ingested usage and cost are prioritized over inferred usage and cost:

Ingest usage and/or cost

If available in the LLM response, ingesting usage and/or cost is the most accurate and robust way to track usage in Langfuse:

langfuse.generation({
  ...
  usage: {
    // usage
    input: integer,
    output: integer,
    total: integer, // if no separate input&output, otherwise derived from input + output
    unit: ModelUsageUnit, // "TOKENS" | "CHARACTERS" | "MILLISECONDS" | "SECONDS" | "IMAGES"
 
    // usd cost
    inputCost: number
    outputCost: number
    totalCost: number
  },
  ...
})

Compatibility with OpenAI

For increased compatibility with OpenAI, you can also use the following attributes to ingest usage:

langfuse.generation({
  usage: {
    // usage
    promptTokens: integer,
    completionTokens: integer,
    totalTokens: integer, // optional, derived from prompt + completion
  },
});

Infer usage and/or cost

If either usage or cost are not ingested, Langfuse will attempt to infer the missing values based on the model parameter of the generation. This is especially useful for some model providers or self-hosted models which do not include usage or cost in the response.

Langfuse comes with a list of predefined popular models and their tokenizers including OpenAI, Anthropic, Google. Check out the full list (opens in a new tab) (you need to sign-in).

You can also add your own custom model definitions (see below) or request official support for new models via GitHub.

Usage

If a tokenizer is specified for the model, Langfuse automatically calculates token amounts for ingested generations.

The following tokenizers are currently supported:

TokenizerPackage
openaijs-tiktoken (opens in a new tab)
claude@anthropic-ai/tokenizer (opens in a new tab)

Cost

Model definitions include prices per unit (input, output, total).

Langfuse automatically calculates cost for ingested generations if (1) usage is ingested or inferred, (2) and a matching model definition includes prices.

Custom model definitions

You can flexibly add your own model definitions to Langfuse. This is especially useful for self-hosted or fine-tuned models which are not included in the list of Langfuse maintained models.

Add model in UI

Models are matched to generations based on:

Generation AttributeModel AttributeNotes
modelmatch_patternUses regular expressions, e.g. (?i)^(gpt-4-0125-preview)$ matches gpt-4-0125-preview.
unitunitUnit on the usage object of the generation (e.g. TOKENS or CHARACTERS) needs to match.
start_timestart_timeOptional, can be used to update the price of a model without affecting past generations. If multiple models match, the model with the most recent model.start_time that is earlier than generation.start_time is used.

User-defined models take priority over models maintained by Langfuse.

Further details

When using the openai tokenizer, you need to specify the following tokenization config. You can also copy the config from the list of predefined OpenAI models. See the OpenAI documentation (opens in a new tab) for further details. tokensPerName and tokensPerMessage are required for chat models.

{
  "tokenizerModel": "gpt-3.5-turbo", // tiktoken model name
  "tokensPerName": -1, // OpenAI Chatmessage tokenization config
  "tokensPerMessage": 4 // OpenAI Chatmessage tokenization config
}

Troubleshooting

Usage and cost are missing for historical generations. Except for changes in prices, Langfuse does not retroactively infer usage and cost for existing generations when model definitions are changed. You can request a batch job (Langfuse Cloud) or run a script (self-hosting) to apply new model definitions to existing generations.

Was this page useful?

Questions? We're here to help

Subscribe to updates