How to Create and Manage ScoreConfigs?
ScoreConfigs are used to ensure that your scores follow a specific schema. Using ScoreConfigs allows you to standardize your scoring schema across your team and ensure that scores are consistent and comparable for future analysis.
ScoreConfigs are required for using Human Annotation and optional for Custom Evaluation via API/SDK.
ScoreConfigs are immutable. However, you can archive configs if you no longer want to use them in annotation. Archived configs can be restored at any time.
To understand detailed properties of the ScoreConfig object, see Scores Data Model.
Via SDK
Create a Score Config
To create a scoreConfig via the Langfuse SDK you need to use the api property on the Langfuse client as specified here. The specific POST and GET API endpoints are documented in the API reference.
Use a Score Config
Using ScoreConfig via the SDK most commonly happens in the context of Custom Evaluation via API/SDK (find instructions how to do that here).
Via Langfuse UI
Create a Score Config
Go to Project Settings
Navigate to Your Project > Project Settings > Scores / Evaluation and click Create new score config.
Add new score config
Set the values:
- Score name:
- Data type: NUMERIC, CATEGORICAL, BOOLEAN
- Constraints on score value range (Min/Max for numerical, Custom categories for categorical data)
Click “submit” to create the score config.
Use a Score Config
Using ScoreConfig via the UI most commonly happens in the context of Human Annotation.
Archive a score config
Go to Project Settings
Navigate to Your Project > Project Settings > Scores / Evaluation and click on the “archive” icon of the score config you want to archive.
Archive score config
- Archiving a score config will ensure that the score config is no longer selectable for Human Annotation.
- You will be able to restore a config by clicking the “archive” icon of an archived config.
- Scores associated with an archived config will still be shown in your project but can be deleted.