Optimizing foundation models
Evaluate Results
The two most common ways to evaluate a generative AI model, and why you usually use both:
| Human evaluation | Benchmark datasets | |
|---|---|---|
| Kind | Qualitative | Quantitative and objective |
| Measures | User experience, contextual appropriateness, creativity and flexibility | Accuracy, speed and efficiency, scalability |
| Best for | Iterative tuning to meet user expectations | Initial testing and comparing models or model versions |
Building a benchmark dataset
- Subject matter experts (SMEs) write relevant, challenging questions about the topic or documents.
- SMEs provide the correct answers and the context they come from.
- The model answers the questions and its answers are scored against the SMEs' answers.
Scoring can be automated with LLM as a judge: another LLM compares the model's answers to the benchmark answers.
Combined approach
Benchmarks show technical capability, and humans show real-world usefulness. AnyCompany tests against benchmarks before production and collects human ratings after, so the model keeps improving.