Developing generative AI solutions
Evaluating an FM
Evaluation checks whether the model meets the business objectives.
Three evaluation methods
| Method | Strengths | Weaknesses |
|---|---|---|
| Human evaluation | The gold standard. Judges coherence, relevance, factuality, and quality | Slow and expensive at scale |
| Benchmark datasets | Standardized comparison across models and over time | May not match your specific use case |
| Automated metrics | Quick and scalable for fast iteration | Miss nuance and may not match human judgment |
Benchmarks to know:
- GLUE: language understanding (classification, Q&A, inference)
- SuperGLUE: harder GLUE tasks
- SQuAD: question answering
- WMT: machine translation
Metrics to know
| Metric | Measures | Best for |
|---|---|---|
| ROUGE | Overlap between generated text and a reference, focused on recall | Summarization |
| BLEU | n-gram precision against a reference | Machine translation |
| BERTScore | Semantic similarity using BERT embeddings and cosine similarity | Any generation where meaning matters more than exact wording |
| Perplexity | How well the model predicts the next token (lower is better) | Language model quality |
| F1 score | Balance of precision and recall | Classification and entity recognition |
Automated metrics give a first read. Combine them with human evaluation for a complete picture.