Optimizing foundation models
Model Evaluation
Three metrics compare generated text with human-written reference text:
| Metric | Measures | Focus | Best for |
|---|---|---|---|
| ROUGE | Overlap of words, n-grams, or sequences with the reference | Recall: how much of the important information was captured | Summarization (also translation) |
| BLEU | n-gram precision against reference translations, with a brevity penalty for overly short output | Precision | Machine translation |
| BERTScore | Cosine similarity of contextual BERT embeddings | Semantic similarity: it recognizes paraphrases and synonyms | Any task where meaning matters more than exact words |
ROUGE variants
- ROUGE-N: n-gram overlap (ROUGE-1 for single words, ROUGE-2 for pairs). Measures fluency and coverage of key ideas.
- ROUGE-L: longest common subsequence. Measures coherence and order.
Limitations
- ROUGE and BLEU depend on exact matches, so they penalize valid paraphrases.
- BLEU struggles to judge fluency and grammar.
- BERTScore is often used alongside them for a fuller picture.
AnyCompany results
Better product descriptions and advice moved the business metrics:
- Conversion rate rose 15%, with ROUGE averaging 0.85.
- Average order value rose 20%, with BLEU at 0.78.
- Customer retention rose 25%, with BERTScore averaging 0.90.