Model evaluation
Introduction
When evaluating the performance of language models, especially those involved in generating or transforming text, specific metrics can be used. These metrics are made to assess the quality of the output, compared to a human-written standard. Three commonly used metrics for this purpose are Recall-Oriented Understudy for Gisting Evaluation (ROUGE), Bilingual Evaluation Understudy (BLEU), and BERTScore.
ROUGE
ROUGE is a set of metrics used to evaluate automatic summarization of texts, in addition to machine translation quality in NLP. The main idea behind ROUGE is to count the number of overlapping units. This includes words, N-grams, or sentence fragments between the computer-generated output and a set of reference (human-created) texts.
The following are two ways to use the ROUGE metric:
- ROUGE-N: This metric measures the overlap of n-grams between the generated text and the reference text. For example, ROUGE-1 refers to the overlap of unigrams, ROUGE-2 refers to bigrams, and so on. This metric primarily assesses the fluency of the text and the extent to which it includes key ideas from the reference.
- ROUGE-L: This metric uses the longest common subsequence between the generated text and the reference texts. It is particularly good at evaluating the coherence and order of the narrative in the outputs.
ROUGE is widely used because it is not complex. It is interpretable, and correlates reasonably well with human judgment, especially when evaluating the recall aspect of summaries. The evaluations assess how much of the important information in the source texts is captured by the generated summaries.
BLEU
BLEU is a metric used to evaluate the quality of text that has been machine-translated from one natural language to another. Quality is calculated by comparing the machine-generated text to one or more high-quality human translations. BLEU measures the precision of N-grams in the machine-generated text that appears in the reference texts and applies a penalty for overly short translations (brevity penalty).
Unlike ROUGE, which focuses on recall, BLEU is fundamentally a precision metric. It checks how many words or phrases in the machine translation appear in the reference translations. BLEU evaluates the quality at the level of the sentence, typically using a combination of unigrams, bigrams, trigrams, and quadrigrams. A brevity penalty discourages overly concise translations that might influence the precision score.
BLEU is popular in the field of machine translation for its ease of use and effectiveness at a broad scale. However, it has limitations in assessing the fluency and grammaticality of the output.
The BERTScore
BERTScore uses the pretrained contextual embeddings from models like BERT to evaluate the quality of text-generation tasks. BERTScore computes the cosine similarity between the contextual embeddings of words in the candidate and the reference texts. This is unlike traditional metrics that rely on exact matches of N-grams or words.
Because BERTScore evaluates the semantic similarity rather than relying on exact lexical matches, it is capable of capturing meaning in a more nuanced manner. BERTScore is less prone to some of the pitfalls of BLEU and ROUGE. An example of this is their sensitivity to minor paraphrasing or synonym usage that does not affect the overall meaning conveyed by the text.
BERTScore is increasingly used alongside traditional metrics like BLEU and ROUGE for a more comprehensive assessment of language generation models. This is especially true in cases where capturing the deeper semantic meaning of the text is important.
AnyCompany model evaluation
2: BLEU
2: BLEU
Bleu can be used in combination with ROUGE to also ensure the accurate inclusion of critical product features and key phrases that might influence purchase decisions.
In addressing AnyCompany's challenges of high cart abandonment and low repeat purchases, the integration of an FM using generative AI has demonstrated substantial improvements in key business metrics. Post-implementation, the conversion rate increased by 15 percent, thanks to more engaging product descriptions. This enhancement was quantitatively supported by ROUGE scores averaging 0.85. This indicates a high overlap of generated content with reference material, ensuring completeness and relevance that directly encouraged purchases. The average order value saw a 20 percent uplift. This is influenced by the precision and accuracy of technical terms and persuasive language in product descriptions, validated by a BLEU scores of 0.78. These scores are considered good in this context, because they suggest a strong correspondence with the quality of language that historically leads to higher sales.Additionally, the customer retention rate improved by 25 percent, supported by BERTScore evaluations averaging 0.90. This high score reflects excellent semantic quality of the personalized shopping advice and product displays. This indicates the content is not only accurate, but deeply relevant to individual preferences, thus enhancing customer satisfaction and loyalty.These metrics, by confirming the high quality and relevance of the AI-generated content, have played a paramount role in achieving AnyCompany’s goals. These goals included enhancing user engagement, reducing cart abandonment, and boosting repeat purchases through improved personalization and content accuracy.
You have completed this lesson on model evaluation. In the next lesson you will test your knowledge.