Understanding Evaluation Metrics for Language Models

Key Metrics

1. Entropy

  • Definition: Entropy measures the average amount of information produced per symbol in a language. It quantifies the unpredictability or complexity of the text.
  • Formula: [ H(P) = -\sum_{x \in X} P(x) \log P(x) ] where ( P(x) ) is the probability of symbol ( x ).
  • Usage: Used to understand the inherent complexity of a language. Lower entropy indicates more predictability.
  • Sources:

2. Cross Entropy

  • Definition: Cross entropy measures the difference between the true probability distribution of the language (P) and the probability distribution predicted by the model (Q).
  • Formula: [ H(P, Q) = -\sum_{x \in X} P(x) \log Q(x) ]
  • Usage: Commonly used as a loss function during the training of language models. It helps in evaluating how well the model predicts the probability of the next symbol.
  • Sources:

3. Perplexity

  • Definition: Perplexity is the exponentiation of the cross entropy. It represents the model’s uncertainty when predicting the next symbol.
  • Formula: [ PPL(P, Q) = 2^{H(P, Q)} ]
  • Usage: A lower perplexity indicates a better model. It is widely used to compare different language models.
  • Sources:

4. Bits-per-Character (BPC)

  • Definition: BPC measures the average number of bits needed to encode each character in a text.
  • Usage: Specific to character-level models. It helps in comparing the efficiency of different models in encoding text.
  • Sources:

Comparative Insights

Perplexity vs. Cross Entropy

  • Relationship: Both metrics are closely related; perplexity is a more intuitive measure derived from cross entropy. Cross entropy provides a more direct mathematical representation of prediction error.
  • Usage: Perplexity is often reported because it is easier to interpret, while cross entropy is used during the training process.

Bits-per-Character (BPC) vs. Bits-per-Word (BPW)

  • BPC: Used for character-level models.
  • BPW: Used for word-level models. BPW can be converted to BPC using the average word length.
  • Usage: Choose the metric based on the granularity of your model (character-level or word-level).

Best Practices

  • Reporting: When reporting entropy or cross entropy, it is recommended to use bits (log base 2) for consistency.
  • Context Length: Specify the context length used in the language model to provide a clear comparison basis.
  • Symbol Type: Indicate whether the metric is for character-level, word-level, or subword-level models.

Conclusion

While intrinsic metrics like perplexity and cross entropy are useful during the training phase, evaluating language models on downstream tasks (e.g., sentiment analysis, textual entailment) is becoming increasingly important. Benchmarks like GLUE and SuperGLUE help in assessing model performance on a variety of tasks, indicating their real-world applicability.

For more detailed insights, check out the full article on The Gradient.

References