LLM as a Meta-Judge: Synthetic Data for NLP Evaluation Metric Validation

Lukáš Eigler    Jindřich Libovický    David Hurych

ACL SRW 2026

Abstract

Validating evaluation metrics for NLG typically relies on expensive and time-consuming human annotations, which predominantly exist only for English datasets. We propose LLM as a Meta-Judge, a scalable framework that utilizes LLMs to generate synthetic evaluation datasets via controlled semantic degradation of real data, replacing human judgment. We validate our approach using meta-correlation, measuring the alignment between metric rankings derived from synthetic data and those from standard human benchmarks. Experiments across Machine Translation, Question Answering, and Summarization demonstrate that synthetic validation serves as a reliable proxy for human judgment, achieving meta-correlations exceeding 0.9 in multilingual QA and proves to be a viable alternative where human judgments are unavailable or too expensive to obtain.

BibTeX

@inproceedings{eigler2026metajudge,
  title     = {{LLM} as a Meta-Judge: Synthetic Data for {NLP} Evaluation Metric Validation},
  author    = {Eigler, Luk{\'a}{\v s} and Libovick{\'y}, Jind{\v r}ich and Hurych, David},
  booktitle = {Proceedings of the ACL 2026 Student Research Workshop (SRW)},
  year      = {2026}
}