The LLM Evaluation guidebook ⚖️

!! THIS GUIDEBOOK IS NO LONGER MAINTAINED. THE LATEST AND MOST UP TO DATE VERSION (as of Dec 2025) OF IT LIVES HERE: https://huggingface.co/spaces/OpenEvals/evaluation-guidebook

---

If you've ever wondered how to make sure an LLM performs well on your specific task, this guide is for you! It covers the different ways you can evaluate a model, guides on designing your own evaluations, and tips and tricks from practical experience.

Whether working with production models, a researcher or a hobbyist, I hope you'll find what you need; and if not, open an issue (to suggest ameliorations or missing resources) and I'll complete the guide!

How to read this guide

• Beginner user: If you don't know anything about evaluation, you should start by the Basics sections in each chapter before diving deeper. You'll also find explanations to support you about important LLM topics in General knowledge: for example, how model inference works and what tokenization is.
• Advanced user: The more practical sections are the Tips and Tricks ones, and Troubleshooting chapter. You'll also find interesting things in the Designing sections.
• User coming back to the site: Every year I do a dive on a topic, check them out!

In text, links prefixed by ⭐ are links I really enjoyed and recommend reading.

Table of contents

If you want an intro on the topic, you can read this blog on how and why we do evaluation!

Automatic benchmarks

• Basics
• Designing your automatic evaluation
• Some evaluation datasets
• Tips and tricks

Human evaluation

• Basics
• Using human annotators
• Tips and tricks

LLM-as-a-judge

• Basics
• Getting a Judge-LLM
• Designing your evaluation prompt
• Evaluating your evaluator
• What about reward models
• Tips and tricks

Troubleshooting

The most densely practical part of this guide.

• Troubleshooting inference
• Troubleshooting reproducibility

General knowledge

These are mostly beginner guides to LLM basics, but will still contain some tips and cool references! If you're an advanced user, I suggest skimming to the Going further sections.

• Model inference and evaluation
• Tokenization

Yearly dives

• 2023, year of Open Source
• 2024, what should evaluation be for?
• 2025, evaluations to build "real life" useful models

Resources

Links I like

• About evaluation
• About general NLP
• The UltraScale Playbook

Community translations

This guide has been kindly community translated!

• 🇨🇳 https://github.com/huggingface/evaluation-guidebook/tree/main/translations/zh/contents, thanks to @SuSung-boy
• 🇫🇷 https://huggingface.co/spaces/CATIE-AQ/Guide_Evaluation_LLM, thanks to @lbourdois

Thanks

This guide has been heavily inspired by the ML Engineering Guidebook by Stas Bekman! Thanks for this cool resource!

Many thanks also to all the people who inspired this guide through discussions either at events or online, notably and not limited to:

• 🤝 Luca Soldaini, Kyle Lo and Ian Magnusson (Allen AI), Max Bartolo (Cohere), Kai Wu (Meta), Swyx and Alessio Fanelli (Latent Space Podcast), Hailey Schoelkopf (EleutherAI), Martin Signoux (Open AI), Moritz Hardt (Max Planck Institute), Ludwig Schmidt (Anthropic)
• 🔥 community users of the Open LLM Leaderboard and lighteval, who often raised very interesting points in discussions
• 🤗 people at Hugging Face, like Lewis Tunstall, Hynek Kydlíček, Guilherme Penedo and Thom Wolf, and of course my teammate Nathan Habib with whom I've been doing evaluation and leaderboards since 2022

and of course to all the contributors :)

Citation

@misc{fourrier2024evaluation,
  author = {Clémentine Fourrier and The Hugging Face Community},
  title = {LLM Evaluation Guidebook},
  year = {2024},
  journal = {GitHub repository},
  url = {https://github.com/huggingface/evaluation-guidebook)
}