| Intro to LLM Evals | Introduction to LLM evaluation: concepts, goals, and motivations behind evaluating LLM outputs. | – | |
| LLM Evaluation Methods | Tutorial with an overview of methods. - Part 1. Anatomy of a single evaluation. Covers basic LLM evaluation API and setup.
- Part 2. Reference-based evaluation: exact match, semantic similarity, BERTScore, and LLM judge.
- Part 3. Reference-free evaluation: text statistics, regex, ML models, LLM judges, and session-level evaluators.
| Open Notebook | |
| LLM as a Judge | Tutorial on creating and tuning LLM judges aligned with human preferences. | Open Notebook | |
| Clasification Evaluation | Tutorial on evaluating LLMs and a simple predictive ML baseline on a multi-class classification task. | Open Notebook | |
| Content Generation with LLMs | Tutorial on how to use LLMs to write tweets and evaluate how engaging they are. Introduction to the concept of tracing. | Open Notebook | |
| RAG evaluations | - Part 1. Theory on how to evaluate RAG systems: retrieval, generation quality and synthetic data.
- Part 2. Tutorial on building a toy RAG application and evaluating correctness and faithfulness.
| Open Notebook | |
| AI agent evaluations | Tutorial on how to build a simple Q&A agent and evaluate tool choice and answer correctness. | Open Notebook | |
| Adversarial testing | Tutorial on how to run scenario-based risk testing on forbidden topics and brand risks. | Open Notebook | |
