LLM evaluation

Without measurement, every change to prompt, model or retrieval is a gut feeling. An evaluation consists of fixed example cases with expected outcomes and a scoring step, automated or human.

For many applications a few dozen well-chosen cases are enough, covering the typical and the dangerous situations.