Evaluation
All articles
35 total
Agent-as-a-Judge New
An evaluation approach that uses a full agentic system to judge another …
Glossary
Added 6 Jul
·
Upd 6 Jul
·2 min
Trajectory Evaluation New
Scoring the full path an agent takes, including tool calls and …
Glossary
Added 6 Jul
·
Upd 6 Jul
·2 min
Context Rot New
The measurable degradation of an LLM's output quality as its input grows …
Glossary
Added 5 Jul
·
Upd 5 Jul
·3 min
AI Benchmark New
A standardized test that runs the same inputs through one or more AI …
Added 1 Jul
·
Upd 1 Jul
·6 min
How AI Models Are Evaluated: The Hidden Lifecycle New
The invisible pipeline behind every chat box: training, alignment, red …
Guides
Added 1 Jul
·
Upd 1 Jul
·9 min
Agent Evaluation New
Agent evaluation tests an AI system that plans and calls tools across …
Added 29 Jun
·
Upd 29 Jun
·5 min
AI Evaluation New
The whole practice of judging whether an AI system is fit for use, …
Added 29 Jun
·
Upd 29 Jun
·4 min
Model Evaluation New
Model evaluation tests an AI model in isolation, measuring its raw …
Added 29 Jun
·
Upd 29 Jun
·4 min
System Evaluation New
Testing an AI model plus everything around it - retrieval, prompts, …
Added 29 Jun
·
Upd 29 Jun
·5 min
Workflow Evaluation New
Testing a chain of AI steps and hand-offs end to end, because small …
Added 29 Jun
·
Upd 29 Jun
·4 min
A/B Testing for AI Systems
How to design and run A/B tests for AI models and features, covering …
Guides
Added 28 Mar
·
Upd 30 May
·5 min
AI Red Team
A dedicated adversarial testing team that probes AI systems for …
Glossary
Added 28 Mar
·
Upd 30 May
·3 min
AI Spark: Smart Vendor Evaluation Scoring
Use AI to standardize vendor evaluation by scoring proposals against …
Ideas
Added 28 Mar
·
Upd 30 May
·2 min
Architecture Decision Records and Evaluation Methods
Using ADRs and architecture evaluation methods like ATAM to document and …
Frameworks
Added 28 Mar
·
Upd 30 May
·4 min
DeepEval vs Promptfoo for LLM Evaluation in CI
Comparing DeepEval and Promptfoo for automated LLM evaluation: metrics, …
Comparisons
Added 28 Mar
·
Upd 14 Jun
·7 min
Embedding Model Comparison and Selection Guide
How to choose embedding models for semantic search, RAG, and similarity …
Guides
Added 28 Mar
·
Upd 30 May
·4 min
Evaluating RAG System Quality
How to measure and improve both retrieval quality and generation quality …
Guides
Added 28 Mar
·
Upd 30 May
·3 min
Evaluator-Optimizer Pattern
Automated evaluation loops where one model generates output and another …
Patterns
Added 28 Mar
·
Upd 30 May
·3 min
Framework for Evaluating and Selecting AI Vendors
A structured approach to evaluating AI vendors covering technical …
Guides
Added 28 Mar
·
Upd 30 May
·3 min
Full-Stack Observability for AI Systems
How to implement comprehensive observability for AI applications …
Guides
Added 28 Mar
·
Upd 30 May
·3 min
Golden Dataset
What a golden dataset is, how it serves as a curated evaluation …
Glossary
Added 28 Mar
·
Upd 30 May
·3 min
Ground Truth
What ground truth is in machine learning, how verified correct labels …
Glossary
Added 28 Mar
·
Upd 30 May
·2 min
Langfuse
LLM Observability and Tracing
Tools
Added 24 Mar
·
Upd 30 May
·2 min
LLM Evaluation Methods
Measuring Language Model Quality
Guides
Added 28 Mar
·
Upd 30 May
·7 min
LLM-as-a-Judge
Using a language model as an automated evaluator of another model's …
Glossary
Added 8 May
·
Upd 30 May
·5 min
LLMOps Pipeline
Production pipeline design for LLM-specific operations: prompt …
Patterns
Added 28 Mar
·
Upd 30 May
·3 min
Model Cards
AI Transparency Documentation
Glossary
Added 24 Mar
·
Upd 30 May
·3 min
RAG Evaluation
Methods and metrics for measuring the quality of Retrieval Augmented …
Glossary
Added 28 Mar
·
Upd 30 May
·3 min
Red Teaming
What red teaming is in AI, how adversarial testing discovers …
Glossary
Added 28 Mar
·
Upd 30 May
·4 min
Red Teaming and Adversarial Testing for AI Systems
How to plan and execute red team exercises that systematically probe AI …
Guides
Added 28 Mar
·
Upd 30 May
·3 min
Testing AI Systems
Unit Tests to Production Monitoring
Guides
Added 25 Mar
·
Upd 30 May
·5 min
Testing and Evaluating AI Agent Performance
Frameworks for evaluating AI agents that plan, use tools, and take …
Guides
Added 28 Mar
·
Upd 30 May
·4 min
Testing LLM Applications
LLM-specific testing strategies: prompt template testing, structured …
Guides
Added 28 Mar
·
Upd 30 May
·5 min
Testing RAG Systems
How to test Retrieval-Augmented Generation systems: unit testing …
Guides
Added 28 Mar
·
Upd 30 May
·5 min
The Use Case Scoring Framework
From 57 Ideas to 3 Prototypes
Frameworks
Added 24 Mar
·
Upd 30 May
·4 min
35 articles in this section. Search for a specific topic.
Open source projects