Data Analytics — Next cohort Starts Oct 24

Enroll Now

LLM Evaluation & Testing

Build practical skills to measure and improve LLM application quality. Learn to create evaluation datasets, automate tests with Python, design human reviews and LLM judges, assess retrieval and grounding, and test prompt injection, conversation robustness, and tool use. Apply your learning through two complete portfolio projects with runnable Python code and worked solutions.

author

Rutvik Acharya

Principal Data Scientist

Atlassian

Loading modules...

What You'll Learn

Create evaluation datasets, reference answers, and scoring rubrics that reflect real business requirements.

Automate Python tests and measure response quality, latency, and cost.

Evaluate RAG retrieval, grounding, and citations using human reviews and calibrated LLM judges.

Test prompt injection, conversation robustness, and tool actions, then establish regression checks and production monitoring.

Who Should Attend

AI developers building LLM applications, RAG systems, or tool-using agents.

QA engineers expanding into testing AI responses and application behavior.

Data professionals with basic Python knowledge who want practical LLM evaluation skills.

Product managers and technical professionals familiar with LLMs who want to define quality criteria and assess release readiness.

CERTIFICATION

Certificate of Completion

Certificate of Participation
Course
8 Modules
8 Hours
25 Lessons
25 Challenges
Language: English
PREMIUM

Explore Gradient Data Analytics Program and unlock a world of benefits

Structured Curriculum

Experience expert-led learning with our industry-proven curriculum

1:1 Mentorship Sessions

Learn from industry experts with our personalised 1:1 Mentorship

Career Support

We're dedicated to helping you achieve your career goals

I'm Interested

FAQ

FREQUENTLY ASKED QUESTIONS

You will learn to create evaluation datasets, define scoring rubrics, automate checks with Python, evaluate RAG applications, calibrate LLM judges, test tool-using agents, and establish regression checks and production monitoring.
The course is designed for AI developers, QA engineers, data professionals, and product managers who want to understand and evaluate LLM application quality.
Basic Python knowledge and familiarity with LLMs are recommended. The course begins with evaluation fundamentals before progressing to automated testing, RAG evaluation, and agent testing.
The course contains eight modules and 25 lessons, including a portfolio module with two complete worked projects. Lessons include practical explanations, examples, exercises with answers, and diagrams.
No paid AI tools or API keys are required for the included offline exercises and Python projects. They use authored response fixtures, recorded traces, and mocked tools.
Yes. You will work with a runnable evaluation harness that checks structured outputs and business rules. The portfolio projects also include executable Python code and result reports.
Yes. You will evaluate retrieval relevance and evidence coverage, assess answer correctness and grounding, verify citation support, and diagnose failures when evidence is missing or misused.
Yes. The course covers consistent human reviews, scoring rubrics, judge instructions, and calibration against reviewed labels. You will also learn to interpret agreement, failure precision, recall, and unresolved judgments.
Yes. You will study prompt injection, sensitive-data exposure, conversation robustness, and session isolation. Tool-use testing covers authorization, confirmation, arguments, retries, duplicate prevention, and resulting state.
The first project evaluates recorded RAG support-assistant responses, evidence coverage, and citations. The second tests a mock order-support backend and scripted planner for authorized access, confirmation, injection resistance, and duplicate prevention. Both include baseline failures, worked corrections, runnable code, and reports.
Scroll Up
GET FREE CAREER COUNSELING