Evaluating generative AI applications: Review

The 'Custom prompt-based evaluators' section just ended abruptly, with no code, no sandbox, not even a fake YAML file to look at. It describes what a prompt-based evaluator does—using an LLM to judge another LLM's output—which is a concept I'd already seen on blogs. I was hoping for more structure. The module does introduce terms like groundedness, coherence, and relevance, which is genuinely useful. Getting the vocabulary right is half the battle, especially when you're trying to read papers or understand what senior engineers are talking about. Since it's free, it’s a no-risk way to get that vocabulary.

But the whole thing is just a text document with a multiple-choice quiz at the end. The quiz itself was a bit of a letdown; it just checks if you remember the definitions. For a module aimed at developers, I expected at least a small code snippet showing how you might structure a call to Azure AI Studio or use an SDK to run an evaluation. Instead, it just says 'Azure AI Studio provides tools...' and moves on. I finished it in about 30 minutes. It's like reading a good Wikipedia article that's been formatted into learning units. I'm glad I did it, and it gives me the right keywords to search for next, but it's more of a map than a vehicle.