🔬 GenAI Evals, Monitoring, and Automated Prompt Engineering with MLFlow, DSPy

Date: TBA Location: TBA, Melbourne Presented by: MLAI

Join us for an evening diving deep into the practical side of productionizing GenAI — the unglamorous but critical work that separates proof-of-concept demos from reliable, production-grade AI systems.

We'll explore how to evaluate LLM outputs at scale, monitor model behavior in the wild, and automatically optimize prompts using cutting-edge open-source tools like MLFlow and DSPy. Whether you're struggling with inconsistent responses, trying to catch regressions before users do, or tired of manually tweaking prompts, this session is for you.

Expect practical demos, real-world case studies, and honest conversations about what actually works (and what doesn't) when taking GenAI from experimentation to production. This is for builders who want to ship AI systems they can trust — and sleep soundly at night.

There'll be talks, hands-on demonstrations, and plenty of time to network over drinks and dinner with Melbourne's AI engineering community.

🎯 What You'll Learn

How to systematically evaluate LLM quality (beyond vibes)
Setting up monitoring pipelines to catch model drift and failures
Automated prompt optimization with DSPy
Integrating evals and monitoring into your MLOps workflow with MLFlow
Real patterns and anti-patterns from production GenAI systems

👥 Who Should Attend

AI Engineers and Data Scientists working with LLMs
Product teams building GenAI features
Anyone trying to make their AI systems more reliable and maintainable
Curious developers who want to level up their GenAI engineering game

Come ready to learn, ask questions, and connect with others solving the same hard problems. See you there! 🚀