How to evaluate or test the product powered by large language models

About this event

This meetup is moved to Nov 10th. Join our newsletter to stay informed about Innovate QA’s virtual and in-person events, opportunities, news, and offers. Subscribe to the Innovate QA Newsletter (https://preview.mailerlite.io/forms/1828078/166995591270762256/share) Speaker: Narendra Singh Panwar | LinkedIn (https://www.linkedin.com/in/narendra-singh-panwar-0a55aa52/) Topic: How to Evaluate or Test Products Powered by Large Language Models Agenda: 🕛 4:00–4:05 PM PDT — Welcome 🕛 4:05–4:55 PM PDT — Narendra Singh Panwar | LinkedIn (https://www.linkedin.com/in/narendra-singh-panwar-0a55aa52/) \- How to Evaluate or Test Products Powered by Large Language Models 🕛 4:55–5:00 PM PDT — Q&A & Closing Summary : LLMs and AI agents are powering the next generation of products. While the speed of development has increased significantly, evaluating the end-to-end quality of these products remains a challenge. In this session, Narendra Singh Panwar will share practical learnings and techniques from testing consumer-facing products powered by LLMs and AI agents. The discussion will explore how traditional QA practices need to evolve when testing systems that are non-deterministic and capable of making decisions, generating responses, and completing actions. The session will cover how teams can evaluate the non-deterministic nature of AI systems, verify factual accuracy, and determine where AI-assisted evaluation can help. Narendra will also discuss the use of LLM-as-a-Judge and where human evaluation is still necessary to assess the quality of AI-powered experiences. The discussion will also look at AI-powered test automation and continuous customer experience monitoring as ways to support quality evaluation as AI products evolve. Takeaways : What attendees will learn: 1. How to evaluate the non-deterministic behavior of LLM-powered products 2. Approaches for fact-checking and evaluating AI-generated responses 3. Where AI and LLM-as-a-Judge can be applied in quality evaluation 4. Where human evaluation is still necessary 5. How AI-powered test automation can support testing of AI products 6. How continuous CX monitoring can help evaluate end-to-end quality 7. How traditional QA practices can evolve for products powered by LLMs and AI agents 🎙️ Speaker Opportunities at Our Meetups Have expertise in QA, leadership, or tech skills you'd love to share? We're always looking for engaging speakers for our monthly meetups. This is a fantastic platform to showcase your knowledge, connect with the community, and inspire others. If you're interested submit your topic here (https://docs.google.com/forms/d/e/1FAIpQLSfrTCU-5qxxEwWKydzFRclbgiCOEsNOA6_d2bjLBi6K6cCHgA/viewform?usp=publish-editor) — we’d love to hear from you!