NYC Airflow Meetup at Datadog!

About this event

Join fellow Airflow enthusiasts and leaders at Datadog's offices for an evening of engaging talks, great food and drinks, and awesome swag, co-hosted by Astronomer and Datadog! PRESENTATIONS Talk 1: Apache Airflow for Production-ready AI Speaker: George Yates, Field Engineering Manager at Astronomer Running AI workloads in production takes more than a model call. This talk shows how Apache Airflow's Common AI provider brings production-ready AI orchestration into your DAGs: single-turn LLM calls, multi-turn agents with tools, and building blocks for RAG, SQL generation, and file analysis. You'll also see how to productionize it all with durable execution, usage limits, and human-in-the-loop review. Talk 2: Multi-Tenant Airflow at Scale Speaker: Conlan Cesar, Analytics & Data Platforms, Software Engineer at Datadog Datadog is a world-class data platform ingesting more than a 100 trillion events a day, providing real-time insights. Since our internal adoption of Airflow following the release of 3.0.0, the number of teams relying on our internal Airflow platform have grown organically and quickly. This internal Airflow adoption came with a number of platform challenges, requiring novel solutions which could support multi-tenancy, scalability, and bespoke runtime environments. In this talk, we cover the changes we've implemented to allow for a secure and scalable multi-tenant Airflow. This includes plugins for the secrets backend to make different credentials available to Operators transparently, how we inject short-lived JWT tokens without needing to modify operators, methods to scale workers according to live demand with Datadog Autoscaling, and upstream contributions to the triggerer to allow for specific queue assignment. AGENDA 5:30-6:30 PM: Networking, food and drinks 6:30-7:00 PM: Talk 1 - Apache Airflow for Production-ready AI, George Yates (Astronomer) 7:00-7:30 PM: Talk 2 - Multi-Tenant Airflow at Scale, Conlan Cesar (Datadog) 7:30-8:30 PM: Networking