Get a tour through the code of an end-to-end GenAI Airflow pipeline, using custom information in an RAG pattern and fine-tuning GPT-3.5 Turbo for your content generation use case. The pipeline code is open-source and will be provided to you.
You test all your apps; your data engineering and machine learning pipelines should be rigorously tested too! This talk will show you best practices and options to test Apache Airflow pipelines using common Python testing frameworks that can be integrated into a CI/CD pipeline.
Automate MLOps and data pipelines in one data-driven, tool agnostic workflow with Apache Airflow and the Astro Python SDK. See how Airflow orchestrates the ingestion of brain tumour MRI data into DuckDB, as well as fine-tuning a HuggingFace classifier, all in one dynamic workflow!
Learn for free, join the best tech learning community for a price of a pumpkin latte.
Event notifications, weekly newsletter
Delayed access to all content
Immediate access to Keynotes & Panels
Access to Circle community platform
Immediate access to all content
Courses, quizes & certificates
Community chats