Share
The award-winning WIRED UK Podcast with James Temperton and the rest of the team. Listen every week for the an informed and entertaining rundown of latest technology, science, business and culture news. New episodes every Friday.
…
continue reading
Inside Vinted’s Code-Generated Airflow Pipelines with Oscar Ligthart and Rodrigo Loredo
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
Manage episode 515154788 series 2948506
Content provided by The Data Flowcast. All podcast content including episodes, graphics, and podcast descriptions are uploaded and provided directly by The Data Flowcast or their podcast platform partner. If you believe someone is using your copyrighted work without your permission, you can follow the process outlined here https://staging.podcastplayer.com/legal.
The shift from monolithic to decentralized data workflows changes how teams build, connect and scale pipelines.
In this episode, we feature Oscar Ligthart, Lead Data Engineer, and Rodrigo Loredo, Lead Analytics Engineer, both at Vinted, as we unpack their YAML-driven abstraction that generates Airflow DAGs and standardizes cross-team orchestration.
Key Takeaways:
00:00 Introduction.
05:28 Challenges of decentralization.
06:45 YAML-based generator standardizes pipelines and dependencies.
12:28 Declarative assets and sensors align cross-DAG dependencies.
17:29 Task-level callbacks enable auto-recovery and clear ownership.
21:39 Standardized building blocks simplify upgrades and maintenance.
24:52 Platform focus frees domain work.
26:49 Container-only standardization prevents sprawl.
Resources Mentioned:
https://www.linkedin.com/in/oscar-ligthart/
https://www.linkedin.com/in/rodrigo-loredo-410a16134/
Vinted | LinkedIn
https://www.linkedin.com/company/vinted/
Vinted | Website
https://www.vinted.com/?srsltid=AfmBOor87MGR_eLOauCO93V9A-aLDaAhGYx9cnu_oN8s1SAXMlCRuhW7
https://airflow.apache.org/
https://kubernetes.io/
https://www.getdbt.com/
https://cloud.google.com/vertex-ai
Airflow Datasets & Assets (concepts)
https://www.astronomer.io/docs/learn/airflow-datasets
https://airflowsummit.org/
Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.
#AI #Automation #Airflow #MachineLearning
77 episodes
))
