The Financial Times leverages Airflow and AI to uncover powerful stories hidden within vast, unstructured data.

In this episode, Zdravko Hvarlingov, Senior Software Engineer at the Financial Times, discusses building multi-tenant Airflow systems and AI-driven pipelines that surface stories that might otherwise be missed. Zdravko walks through entity extraction and fuzzy matching, linking the UK Register of Members’ Financial Interests with Companies House, and how this work cuts weeks of manual analysis to minutes.

Key Takeaways:

00:00 Introduction.

02:12 What computational journalism means for day-to-day newsroom work.

05:22 Why a shared orchestration platform supports consistent, scalable workflows.

08:30 Tradeoffs of one centralized platform versus many separate instances.

11:52 Using pipelines to structure messy sources for faster analysis.

14:14 Turning recurring disclosures into usable data for investigations.

16:03 Applying lightweight ML and matching to reveal entities and links.

18:46 How automation reduces manual effort and shortens time to insight.

20:41 Practical improvements that make backfilling and reliability easier.

Resources Mentioned:

Zdravko Hvarlingov

https://www.linkedin.com/in/zdravko-hvarlingov-3aa36016b/

Financial Times | LinkedIn

https://www.linkedin.com/company/financial-times/

Financial Times | Website

https://www.ft.com/

Apache Airflow

https://airflow.apache.org/

UK Register of Members’ Financial Interests

https://www.parliament.uk/mps-lords-and-offices/standards-and-financial-interests/parliamentary-commissioner-for-standards/registers-of-interests/register-of-members-financial-interests/

UK Companies House

https://www.gov.uk/government/organisations/companies-house

Doppler

https://www.doppler.com/

Kubernetes

https://kubernetes.io/

Airflow Kubernetes Executor

https://airflow.apache.org/docs/apache-airflow/stable/executor/kubernetes.html

GitHub

https://github.com/

Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.

#AI #Automation #Airflow #MachineLearning