Share
Stay current on JavaScript, Node, and Front-End development. Learn from experts in programming, careers, and technology every week. Become a supporter of this podcast: https://www.spreaker.com/podcast/javascript-jabber--6102064/support.
…
continue reading
Fetch error
Hmmm there seems to be a problem fetching this series right now.
Last successful fetch was on July 25, 2025 11:28 (
What now? This series will be checked again in the next day. If you believe it should be working, please verify the publisher's feed link below is valid and includes actual episode links. You can contact support to request the feed be immediately fetched.
Manage episode 392812462 series 2920782
Content provided by Lars Wikman and Andreas Ekeroot. All podcast content including episodes, graphics, and podcast descriptions are uploaded and provided directly by Lars Wikman and Andreas Ekeroot or their podcast platform partner. If you believe someone is using your copyrighted work without your permission, you can follow the process outlined here https://staging.podcastplayer.com/legal.
Lars dove into data pipelines, and emerged bearing arrows and wishing for a lot fewer copies.
What is there to think about regarding data pipelines, what is interesting about them?
Which tools are out there, and why might you want to use them?
Why all this talk about making fewer copies of data?
What does Lars' current ideal pipeline look like, and where does Elixir fit in?
Links
- Matt Topol
- Apache Arrow
- Large language models
- Vector search
- BigQuery
- sed
- AWK
- jq
- Replacing Hadoop with bash - "Command-line Tools can be 235x Faster than your Hadoop Cluster"
- Hadoop
- MapReduce
- Unix pipes
- Directed acyclic graph
- tee - to "materialize inbetween states"
- Apache Beam
- Apache Spark
- Apache Flink
- Apache Pulsar
- Airbyte - shoves data between systems using connectors
- Cronjob
- Fivetran - Airbyte competitor
- Apache Airflow
- ETL - Extract, transform, load
- Designing data-intensive applications
- Stream processing
- Ephemerality
- Data lake
- Data warehouse
- The people's front of Judea
- DBT - SQL-SQL batch-work-thingy
- SQL with Jinja templates
- Snowflake - data warehouse thing
- Scala
- Broadway
- Oban - "robust job processing for Elixir"
- Dashbit
- pandas - Python data library
- APL
- Arrow flight
- GRPC
- DataFusion - query execution engine
- Polars - "DataFrames in Rust"
- Explorer - built on top of Polars
- Voltron data
- The Composable Codex
- Pyarrow - Arrow bindings for Python
Quotes
- I've been reading a lot about data pipelines
- What's so special about data pipelines?
- There's a lot of special tooling
- There's a lot of bad, bad tooling
- Less than optimal tooling
- Converging on something biggerlk
- He got me eventually
- All of your steps in one bucket
- What tools do you associate with data?
- I inherited a data pipeline
- BashReduce
- Iterate on the L and the T
- The modern data stack
- And then you demand more work
- No unnecessary copies
- Barely a copy
- Reconnecting with my Python roots
65 episodes
))
