Want to create an interactive transcript for this episode?
Podcast: DataTalks.Club
Episode: Build Your Own Data Pipeline - Andreas Kretz
Description: We talked about:
Andreas’s background
Why data engineering is becoming more popular
Who to hire first – a data engineer or a data scientist?
How can I, as a data scientist, learn to build pipelines?
Don’t use too many tools
What is a data pipeline and why do we need it?
What is ingestion?
Can just one person build a data pipeline?
Approaches to building data pipelines for data scientists
Processing frameworks
Common setup for data pipelines — car price prediction
Productionizing the model with the help of a data pipeline
Scheduling
Orchestration
Start simple
Learning DevOps to implemen...