Data pipelines are necessary for the flow of information from the source to the consumers, typically data scientists, analysts, and software developers. Managing the flow of data from many sources is a complex task with maintenance costs limiting the scale of building a large, reliable data warehouse. This presentation presents a set of applied data engineering principles that can be used to build robust, easily manageable data pipelines and data products. Examples are shown using Python on AWS.

Sponsored and hosted by Man AHL

****

www.pydata.org

PyData is an educational program of NumFOCUS, a 501(c)3 non-profit organization in the United States. PyData provides a forum for the international community of users and developers of data analysis tools to exchange ideas and learn from each other. The global PyData network promotes discussion about best practices, new approaches and emerging technologies for data management, processing, analysis and visualization. PyData communities approach data science using many languages, including (but not limited to) Python, Julia, and R.

PyData conferences strive to be accessible and community-driven, with presentations at beginner to advanced levels. PyData tutorials and lectures provide participants with the latest project features and advanced use cases. 00:00 Welcome!
00:10 Help us add timestamps or captions to this video! See description for details.

Want to help add timestamps to our YouTube videos to increase discoverability? Read more here: https://github.com/numfocus/YouTubeVideoTimestamps

Please take the opportunity to connect and share this video with your friends and family if you find it helpful.