In this data engineering project, we will learn how to build and automate an ETL process that can extract current weather data from the open weather map API, transform the data, and load the data into an S3 bucket using Apache Airflow. Apache Airflow is an open source platform used for orchestrating and scheduling workflows of tasks and data pipelines. This project will be executed entirely on the AWS cloud platform.
We'll cover the fundamental concepts of Apache Airflow, such as DAG and Operators, and I'll show you how to install Apache Airflow from scratch and plan your ETL pipeline. I will also show you how to use a sensor in your ETL pipeline.
Because this is a hands-on project, I highly encourage you to first watch the video in its entirety without following along so you can better understand the concepts and workflows. Then you can try to replicate the example I showed without watching the video. video, but refer to the video if you're stuck, or you can watch the video in its entirety a second time and follow along this time.

Remember, the best way to learn is to do it yourself – get your hands dirty!
If you have any questions or concerns, please ask or leave comments in the comments section below.

*Books I Recommend*
1. Grit: the power of passion and perseverance https://amzn.to/3EZKSgb
2. Think and Get Rich!: The Original Version, Restored and Revised: https://amzn.to/3Q2K68s
3. The Rental Property Investing Book: How to Create Wealth with Intelligent Real Estate Investing: https://amzn.to/3LLpXRy
4. How to Invest in Real Estate: The Ultimate Beginner's Guide to Getting Started: https://amzn.to/48RbuOb
5. Introducing Python: Modern Computing in Simple Packages https://amzn.to/3Q4driR
6. Python for Data Analytics: Data Wrangling with Pandas, NumPy and Jupyter 3rd Edition: https://amzn.to/3rGF73G

***************** Commands used in this video *****************
sudo apt update
sudo apt install python3-pip
sudo apt install python3.10-venv
python3 -m venv airflow_venv
sudo pip install pandas
sudo pip install s3fs
sudo pip install apache-airflow
air flow independently
sudo apt install awscli
configure aws
aws sts get-session-token
***************** USEFUL LINKS *****************
Extract current weather data from the Open Weather Map API using Python on AWS EC2: https://www.youtube.com/watch?v0_caTDCZnd0&t13s

How to connect Visual Studio Code to AWS EC2 remotely: https://www.youtube.com/watch?vsQQjMnEkGjs&t1224s

PostgreSQL playlist: https://www.youtube.com/watch?voFaLUCWRnRE&listPLACD_PaYcVF09khO58CISr08Uy6w3cAIF

Weather Map API: https://openweathermap.org/api

Github repository: https://github.com/YemiOla/data_engineering_project_openweathermap_api_airflow_etl_aws

Don't forget to LIKE, SHARE, COMMENT and SUBSCRIBE our channel for more AMAZING videos.

DISCLAIMER: This video and description contain affiliate links. This means that if you purchase through one of these links, we will receive a small commission at no cost to you. This will help us continue to create great and valuable content for you.

Please take the opportunity to connect and share this video with your friends and family if you find it useful.