Airflow

Alright, Let’s Start You know how some things just need all the pieces lined up perfectly, like assembling furniture without an instruction manual? That’s kind of what we’ve been doing here—building a whole setup for handling cryptocurrency data, step by step. And now, it’s time to piece it together with Airflow. We’re not just moving data around aimlessly, though. The plan is, as it usually goes, to get the data from CoinMarketCap, shape it into something useful, and send it to the FastAPI app for storage. And once it’s sitting pretty in a database, we’ll have what you could call a playground for analysis. ...

So, you’ve got weather data from the Weather.gov API. You’ve built a FastAPI app that transforms and stores data using a Type 2 Slowly Changing Dimension (SCD) model. And now you’re wondering: “How do I tie this all together in a way that looks effortless but secretly makes me feel like a data wizard?” Enter Airflow, the glue that orchestrates your data pipeline. Today, we’re bringing together orchestration, transformation, and storage into a single, elegant workflow that will: ...

Well, here it is: getting Apache Airflow up and running! 🎉 To begin, you’ll want to familiarize yourself with the example docker-compose.yaml that Apache provides in the Airflow repository: 🔗 Official Docker Compose Example 🔍 Overview of the Services, Environment Variables, and Volumes The Apache-provided docker-compose.yaml file is described as: “Basic Airflow cluster configuration for CeleryExecutor with Redis and PostgreSQL.” This setup provides a fully functional Airflow instance using a CeleryExecutor, supported by Redis for task queuing and PostgreSQL for the metadata database. Below is a breakdown of each service in the stack and its role: ...