Posts

In the first part of this series, we set up a FastAPI application with a PostgreSQL database to store weather forecasts. We also designed a robust schema that tracks changes over time using a composite primary key. If you missed it, check out Part 1. In this post, we’ll: Create query endpoints to fetch forecasts. Implement a type-2 SCD approach for tracking forecast revisions. Discuss strategies for filtering and pagination. The Challenge 🧠 Weather forecasts often change over time as meteorological data improves. To support historical analysis and ensure data consistency, we need to: ...

When I set out to create a weather forecast API, I wanted something scalable, efficient, and built on modern technology. What started as a simple project to integrate weather data from the weather.gov API quickly grew into a complex, yet rewarding system. To keep things manageable, I decided to break this journey into multiple parts. In this post, I’ll cover the foundational steps to get started: setting up FastAPI, creating the database, and handling weather data with a robust schema. ...

Building a personal data project starts with gathering data. What better way to start than by exploring a free and well-documented API? In this article, we’ll dive into the National Weather Service API (api.weather.gov) to understand how it works, and we’ll write Python code to retrieve and explore weather data. So… what are you going to be doing? Let’s read through the docs and try to understand how to navigate the API.📜 I’m gonna write some Python code to fetch weather data. I’m gonna analyze the data structure of the forecastHourly endpoint. 📍 Understanding the API The National Weather Service API organizes data using a grid-point system. Each grid point covers an area of 2.5 km x 2.5 km. The /points endpoint is where you get information about retrieving data for a specific location. ...

Well, we’ve done it. We have our orchestration layer set up with Airflow, a documentation server, and a notification server. Now it’s time to add a data warehouse to our stack. As I’ve mentioned before, it doesn’t really matter what you use as long as your data is centralized. Having data scattered across multiple sources only forces you to write a bunch of ETLs to bring it back together again, which is unnecessarily time-consuming. I try to keep all of my development on a single database instance locally and then commit to a single cloud DB instance for analytics. ...

Well, here it is: getting Apache Airflow up and running! 🎉 To begin, you’ll want to familiarize yourself with the example docker-compose.yaml that Apache provides in the Airflow repository: 🔗 Official Docker Compose Example 🔍 Overview of the Services, Environment Variables, and Volumes The Apache-provided docker-compose.yaml file is described as: “Basic Airflow cluster configuration for CeleryExecutor with Redis and PostgreSQL.” This setup provides a fully functional Airflow instance using a CeleryExecutor, supported by Redis for task queuing and PostgreSQL for the metadata database. Below is a breakdown of each service in the stack and its role: ...

Next up on our data stack journey is the push notification server. I’ll be using multiple notification channels, hoping that I’ll pay attention to at least one when something critical goes down. Alerting is a key piece of analytics infrastructure for me—data should come to you, not the other way around. So, what are doing here? We’re bridging the gap between the data stack and the real world by setting up a notification system to deliver push notifications straight to my phone. Once my Airflow server is up and running, I’ll create a custom function to send alerts using this server. ...

Welcome to the second post in my data stack setup journey! 🎉 My goal is to hustle through setting up all the components so I can get to the good stuff: data and analytics engineering. But first, I need a documentation server, because, let’s be real—I’m already forgetting where I’ve documented things. 🙃 What is Bookstack? BookStack describes itself as a “simple, self-hosted, easy-to-use platform for organizing and storing information.” I like to think of it as: a wiki that’s easy to host, simple to back up, and doesn’t make you cry. ...

I’m in the process of spinning up a data stack on my home servers for portfolio projects, and I thought I’d share the journey. What Is a Data Stack, and Why Do I Need It? A data stack is a collection of tools, technologies, and platforms that work together to manage, process, and analyze data. If you need to move data between multiple systems into a centralized location for analytics, dashboards, reporting, or other uses, you’ll quickly encounter challenges like tech debt and messy workflows. ...

Hi there, and welcome. I’m Wes—a business intelligence developer, data engineer, data analyst, analytics enthusiast, and occasional referee in the eternal smackdown between data chaos and organizational sanity. If you’ve ever wondered who’s behind the dashboards that light up your morning meetings or the ETL pipelines that ensure your numbers don’t resemble Mad Libs, well, that’s me. Let’s dive in. A Little Backstory (The Mildly Heroic Origin Tale) I started my career in healthcare, where my role involved taming data beasts hiding in electronic health records, financial systems, and various reporting tools. What began as a nursing informaticist role soon morphed into a full-fledged data odyssey. Along the way, I picked up an obsession for translating raw data into insights people can actually use (and maybe even understand). ...

Hello, world! 🌍 After much procrastination, countless cups of coffee ☕, and a few too many Google searches about “how to start a blog,” I’m finally here. Welcome to Wes’s Webzone, where I’ll be sharing my thoughts, insights, and occasional ramblings about data programming, music, and whatever other hobby I am currently persuing. Why a Blog? Great question! Honestly, it’s a mix of: Creativity: I need creative outlets. I don’t feel like I get the chance to share enough. Communication: Writing regularly is a chance to improve how I share ideas and connect with others. Accountability: By putting my thoughts out there, I’m committing to learning, growing, and refining my ideas. Fun: Because why not? 😊 I’ve always wanted a digital space to call my own—like a cozy corner of the internet where I can share ideas, learn from others, and maybe even inspire someone along the way. What to Expect Here’s what’s on the menu (no promises): ...