A source system started populating a status field with a new value we had never seen. Our transformation has a case statement that maps known values and defaults everything else to one bucket. So nothing failed. No job…
One of our models is expensive and only needs rebuilding weekly, or when somebody asks for it. Right now it runs every night with everything else because the scheduled job builds the whole project. Disabling it in…
I want to parameterise a transformation model so the same code can run against different sets of ids, one set for a backfill, another for the daily run without duplicating the model. Passing a single value works fine.…
Our transformation project writes to the same schema whether it runs on my laptop or in the scheduled production job, so a local run can overwrite a production table. That is obviously wrong and I would like to fix it…
Small team, one warehouse, ingestion is a managed connector that lands on its own schedule, then about seventy dbt models on top. Airflow keeps getting suggested and from here it looks like a whole second system to…
Two years in this role, I own ingestion from six sources into Snowflake, roughly ninety dbt models and an Airflow schedule that ties it together. In interviews I get asked about ETL tooling and I never know whether to…
One new model, runs hourly, reads from a large event table and writes about 400k rows a day. The bill went from roughly $900 to $2700 a month and this is the only change we made. The model looks cheap when I run it once…
Five of them are S3 to Redshift copies and one kicks off an EMR job. They still run every night and nobody currently here can safely change a line of them. Team is three engineers and we already use Step Functions…
A scheduled task pulls from an API, writes to staging, then merges into the warehouse. If the task dies halfway through the write and the retry runs, we get duplicate rows for the records that landed before the crash.…
Events can arrive up to four days after the event timestamp. My daily rollups are built once at 2am for the previous day, so anything late is simply missing until somebody notices and asks for a backfill, which…
Built our first real scheduled pipeline about six weeks ago. It has needed manual intervention four times: an API rate limit, an upstream schema change, a disk filling up, and one thing I still cannot explain. I…
Customer dimension, about 40 million rows, sourced from three systems with imperfect keys. A full rebuild takes 25 minutes and is dead simple. A merge takes 3 minutes but I am nervous about drift and about the key…