Cy

We merge on an API id that I am now realising nobody has ever verified.

I'd skip the folding panel and put a fixed one on the roof. Folding kit means remembering it, deploying it, securing it against theft and stowing it again, and after one season I stopped bothering entirely. The panel you never think about is the one that actually charges your battery.

I'd skip the folding panel and put a fixed one on the roof. Folding kit means remembering it, deploying it, securing it against theft and stowing it again, and after one season I stopped bothering entirely. The panel you never think about is the one that actually charges your battery.

The middle ground is a rolling reprocess window. You know your late arrival horizon is about four days, so rebuild the last five or seven days every night instead of only yesterday. That costs roughly seven times one day, which is nothing next to a 90 day backfill, and your numbers become correct within the window automatically.

Two details that matter:

• Partition by event date, not ingestion date, and always reprocess whole partitions.
• Keep the ingestion timestamp as a column anyway. It lets you answer 'what did this number look like on Tuesday' when finance asks, and they will.

If you have a genuine long tail beyond four days, add a weekend job that reprocesses the last 45 days. Nobody notices a correction that lands on a Sunday.

Because the tutorial always shows yesterday. Almost every real pipeline has a window.

Four in six weeks for a first pipeline is unremarkable, and three of your four causes were external. Nobody builds something that survives an unannounced upstream schema change.

What separates a good pipeline from a bad one is not that it never breaks, it is how fast you know and how cheap the recovery is. Ask yourself:

• Did you find out from a monitor, or from a person asking why the dashboard was empty?
• Could you rerun the failed day safely, or did it need surgery?
• Did you add a check after each incident so that class of failure is now noisy rather than silent?

If the answers are monitor, rerun and yes, you are doing this correctly and the count does not matter. The unexplained one is the only item on your list I would chase. Unexplained failures come back.

That is the one to fix. Alerting on data freshness rather than on task failure catches a whole category of silent problems.

That minute is the most expensive minute in your day and it teaches you nothing.

Do both for a fortnight. Build the merged version into a shadow table, compare row counts and a few aggregates against the rebuild every day, and you find out whether your keys hold up without betting anything on it.