Sabine Korte

Rough family pattern that has held up for me over years of tin diving: the alliums and the umbellifers are the short lived ones, so onion, leek, and parsnip in particular I treat as one season seed and buy fresh every year without any guilt. Brassicas, lettuce, beans, tomato, and anything in the cucumber and squash family have all come up for me years after the date on the packet, sometimes embarrassingly well. So if your tin has a 2019 parsnip packet, that is almost certainly compost, and a 2019 courgette packet is probably still fine. Test rather than trust either statement, but that tells you which ones to test first.

One test I use before keeping anything: if the hub is down for a day, does the house still work by hand? Anything that fails that test gets a physical switch or gets deleted, because eventually you will be away and someone else will be standing in your dark hallway.

Going to push back on the "middleware isn't enough" consensus, partly.

If you namespace routes by resource - everything under /api/org/:orgId/*, middleware can resolve membership for :orgId once and attach it, and then a missing check in a handler is a much smaller blast radius because the wrong org can't even reach the handler. That got us from thirty ad-hoc checks to one, and the per-object checks that remain are genuinely about the object, not the tenant.

So: middleware for tenancy, query scoping for objects. Doing only one of the two is where people get hurt.

Timeouts rather than NXDOMAIN on brand new nodes points at the data path, not at CoreDNS's answers. Two things are almost certainly happening together.

First, kube-proxy on a new node has to program its rules before the ClusterIP for CoreDNS resolves to anything. Pods can be scheduled and start making DNS queries before that is complete, and you get exactly a 30-60 second window of timeouts.

Second, two CoreDNS replicas for 60 nodes is thin, and the morning ramp is when every new pod does its startup lookups at once. Run the cluster-proportional autoscaler for CoreDNS so replicas track node count, and spread them with a topology constraint so they are not both on the same node.

The change that will help most though is NodeLocal DNSCache. It puts a caching resolver on every node, so pods talk to something local over TCP to CoreDNS and stop depending on the ClusterIP path being ready. It also removes most of the conntrack UDP race problems people hit at this scale.

Going to disagree with pure thermodynamics, because plenty of us carry half-empty pens on warm trains without a cap full of ink. Pull the nib and feed and check the nib is fully seated and that the feed channel is not crudded up, especially if you have cleaned it recently and pushed things back in by feel. A feed that sits a fraction proud does not buffer properly and you get exactly this: fine when static, wet when moved and warmed.

Also check the piston seal by holding it nib down under a bright light and watching for a bead forming at the nib slit over a minute or two.

Other big source of the same symptom: DIVIDE. A ratio total is never the sum of the ratios, and it should not be. If you want a weighted total, sum the numerator and denominator separately and divide at the end.

Minor style point that saves pain later: bound your ranges. C2:C10000 rather than C2:C on a sheet that will grow. Open-ended ranges inside an ARRAYFORMULA make Sheets consider the entire column, and on a big file you will feel it.

the hello world floor is such an obvious control and i have never once done it.

And do it right after publishing rather than at 6am the next day when the schedule fires and nobody is watching.

That is a dynamic data source. Somewhere you have a step that builds a path or URL from a parameter, a column value, or a date, and the service cannot work out what it is going to call before it calls it, so it refuses to attach credentials.

The fix is to keep the base address literal and push the variable part into an option record: Web.Contents("https://host/api", [RelativePath = "reports/" & period]) instead of Web.Contents("https://host/api/reports/" & period). Same for SharePoint, hardcode the site root and navigate to the folder in later steps rather than concatenating the folder into the initial call.

Worth adding a pod disruption budget and checking per-zone headroom before draining anything that has attached storage. This failure mode is entirely predictable and almost always discovered during maintenance.

Sanity check for anyone finding this later: select the column and look at the alignment before doing anything else. Right aligned means Sheets sees a real date or number, left aligned means text. Fastest diagnostic in the whole application and it takes no clicks.

The compression argument is the one I can take to my manager, because "we hit the Pro ceiling in November" is a date and a consequence.

In-place, and the StatefulSets are the deciding factor. Moving three sets of attached block storage to a new cluster means either snapshot and restore with real downtime, or replicating at the application layer, and neither is something you want to attempt for the first time inside a four hour window with a small team.

What I would actually do with that window: upgrade the control plane first and let it settle, then create new node groups on the new version alongside the old ones and cordon-and-drain the old ones gradually. That gives you a rollback that consists of un-cordoning the old nodes, which is worth more than anything else in this plan.

Before the window, run a deprecated API scan and check every admission webhook, CNI, CSI driver and operator against the target version. Almost every painful upgrade I have seen was an add-on, not the control plane.

If all 30 sites have identical column layouts, stacking is easy. If three of them have an extra column someone added locally, fix that first or the stack silently misaligns and you will chase it for a day.