Our deploy job used to finish in about two minutes. Since some point in the last few weeks the same job takes twelve to fifteen, on the same chart against the same cluster. What makes it strange is that nothing fails.…
This morning every command against a cluster that was working yesterday returns an unauthorized error. No deployment, no upgrade, no configuration change that I am aware of, and the workloads themselves are still…
I scaled a workload up to twenty replicas. The next morning a handful of those pods had been recreated, deleted and made again, with creation timestamps hours after the rest. The workload is healthy now, which is almost…
Fresh node, packages installed from the distribution repositories, and the kubelet will not come up. The service log says it failed to validate the service connection because the container runtime API version is not…
Small cluster, no log stack yet, everything goes to stdout and we read it with kubectl. Service crash looped for about an hour overnight, the node got rebooted by the platform, and by the time I looked there was nothing…
One deployment out of about forty takes roughly twenty minutes for old pods to disappear on every rollout. terminationGracePeriodSeconds is the default 30. The rest of the cluster rolls in seconds. on a stuck pod shows…
A container got OOMKilled with a 512Mi limit, exit code 137. had been showing it steady around 300Mi for the previous ten minutes, and our dashboard agrees. Nothing in the app logs suggests a spike. I am new to running…
Six node pool, all the same instance type at 4 vCPU. Deployment has ten replicas, nine are Running and one has been Pending since last night. Events say 0/6 nodes are available with a string of reasons I have read…
Cluster autoscaler adds nodes during our morning ramp and for about 30 60 seconds afterwards we get a burst of resolution failures for in cluster service names, mostly from pods that were just scheduled onto the new…
Managed cluster, three nodes of 4 vCPU and 16 GB, and I am trying to size the next pool honestly rather than by guessing. Between what the provider reserves and the DaemonSets we run for logging, metrics and CSI, a…
Drained a node for maintenance. One StatefulSet pod will not reschedule, its PVC sits in Pending and the pod is Pending with FailedScheduling. The PV it used shows status Available in . Storage class is the cloud…
Two minor versions behind on a managed cluster, about 40 nodes, roughly 200 deployments plus three StatefulSets with attached block storage. We have a four hour maintenance window on a Sunday and a small team. In place…