Still Waiting

Why I kept skipping it

I could dress this up.

I won’t.

Part of it was laziness.

If it works, don’t touch it.

Part of it was timing.

Upgrades on production happen off-hours.

Production runs on European hours. I’m in Nepal.

Their quiet hours are my late nights.

And the weekend?

The weekend could work.

But the weekend usually belongs to everything else in my life.

So the upgrade waited.

Then it waited some more.

Nobody asked, so nobody noticed.

Including me.

The push

Our dev cluster was already on a much newer Kubernetes.

Then we upgraded Proxmox, the hypervisor everything runs on.

Dev came through it fine.

Prod didn’t.

Networking on the old prod cluster went sideways. DNS stopped resolving properly, and I had to restart networking to bring it back.

Same hypervisor.

Same upgrade.

The new cluster survived.

The old one didn’t.

That was enough.

And honestly, it had to be done anyway.

Better to pick the day myself than have the next Proxmox upgrade pick it for me.

The bill

Here’s the thing about skipping upgrades.

You don’t skip them.

You just move them into the future.

With interest.

You can’t jump from 1.26 to 1.31.

Kubernetes control-plane upgrades must proceed one minor version at a time.

So that’s what I did.

v1.26.6+rke2r1
  → v1.27.16+rke2r2    (August 2026)
  → v1.28.15+rke2r1    (August 2026)
  → v1.29.15+rke2r1    (August 2026)
  → v1.30.14+rke2r2    (October 2026)
  → v1.31.14+rke2r1    (November 2026)

Five passes.

Each one the same routine.

Take an etcd snapshot. Copy it off the node.

Three control-plane nodes first, one at a time, checking etcd before and after each one.

Then six workers, also one at a time. Drain it, upgrade it, wait for it to come back, wait for storage to be healthy again, next.

Nine nodes, five times.

Forty-five node upgrades.

No data lost.

And every single pass dug up something that had been quietly waiting for me.

Storage flags left over from the last pass, blocking the next drain.

An Ubuntu service grabbing Longhorn’s disks so a pod couldn’t mount its volume.

A kernel limit nobody had ever touched, suddenly throwing “too many open files.”

A crash-looping pod whose disruption budget refused to let the node drain.

Images whose tags had been deleted upstream, running fine only because nobody had restarted them.

Stale iSCSI sessions on a node, so a volume simply wouldn’t attach there.

People ask me which one hurt the most.

All of them.

None of them were hard on their own.

It was the pile.

Every one of them had been sitting there, harmless, for as long as nobody touched anything.

The upgrade didn’t create the problems.

It found them.

The one I did to myself

And then there was the one that was my own fault.

A long time ago, I’d edited the ingress-nginx config by hand.

A small change, so one ingress could use a config snippet for its authentication.

It worked.

I forgot about it.

You can probably see where this is going.

The 1.29 pass replaced the ingress controller with a newer one, rebuilt from its defaults.

My hand edit was gone.

And the new defaults were stricter. Snippets off.

The ingress stopped working.

At the start of the week, somebody reported it.

The quick fix took minutes.

The real fix came first thing in the next pass. RKE2 deploys ingress-nginx using its bundled chart. To customise that chart without editing its defaults, you use a HelmChartConfig. So that’s where the setting lives now, outside the chart’s defaults, where a chart upgrade won’t simply overwrite my change.

It has survived two controller upgrades since.

If this sounds familiar, it’s because I’ve written about it before.

A change made by hand.

Never written down.

Undone by automation.

Different tool.

Same lesson.

Apparently I need to learn it more than once.

The ceiling

Somewhere around 1.28, I looked at Rancher.

Rancher 2.10 supports Kubernetes up to 1.31.

Past that, Rancher has to move first.

I looked at the upgrade path.

Six steps.

And I said:

“Let it be.”

Not because it was hard.

Because life was already full.

Personal stuff. Being sick. Too much, all at once.

So I parked it.

Again.

Then I finished 1.31.

And there it was.

The ceiling.

No more parking. Kubernetes couldn’t go any further until Rancher did.

You can’t skip that either

My first question was the lazy one.

“Can I just go straight to 2.15?”

No.

Rancher only supports one minor version at a time.

And it gets better.

Newer Rancher versions drop support for older Kubernetes.

Here are the Kubernetes ranges each Rancher version’s release notes list. These are the broad supported ranges, not the exact builds:

Each version links to its release notes.

Rancher 2.13 doesn’t support 1.31 anymore.

And that matters twice for me, because Rancher runs on the production cluster. It’s not just managing that Kubernetes version. It’s living on it.

So I can’t even finish the Rancher upgrades first and deal with Kubernetes later.

They have to take turns.

When upgrading Rancher, the Kubernetes version underneath it has to be supported by both the Rancher version I’m running and the version I’m moving to.

When upgrading Kubernetes, the new version has to be supported by the Rancher version currently running on it.

Rancher 2.10 → 2.11 → 2.12
Kubernetes 1.31 → 1.32
Rancher 2.12 → 2.13
Kubernetes 1.32 → 1.33
Rancher 2.13 → 2.14
Kubernetes 1.33 → 1.34
Rancher 2.14 → 2.15

A zipper.

Not a straight line.

That’s the real cost of skipping.

Not one big upgrade.

A long, ordered list of small ones you can’t reorder, can’t combine, and can’t skip.

One more thing I learned the hard way: “Rancher supports Kubernetes 1.32” and “there’s an RKE2 1.32 build I can actually install” are two different questions.

Rancher’s support is per Kubernetes minor version. Which exact RKE2 build to install is a separate decision, and the answer isn’t always the one the release channel suggests.

Which brings me to the next part.

The versions that weren’t there

When I first planned this, I built the path from Rancher’s GitHub releases.

v2.10.12. v2.11.19. Lots of nice, recent patch versions.

Most of them aren’t public.

They’re for paying Rancher Prime customers. The release exists on GitHub, but there’s no public image to pull.

To be clear, that’s my experience with the community distribution, the free one I run. Prime customers get those builds. GitHub isn’t hiding releases from everyone.

When I checked, on 1 November 2026, Rancher’s public Helm chart repositories (both stable and latest) topped out at .3 on every minor version from 2.10 to 2.15, and Docker Hub had no .4 image for any of them.

That was the newest a community user like me could install that day. It can change, so don’t take my numbers as current.

What doesn’t change is the lesson: the release list isn’t the answer. Check the chart and the image separately before you plan around a version.

So the “latest 2.10” I was supposedly behind on?

I was already on it.

RKE2 had pulled the same trick on me twice already. For 1.30 and again for 1.31, its release channel pointed at builds that turned out to be Prime-only: no files on the GitHub release, no runtime image on Docker Hub.

So now, before I plan around any version, I check two things: the release actually has files, and the image actually exists.

For the rest of the zipper, I already did: RKE2 has public builds for 1.32, 1.33 and 1.34.

And as of 1 November 2026, the 1.32 channel is already doing it again. It points at v1.32.13+rke2r2, which has no files and no image. r1 is the public one.

Upgrade day

Honestly?

The Rancher part was the boring part.

Take a snapshot. Save the current values. helm upgrade. Watch.

Both Rancher upgrades, 2.10.3 to 2.11.3 and then 2.11.3 to 2.12.3, ran on the same cluster: RKE2 v1.31.14+rke2r1.

Rancher came up on 2.11 and upgraded its own pieces by itself: the webhook, Fleet, the provisioning bits.

For a minute the logs were full of errors about chart versions that couldn’t be found.

Then Rancher refreshed its chart repository and they stopped.

Looked scary.

Was nothing.

Halfway through the upgrade to 2.12, my VPN dropped.

Helm said the release was stuck in pending-upgrade.

I didn’t panic.

Nothing was actually wrong. A dropped VPN on my laptop doesn’t break a cluster.

But “nothing is wrong” is a feeling. I wanted to check.

Helm’s status is just a record Helm writes for itself. It says what Helm thinks happened.

So I looked at what the cluster was actually running instead.

# what Helm recorded
helm -n cattle-system history rancher

# which image the Rancher deployment is configured to run
kubectl -n cattle-system get deploy rancher \
  -o jsonpath='{.spec.template.spec.containers[0].image}'

# which image the running pods actually started from
kubectl -n cattle-system get pods -l app=rancher \
  -o jsonpath='{range .items[*]}{.metadata.name}{" "}{range .status.containerStatuses[*]}{.imageID}{"\n"}{end}{end}'

# which version Rancher itself reports
kubectl get settings.management.cattle.io server-version

The first time I checked, Helm said “pending”, but the Rancher deployment was still configured with the 2.11 image, its pods were still running 2.11, and Rancher still reported 2.11. The upgrade hadn’t reached that point yet. Rancher was fine, just not upgraded.

If it had stayed stuck, I’d first make sure the old Helm operation was really gone, then look at the release state and choose the right way to recover before retrying.

It didn’t come to that.

A few minutes later, Helm said deployed, the deployment and its running pod were on the 2.12 image, and Rancher reported 2.12.

Then the webhook and Fleet upgraded themselves to their 2.12 versions, like they had for 2.11.

That’s when I called it done.

Not when Helm said so.

When the cluster did.

The new Rancher even restarted itself once, on purpose, with a [FATAL] line in the log.

Turns out that’s by design.

Fine.

Two Rancher versions done.

Nobody has reported anything.

Which, for once, is exactly what I wanted.

Not just Rancher

This isn’t a Rancher problem.

It’s a me problem.

The same habit has cost me before.

At one point, some of our apps were still on PHP 7.4, 8.1 and 8.2, long after they should have moved on.

When things went wrong there, being that far behind made everything harder.

That’s a story for another day.

But it’s the same story.

Hold on

Right now, production is on Kubernetes 1.31 and Rancher 2.12.

Kubernetes 1.32 is next.

Then three Rancher upgrades and two more Kubernetes upgrades, taking turns, until Rancher 2.15.

And this time I’m not parking it.

Here’s what changes:

Subscribe to release announcements. Maybe write a small script that tells me when a new version is available.

Read up before I upgrade.

Try it on dev first.

Then prod.

And don’t skip it.

One minor version is a boring evening.

Six at once is a project.

Boring is the goal.

Nobody asked me to upgrade.

Nobody is going to.

That’s exactly why I have to.

If you want to check my homework

Rancher Manager documentation

RKE2 documentation

Rancher releases: each minor’s release notes list the Kubernetes versions it supports

RKE2 releases