It was supposed to be a normal weekend.
I was on a train, about an hour into the journey, meeting a friend and enjoying the evening. I was in Australia, half of my team was in Nepal, and the other half was in Germany.
It was late.
I was already thinking about getting home.
Let’s sleep.
Apparently, God had other plans.
Because somewhere in February, one small application in our dev environment had decided that it wanted to see the world.
Unfortunately, it chose to do that by connecting to random public IP addresses.
And it took us a while to figure that out.
It started innocently enough
On February 12, I deployed an application to our dev Kubernetes cluster.
Nothing unusual.
This was pretty much my routine.
A developer gives me an application, I make sure there’s a Dockerfile, the CI pipeline builds the image, the manifest gets updated, Fleet picks it up, Kubernetes deploys it, pods come up healthy.
Done.
Next.
I wasn’t sitting there thinking:
“Today feels like a good day to introduce malicious code into our infrastructure.”
It was just another application.
At the time, I also wasn’t particularly strict about vulnerability scanning.
We had Trivy in our CI/CD pipelines.
We'd been using it for some of our projects since around 2022.
But somewhere during the endless process of improving our GitLab pipelines, standardizing workflows and moving more things towards GitOps, that scan disappeared.
Nobody complained.
Nobody told me to remove it.
I didn’t have some meeting where I proudly announced:
“Security is no longer required.”
It just… disappeared.
Looking back, I have one question for myself:
Why the hell did I remove that?
Then the CPU started going crazy
Around February 17–18, we started seeing flooding.
The first symptom wasn’t some sophisticated security alert.
It was simply:
High CPU.
I found a suspicious process and killed it.
Problem solved.
Except it wasn’t.
The process came back.
So I deleted the VM.
Surely that would solve it.
Except the VM was part of Kubernetes.
And my brain, which apparently had temporarily resigned from its senior DevOps position, forgot one small detail.
Kubernetes doesn’t care that you deleted a VM.
The workload still existed.
So Kubernetes did exactly what it was supposed to do.
It brought the workload back.
On another node.
And so did my problem.
At some point, I was basically playing Whack-a-Mole: Kubernetes Edition.
Except every mole had root access.
March 9: this is getting serious
By March 9, the Slack messages had changed tone.
Someone wrote:
“Hey guys, we have to figure this out today. Nightmare situation right now.”
Then came March 12.
This time the messages were much worse.
The datacenter had detected suspicious behaviour.
Some VMs were down.
Some ports were blocked.
The datacenter firewall was blocking our cluster because it was trying to prevent the problem from affecting other customers.
One of our VMs was generating a huge number of firewall sessions.
And the traffic pattern was particularly unpleasant.
The machine was trying to connect to port 80 on a large number of randomly distributed public IP addresses around the world.
The words “compromised system” and “botnet activity” started appearing.
Then another email arrived.
Our IP had been blocked by AWS.
Wonderful.
Maybe it’s the IP?
We had a shared NAT IP that was used by multiple projects.
So my first thought wasn’t:
“One of our applications is compromised.”
It was more like:
“Maybe we’ve just burned the NAT IP.”
Honestly, given what we knew at the time, that didn’t sound completely unreasonable.
People started asking if we could get another IP.
I said we could temporarily do something, but we had a fixed pool and changing the IP wasn’t really going to solve the underlying problem.
Meanwhile, we were being asked to make backups of production databases and store them somewhere outside the cluster.
I created some temporary off-cluster storage for the backups.
At this point, it felt like an extremely cheap insurance policy.
The dev cluster was becoming so problematic that we decided to shut it down temporarily.
That cluster wasn't particularly large.
But those VMs were supporting a surprisingly large number of services.
I’m deliberately not naming them here.
Let’s just say there were enough of them that shutting down the dev cluster was not exactly a quiet decision.
So our little “dev problem” was no longer behaving like a dev problem.
My brilliant solution
Eventually, after fighting the dev cluster long enough, we decided the application still needed to reach the client.
And I had no idea the application itself was responsible.
I genuinely believed the problem was somewhere in the dev infrastructure.
So…
I deployed the same application to production.Yes.
I know.
That was the moment I knew I had fucked up.
Not because I knowingly deployed something malicious.
But because the problem had now followed me into production.
And then I stopped killing it
By this point, I was tired.
Really tired.
Every time I saw the malicious process, I’d kill it.
Problem goes away.
Then it comes back.
New VM.
Same problem.
New node.
Same problem.
I was looking at the compromised server again, and instead of killing the process like I’d done every other time, I thought:
Fuck it. Let me see what this thing actually does.
I started looking at the process tree. Something like:
pstree -ap <PID>At some point I thought: “Can I just ask ChatGPT?”
So I did.
It helped me make sense of a few things and gave me some leads to follow.
But I still had to look at the machine.
And then I saw it.
next-server
└─ sh
└─ wget
└─ <external script>I just stared at it.
Oh.
Then came the second thought:
“Wait… next-server?”
That was the moment I knew I’d been looking in the wrong place.
The application was the headache.
And now I finally had something I could follow.
The problem was that next-server wasn’t exactly
unique in our environment.
Almost every application we ran was using Next.js.
Which one?
I started listing the Next.js applications running in the cluster.
There were only a handful, which was lucky.
I checked the versions.
I checked what was actually running.
Then I started correlating deployment times.
One application started looking very suspicious.
The timing lined up.
The behaviour lined up.
I removed it.
And then we watched.
No more flooding.
No more malicious connections.
Silence.
After days of fighting VMs, nodes, clusters, IP addresses and firewalls…
It was the application.
After all of that, it was the fucking application.
I wasn't happy about the incident.
I was happy because I had finally found the damn thing.
The part of the incident I actually enjoyed
So the incident had no real timezone.
Germany would report things during their day.
Nepal would be online.
And I would be dealing with it in the middle of the night.
Sleep wasn’t exactly part of the architecture.
There was also another reality happening at the same time.
The business needed to keep running.
People were looking for alternative IPs.
Alternative routing.
Ways to keep services available.
Ways to get customers working again.
But I was focused on something slightly different:
I wanted to know what was actually causing this.
By the time I was offered help directly on the final day, I had already found the source.
And honestly, finding it gave me a weird amount of confidence.
I remember thinking:
“Okay. Maybe I’m not actually terrible at this.”
It was a real problem, affecting real systems, with real consequences.
Then came the cleanup
Once we knew which application was responsible, we removed it from Fleet.
Earlier, I had been deleting VMs and Kubernetes was happily recreating the workload.
Now we removed the actual workload.
We rebuilt affected VMs.
Rotated credentials.
Upgraded Next.js.
Scanned containers manually while investigating.
Later I realized something uncomfortable: if this project had been covered by the same kind of scanning, Trivy could have given us another opportunity to catch the vulnerable dependency before it ever reached the cluster.
I had been thinking mostly about scanning the Docker image.
But scanning the project and its dependencies earlier in the process made much more sense for what I wanted.
We also rebuilt the dev cluster from scratch.
Because, well…
The business still had to run.
During the period where our network access was restricted, we also had to make temporary changes to some applications so they could continue reaching AWS resources through alternative paths.
Eventually, after monitoring the environment and confirming with the datacenter that the malicious traffic had stopped, the restrictions were lifted.
The patient survived.
The weekend did not.
The security check I shouldn’t have lost
The part that bothers me most isn’t that we had a vulnerable application.
It’s that we had a security control and somehow let it disappear.
Trivy had been there for years.
Somewhere between changing pipelines, standardizing workflows and moving things around, I removed it.
Not because it was useless.
Not because anyone told me to.
It just disappeared.
Looking back, that's probably the part I'll remember longer than the incident itself.
It wasn't some sophisticated security decision.
It was just something that quietly disappeared while I was busy changing everything else.
The thing I actually learned
I spent too long treating symptoms.
Kill the process. Delete the VM. Watch Kubernetes bring it back.
I was fixing what I could see instead of asking what kept causing it.
Eventually, I stopped trying to kill the problem and started trying to understand it.
That was the difference.
And the second lesson was even more uncomfortable:
A security control doesn't have to be deliberately removed to become useless.
Sometimes it just disappears while you're busy improving everything else.