1. Introduction
I went to KubeCon + CloudNativeCon Japan 2026 in Yokohama.
The main conference ran over two days, with a co-hosted event held the day before it. What I was able to attend was the co-hosted event and day one of the main conference — two days' worth; a personal commitment kept me from day two. What follows is what I took away from that.
To be honest, I wasn't especially bullish on Kubernetes going in. My view was that K8s is a tool for organizations that need flexibility to absorb large-scale traffic — and therefore has nothing to do with a fledgling app developer like NASEBANAL. And in practice, the NASEBANAL apps all run on Cloudflare Workers. Serverless gets you scalable deployment without ever thinking about the K8s layer, and as long as I'm building public services where traffic is modest and productivity matters more than operational cost, I figured I simply wouldn't have occasion to touch K8s.
I went anyway for two reasons. One, the venue happened to be close to home. Two, I've had touchpoints with Kubernetes through my time at Pivotal (later VMware) and at Cloudflare, and I wanted to catch up on where the trends had moved.
What I came home with were insights that connect directly to the NASEBANAL business. Not in the direction of "you should pick K8s as your runtime" — in a different direction entirely: K8s came into view for a set of jobs that have nothing to do with running the application.
The photo above is from KubeAuto Day Japan, a KubeCon co-hosted event held the day before the main conference, under the theme "Making Kubernetes Smarter through AI-Driven Automation." Of the lessons below, the second and the fourth come mostly out of the discussions there.
Here are the four lessons that taught me the most.
2. Lesson 1 — Kubernetes as a Harness
The single biggest takeaway was a reframing: K8s not as the environment where the application runs, but as the environment where the tests run.
On a recent project, our CI/CD pipeline used self-hosted runners to execute jobs inside a private network — and what marshalled those runners was Kubernetes. Looking back, there was nothing exotic about that setup. The requirement that produced it — testing an integration with a service reachable only from inside a private network — isn't a rarity in enterprise development; it's close to the default. GitHub-hosted runners can't reach it, so you stand up self-hosted runners. And then the question becomes how you scale those runners and how you make them disposable — a question for which K8s is one of the strongest answers available today.
What this means for NASEBANAL
That framing connects directly to the API quality assurance work I care about. In the Contract-Driven Development setup I wrote about two posts ago, consumer-side tests run against a Specmatic stub while provider-side contract tests run against the real service. Today that fits entirely within my laptop and hosted runners, because I'm a team of one. But the moment I want to apply that same machinery to an API sitting inside a client's private network, everything changes: neither the stub nor the contract tests mean anything unless they run somewhere that can actually reach it. That's when K8s becomes a realistic option — as the substrate for declaratively creating test environments and tearing them down when they're done.
In other words, K8s for NASEBANAL may not be where the app goes — it may be where the quality assurance machinery goes. That was the real shift in perspective for me.
The session that landed hardest in this context was Mercari's "CI as the Guardrail for AI-Native Development." Fast, reliable unit and e2e tests; regression tests; security scanning — the argument was to treat these as CI-as-the-platform-for-guardrails, and that scaling that platform *is* the work of improving the core component that accelerates AI-native development.
The more code AI writes, the more the vessel that catches and verifies that output matters. And once you want to run that vessel at volume — massively parallel, disposable — you're back to talking about the execution substrate underneath it. That's what got me interested in using K8s as a harness.
3. Lesson 2 — What to Expect of an Agent: Probabilistic vs. Deterministic
The second lesson was the debate around agent architecture: what we should expect an agent to do in the first place. A lot of this conference's airtime went here.
Its centerpiece was the fireside chat at KubeAuto Day Japan, the community event held the day before KubeCon proper: Kelsey Hightower and Laurent Gil, President & Co-Founder of Cast AI, on "From 'The Hard Way' to 'The Invisible Way'."
What Kelsey Hightower argued in the session ran roughly like this. Kubernetes was built for a simple mission in the first place — keeping applications running reliably — and complexity was never inherently required of it. Which is precisely why the next phase points toward becoming invisible: fading out of view for the people using it. Coming from the author of "Kubernetes The Hard Way," the title "The Invisible Way" is really the story of the product maturing.
And it was in the course of that discussion that the conversation turned to today's agents. That's the part that grabbed me.
Kelsey Hightower touched on this, and it matches something I've felt for a while: we're in a period of over-expectation around agents.
To set the terms: by *agent* I mean an interface that takes an AI engine like Claude, layers additional context onto it through something like RAG, and combines it with MCP to direct work. The keyword attached to that is Probabilistic. And set against it is Deterministic.
- As long as something is Probabilistic, its answers are generated on the basis of probability. Output therefore fluctuates, and sometimes it is simply wrong.
- A Deterministic approach is what rule-based programming has always been. Its stability comes from logic.
On an earlier project, the worry was knowledge becoming siloed in individuals, and agents were the hoped-for answer. But once you're invoking AI on every transaction, the cost problem is unavoidable — and because errors happen, you sometimes need a human double-check on top. Both the spend and the verification effort scale linearly with transaction volume.
As things stand today, agents don't just carry probabilistic error — once you're running LLM inference on every single transaction, I think even the energy consumption becomes hard to ignore. Which is why, for anything where you would be putting the same question to the agent over and over — transactions that ask for essentially the same judgment every time — I believe the better approach is to have the AI write rule-based code once, have a human double-check it, and then run that code. Rather than re-asking the agent each time, fix the part whose answer is stable as code. Do that, and digitalization can spread widely into the areas where DX has historically stalled — and done well, that should itself reduce environmental cost.
The probabilistic process runs once, at design time; execution stays deterministic. It resembles the point I made in my previous post about settling the Figma screen design by hand before moving to implementation — similar in the sense that an intermediate artifact gets a human double-check before it becomes the implementation.
One more thing: try to actually operate agents and you inevitably arrive at cost. I'll pick that thread up again in Lesson 4.
4. Lesson 3 — Agents for Root-Cause Investigation During Incidents
The third lesson came from a talk by NVIDIA on using agents for root-cause investigation during incidents — a theme that also touches how I operate my own products.
Structurally, the agent host has three routes: Metrics MCP → Prometheus / Thanos, Logs MCP → Grafana Loki, and Skill MCP → approved runbooks. It carries incident context along as it chooses the next scoped call to make and assembles an answer. In short: reach metrics in Prometheus and logs in Grafana Loki through MCP, then tune the prompt so the whole thing serves root-cause investigation when something breaks.
The line the slide emphasized was this: "MCP standardizes the interface. Each source retains its own vocabulary, authorization, and policy." That, too, is continuous with the deterministic argument from Lesson 2. Rather than granting the agent unrestricted access to all data, you route it through a standardized MCP interface while authorization and policy stay owned by each source. It raises the agent's degrees of freedom without raising the blast radius.
It's something I felt at the Linux Conference I attended last year as well: open-source software is increasingly shipped to run on K8s. If you want to stand up an open-source service yourself, K8s is no longer something you can ignore. And wiring agent integration into it via MCP might genuinely deliver earlier root-cause discovery.
5. Lesson 4 — FinOps for Agents
The fourth lesson is FinOps applied to agent usage itself. As I said at the end of Lesson 2, operate agents in earnest and you arrive at cost. At KubeAuto Day, both Cast AI and Kimchi took up the FinOps of running agents — a practical and interesting discussion.
Cast AI — caching, and Token In vs. Token Out
The point that stuck with me from Cast AI was this: when you look at what agent usage actually costs, a substantial share of the queries are in fact served from cache. Beyond that, the cost factors are Token In and Token Out — and Token In is the one with the larger impact on cost (as I recall it, the argument was that it outweighs Out).
This is where the lesson was for me. The point is that Token In and Token Out are priced differently — in Claude's case, output runs roughly 5× the input rate per token. Which means you cannot measure cost by "how many tokens did we use in total": you have to estimate cost with the two kept separate.
Given that, the reason Token In — not the pricier Out — dominates the total is that the volume flowing in is overwhelmingly larger: context, the reference material RAG injects, conversation history, tool definitions — all of it rides along on every single call. A 5× price gap loses to a volume gap bigger than that. And on the In side there's a further split, because a cache hit reads at roughly a tenth of the input rate. So the unit of a cost estimate isn't "total tokens" — it's a breakdown of Token In (cache hit vs. miss) and Token Out.
I'll be honest: I had already come across the fact that tokens are tracked separately as Token In and Token Out — I'd seen it when I looked into this myself. What hadn't occurred to me was that the two carry different unit prices, and that a cost estimate has to keep them apart.
Kimchi — open-source engines are catching up, so find the best mix
The other thread came from Kimchi, a coding agent that runs in the terminal. The session showed multi-model orchestration: seven specialized roles (Planner, Builder, Reviewer, Explorer, Researcher, Judge, and so on) with an orchestrator that automatically routes each task to the best-fit model.
And accuracy isn't the only place that pays off. Open-source engines like Kimi have improved in accuracy to the point of being genuinely usable today. So when you optimize for cost, you don't run everything through the top-tier model — you find the best mix, assigning a high-capability model where accuracy is required and a lighter, cheaper one where it isn't. And Kimchi was presented as the open-source software that makes that mix workable.
Which means multi-model orchestration isn't only "an architecture for better accuracy" — it's also the implementation mechanism for cost optimization. Read that way, the dashboard from the same session comes into focus: Total Cost, API Calls, Avg Duration, Error Rate, and usage broken out per model — agent execution viewed through operational metrics.
Listening to all this, I kept thinking back to a recipe I published in the NASEBANAL API Specs Cookbook: using NASEBANAL Recorder to visualize how many tokens a software development effort actually consumes. It uses a Claude Code Stop hook to record four values — Input / Output / CacheRead / CacheCreate — as a per-project time series (the post that introduced it).
I built it out of a purely personal motive — I wanted to know what my own development was costing. But it's exactly the Token In / Token Out / cache breakdown Cast AI was pointing at. And what I learned at this conference is that a lot of people are eyeing this space as a business. Visualizing and optimizing an agent's token consumption — there does appear to be a market there.
6. Wrapping Up
Looking back — and this echoes what Kelsey Hightower said at KubeAuto Day — as far as what I saw, this year's KubeCon was less about how to use Kubernetes itself and more about how to use Kubernetes for AI and agent development, and how to optimize its cost. Precisely because adoption has accumulated and operations have matured, attention has moved to the layer above. It felt like the ecosystem has entered a new stage.
And for NASEBANAL, those two days brought the possibility of using K8s back into view (missing day two is my one regret) — not as the app runtime, but as the harness for quality assurance, and potentially as the foundation for observability down the road.
When the timing is right, I want to renew the CKA and CKAD that lapsed last year, so that K8s becomes something I can cover when supporting clients building software in house.
