Search for a command to run...
No comments yet. Be the first to comment.
Lessons from Architecture Dojo, a recurring session at AWS Summit Japan where we tackle real-world design challenges and debate trade-offs live on stage. Each post digs into the architecture patterns, constraints, and decisions behind the systems and scenarios we've examined.
Every architecture decision is a bet. A bet that your constraints won't change, that your assumptions will hold, that the trade-off you're making today won't haunt you in two years. Since 2022, I've b
A proof of concept that passed in 2022 failed its load test in 2025. Nothing in the technology had regressed. The acceptance criteria had grown up. The project behind that sentence is one of the large
On stage, you declare a design correct. In production, you find out what that correctness was quietly resting on. In 2022, at the first Architecture Dojo at AWS Summit Japan, I stood on stage and led
An AI coding agent does not bend a productivity curve on its own. A team does. Coding agents, models, and IDE plugins ship new versions every few months, and most of what gets called "best practice" i
Most teams pick the AI agent evaluation tool their framework integrates with, then a quarter later notice that one corner of the evaluation space is covered and the rest is exposed. The market has no
Build at Scale — System Design, Distributed Systems, and AI
8 posts
Architecture lessons from real-world systems at scale. I write about the trade-offs, constraints, and design decisions behind production systems, from cloud architecture and distributed systems to AI in production and the organizational choices that shape how systems evolve. Each post digs into why a particular design choice was made, what alternatives were considered, and what you can take away for your own systems.
The AI era changes exactly one thing about architecture. The component at the center of your system is now probabilistic. Everything else, the discipline of starting from the problem, naming constraints, and designing trade-offs, survives intact. What changes is where you apply it. That was the hypothesis we put on stage this June.
Architecture Dojo is a session I've been part of at AWS Summit Japan since 2022, where solutions architects answer design challenges on stage and walk the audience through the trade-offs. I hosted the 2025 edition, where we dissected two production systems, and this year's edition, where the format returned to design challenges with a twist: both problems put an LLM at the center of the system. Registrations, including the livestream, reached 3,566, the highest in the session's five-year history, up from 2,300 in 2025. Two colleagues took one problem each and carried it end to end, from logical design through AWS physical design to measured results: Tomoya Okuno on an AI shopping assistant, Yuki Matsuda on an AI-driven development platform.
Neither answer was built alone, though. I wrote the problems and circulated them across AWS's technical organization months before the Summit; more answers came back than the stage could hold, and the two below were sharpened through rounds of review and discussion. The designs are theirs; the retelling and any errors are mine. (The session deck is public, in Japanese.)
TL;DR
Control the LLM from outside. Cost and latency targets are met not inside the model but in the deterministic architecture around it: semantic cache, quality-based routing, thresholds, templates. The shopping assistant cut inference cost to a fifth, closing a 4.9x budget gap with 3.4% quality degradation.
Distribute trust; keep ownership human. Verifying probabilistic output with probabilistic means doesn't close the loop. The development platform splits the roles: AI generates, deterministic machinery judges (formal specs, reference-model differential tests), humans decide.
Observe and adapt. A probabilistic core cannot be fully designed up front. Both answers end in the same loop: observe production behavior, adapt thresholds and specs. Build that loop into the architecture from day one.
The first challenge: an apparel e-commerce company pilots an AI shopping assistant. Users ask in natural language ("find outdoor wear for summer camping," "can I machine-wash this?") and the assistant searches, recommends, and answers questions through the company's existing e-commerce APIs. An alpha test on a few percent of users showed good response quality with Claude Haiku. Then came the full-rollout estimate: inference cost and latency both missed their targets.
What makes this a design challenge worth answering is where the cost target comes from. The assistant's job is to lift conversion, and a session runs a median of 3 turns. The company's own numbers are known: an average order value of 5,000 yen, a 3% conversion rate, a 5% margin. Only the lift has to be assumed, at 12 to 30%. On that assumption each session is worth 0.9 to 2.25 yen in added margin, and spread over 3 turns that leaves 0.3 to 0.75 yen of budget per request. The target was set in the middle: 0.5 yen. One unmeasured multiplier sets that ceiling, which still works as a hard pass-or-fail line: a design that exceeds it breaks the business case, no matter how well it is engineered. Haiku, the model that cleared the quality bar, costs 2.43 yen per request even with prompt caching, 4.9x over. Traffic splits roughly into product search 30%, product detail questions 25%, orders and shipping 20%, ambiguous consultation 15%, everything else 10%. The design must cut cost fivefold and bring first-token latency under 3 seconds at the median, without giving up more than 10% of the alpha's quality.
Okuno-san's answer starts by refusing to treat "LLM cost" as a single number. He decomposes it into three drivers: how many times the model is invoked, what each token costs, and how many tokens flow in and out per invocation. Each driver gets its own countermeasures, and every one trades against response quality, so nothing ships without quality measured before and after. The design process changed too: he worked with a coding agent as a partner, simulating candidate configurations and measuring them rather than arguing them.
The evaluation dataset was built from real alpha utterances, structured as sub-intent × query characteristics × context. Zero-shot generation of test queries mislabeled too often, so labels followed a written policy and were reviewed by an LLM over several rounds. Building the set that way also exposed its own limit. Context is the unbounded axis: what the user is browsing, what sits in the cart, what was said earlier in the session. Multiply that by sub-intents and phrasings and the cases outgrow what any offline set can hold, let alone label. So the design drew that line on purpose: enough offline cases per sub-intent, with the combinatorial tail left to continuous evaluation against live traffic.
The first lever is a semantic cache: embed the query, search near neighbors among pre-seeded query vectors, and return a stored response when similarity clears the threshold kept with each entry. That is a deterministic answer at zero inference cost. Response bodies live in a cheap key-value store, so only vectors sit in the index. New entries are written asynchronously by a backfill service that strips personal data, restricts which use cases may be cached, and safety-checks responses before they reach another user, closing the cache-poisoning path. The cache fits the roughly 40% of traffic that repeats: greetings, FAQ and policy questions, popular product details. The other 60% resists it, open-ended search and consultation most of all: "cotton summer shirts under 5,000 yen" and "cotton summer shirts over 10,000 yen" embed above 0.9 similarity while requiring opposite answers. Even assuming hits on a bit under a quarter of traffic, a number only production can confirm, the cache was projected to cut cost by roughly 14% and first-token latency by 11%. Useful, not sufficient.
The second lever is cheaper models, and it fails in an instructive way. Popular lightweight open models (GPT-OSS, Gemma 4, Qwen, Nemotron) were swapped in as the response generator and scored with AgentCore Evaluations across six quality axes. Cost landed at 0.01 to 0.07 yen per request and first-token latency around 1.2 seconds, both comfortably within target, but quality dropped 20 to 30% against the 10% allowance. Wholesale replacement was off the table. I would put a date on that verdict, though: open models improve faster than architectures get rebuilt, and customers I work with re-run this comparison monthly or quarterly. What outlives any one round of measurement is the habit of stating a quality allowance before switching models at all.
The third lever is where the design gets interesting. Instead of asking which model is good enough for everything, it asks which queries can stay on the cheap path. It decides that from a small model's plan, before any response is generated. One measurement made that possible. Neither the query's intent nor that model's own uncertainty over its tool-planning and tool-argument tokens predicted response quality on its own. Pairing those token probabilities with the shape of the tool-use output did. How the plan looks predicts how well the answer will land. So the design routes on that combination, the same pattern Alexa for Shopping (formerly Amazon Rufus) uses.
A self-hosted Qwen3.5-4B runs the common front stage, cheap and fast by necessity because every request crosses it: intent classification, tool planning, and tool-argument extraction. Self-hosting pays off twice here, on unit economics and by exposing the per-token probabilities that managed APIs often hide. Because its output is what the routing decision reads, this front stage doubles as the router. A routing score multiplies two terms. One is an error probability regressed from the intent, the tool-use patterns, and the entropy of those token probabilities. The other is a business impact weight set coarsely from revenue exposure (product details high, small talk low). Low scores run the ReAct loop on Nova 2 Lite, picked for reading long tool outputs; high scores escalate to Claude Haiku. The escalation threshold is worked backwards from the 0.5 yen budget rather than hand-tuned: the share of traffic you can afford to send to an over-budget model is a function of how cheap the rest is. On the evaluation set, escalating the highest-scoring 20% of traffic caught 20 of the 38 failing queries, about 2.6 times what escalating the same share at random would be expected to find.
Latency got the same outside-the-model treatment. Intent classification and the tool-related extraction split into parallel inferences, roughly halving that front stage, and prompts are split by coarse intent so each request carries only the instructions it needs. Frequent tool-call sequences are compiled into fixed workflows, improving precision and cutting ReAct round trips at once. And FAQ and policy answers come from templates, which doubles as a hallucination guard on exactly the queries where wrong answers are cheapest to prevent.
The final physical architecture runs on Amazon Bedrock AgentCore with Strands agents: Haiku and Nova 2 Lite on Bedrock, Qwen and the embedding model self-hosted on GPU instances, Valkey holding sessions and query vectors, and DynamoDB holding cached responses, which a stream-driven pipeline on Kinesis and Lambda writes in the background. AgentCore Evaluations is wired in for the continuous evaluation the dataset design promised. A query now leaves the system one of four ways: from the cache, from a template, from Nova 2 Lite, or from Haiku. Measured on the evaluation dataset: 0.49 yen per request, time to first token 2.15 seconds at the median, quality degradation 3.4% relative to Haiku alone. All three targets met.
Notice where the machinery lives. Vector comparison, a regression over token probabilities and tool-use patterns, thresholds, static templates: none of it is inside the model. The non-functional requirements are engineered in the deterministic architecture wrapped around the probabilistic core, which is the first response. The more central the LLM, the harder the outside has to work. There is more economics underneath this answer than one section can hold, including where self-hosting breaks even, so it gets a companion post of its own.
The second challenge moves from serving AI to building with it. A fintech company runs a payment platform. A five-person pilot team adopted AI-driven development: AI structures requirements, generates design docs, code, and tests; humans verify and approve at each stage. Per-developer commits rose roughly 20x, and features that took months began shipping in days. Scaling that to a team of about 50 surfaced three problems. Ambiguity in natural-language specs lets AI inject subtle bugs. Those bugs slip through the tests and surface after release. And pull-request review has become the bottleneck, because review capacity does not multiply by 20 just because code generation did.
Matsuda-san's answer distributes trust across three parties: generation belongs to AI, judgment belongs to deterministic machinery, and decision belongs to humans. The deterministic machinery comes in two layers, and the first is formal specification. Take a payment authorization rule: "a payment can be executed only by the account holder, from their own account, within the amount they can spend." Does "their own account" mean only accounts in their own name, or also an account in someone else's name that they have linked to theirs? An AI will implement one reading, and natural language never says which. Formalized as a Cedar policy, the rule pins its meaning to a single interpretation, and because Cedar is machine-analyzable, contradictions within a policy and inconsistencies across policies are caught by automated analysis when the spec is written, not in production. AI drafts the formalization; deterministic checkers validate structure and logic; humans review and approve.
Two structural choices back the layer up. The spec lives in its own repository, distributed to the implementation as a versioned artifact. Whatever resists formalization stays in natural language rather than being forced into a policy, which keeps visible where a machine-checkable guarantee ends and human reading begins. The same policy file then works multiple shifts, checked at spec approval, exercised in CI, and embedded in production as the live authorization engine. The spec is not documentation that drifts from the code. It executes.
The second layer handles what formal policies cannot express: complex business behavior. The running example was a point-reward rule: 1% cashback until cumulative payments pass 1,000 yen, 5% beyond, with corrections when payments are cancelled. Cancel one payment and the rates on the ones that remain have to be recomputed: if a later payment earned 5% only because earlier ones pushed the cumulative total past 1,000 yen, cancelling one of those earlier payments should pull that rate back down to 1%. An implementation that simply subtracts the cancelled payment's points still passes the tests a developer would naturally write, one payment and one cancellation, while quietly leaving the balance wrong. Since the cases cannot be enumerated, the design stops enumerating them. An AI generates a reference model from the spec: the expected behavior as a deliberately minimal implementation that recomputes points from the whole history every time, simple enough to trust as an oracle. Property-based testing then generates operation sequences at volume, payments and cancellations in random order, and replays each against both the implementation and the reference model, flagging any behavioral difference. To check that the gate actually bites, Matsuda-san built a sample payment service with exactly this cumulative-cap bug left in it, and the differential run flagged 23 mismatches out of 87 cases. The approach follows the lightweight formal methods Amazon S3 uses to validate its key-value storage node (Bornholt et al., SOSP 2021).
With deterministic gates in place, the review bottleneck answer follows: human review stops being the default. No divergence from the reference model means auto-merge, with integration and non-functional testing still ahead in staging. Divergence means an AI reviewer comments inline at the code that caused it, with the failing case attached and the business impact explained, offering two paths: fix the implementation or revise the spec. The fail decision itself comes from the deterministic trace, not the AI's opinion. With the hunting mechanized, human review narrows to what machines cannot judge: is this change one we take responsibility for shipping?
The question I put to him on stage was whether all this is too heavy for a one-line fix or an urgent bug. It is, and the answer is to let a human decide case by case rather than encode a rule about which changes qualify. Any change can go straight to the implementation pipeline, but it forfeits auto-merge: the pull request carries a label recording that the machine check was skipped, and a human must review it. Speed is bought by trading a mechanical guarantee for a recorded one, and once the change settles, the spec and reference model absorb it so the next revision is back under machine verification. Another team's answer to the same problem used operational rules instead of structure, a lighter place to start. Which of the two fits depends on the organization's maturity.
The same trust structure extends to DevSecOps, on a principle Matsuda-san states plainly: what must be verified does not change just because the code's author became an AI. Tool-based checks stay deterministic gates, AI review stays advisory, and AI-specific risks get their own countermeasures. The concrete one: AI coding agents hallucinate package names, and an attacker can register those names first, a vector known as slopsquatting (Spracklen et al., USENIX Security 2025). Rather than fight a detection arms race, the design goes structural. Builds resolve only against an internal registry of approved packages, so a hallucinated dependency fails at install time because it was never approved. New packages are scored with OpenSSF Scorecard and auto-imported above a threshold; what falls below goes to a human rather than an auto-reject, since small, finished libraries score poorly on activity metrics however widely they are trusted.
Physically, the platform is two pipelines on AWS CodePipeline and CodeBuild. In the spec pipeline, generation agents run on AgentCore Runtime with validation tools behind AgentCore Gateway, and approved specs and reference models are published through CodeArtifact. In the implementation pipeline, CodeBuild runs the differential test against the pull request and, only when it finds a difference, invokes Kiro CLI in headless mode to write the review comments.
One detail from the design's own history stuck with me as the host. An early iteration had an AI agent judging whether production behavior matched the spec, and the revision notes catch the contradiction bluntly: that puts a probabilistic verdict back at exactly the point the platform had just made deterministic. Judgment moved into a Lambda function comparing outputs against the reference model, and the AI kept only interpretation, classifying mismatches and drafting spec updates. The platform's own design needed the same discipline applied to it. That is the second response: distribute trust between AI and deterministic mechanisms, and keep ownership human.
Put the two answers side by side and the same shape appears at the end of both. The shopping assistant observes cache-hit rates, escalation ratios, cost, and latency, and adapts its thresholds; the platform observes divergence and human review load, and adapts specs and thresholds. Matsuda-san has a name for the spec side of this: a living specification, revised from what the reference model finds in production rather than frozen at approval.
The loop is structural, not optional. A deterministic system's behavior is, in principle, fixed at design time. A probabilistic core's is partly unknowable until you watch it under real traffic: which queries the cache absorbs, which escalations the scoring misjudges, which spec gaps production exposes. So "what do we observe, and what does each observation adjust" becomes an architecture requirement, designed in from the start. That is the third response. Readers of the 2025 edition's lessons will recognize evolutionary architecture with the dial turned up: the observe-adapt cycle stops being how you improve the system and becomes how it stays correct at all. It also widens the architect's job. What we design and observe now spans four objects: the system's behavior, its non-functional requirements, the process that generates it (a response in one problem, the code itself in the other), and the trust and responsibility distributed across that process. In production they surface as one system, so keeping all four adjusted, continuously, is the work.
Which brings back the hypothesis from the top. What changed: a probabilistic, non-deterministic component now sits at the center of the design. The same input no longer guarantees the same output, so the relationship between inputs and outputs has become effectively infinite, no longer a space you can enumerate, test, and declare correct. That is the whole list, and it is why trust was the hardest problem on that stage: when you cannot enumerate the pairs, you stop checking outputs one by one and start comparing behaviors, which is exactly what the reference model is for.
What did not change: start from the problem, make constraints and trade-offs explicit, build structures that can evolve, the discipline this dojo has been rehearsing since 2022. Apply the unchanged discipline to the changed center and you get the three responses in this post. To be honest about their novelty, as we were on stage: controlling from outside and observing to adapt are old disciplines that a probabilistic center makes bite harder. Only distributing trust is genuinely new.
Nearly 1,200 people stayed for the final slot of AWS Summit Japan to sit through this. I take that less as a compliment to us than as a signal of how many architects are wrestling with the same thing: systems whose most important component refuses to behave deterministically.
There is one more thing I hoped that room would carry home, because it is the reason the session exists. Business in the AI era is moving fast, and it should: none of this work continues unless the business succeeds. Right now, though, much of that energy goes into consuming what is already built, which product to adopt, how to use it well, how many tokens to buy. I believe technology has kept advancing through the ingenuity that grows out of human curiosity, and a business that stops engineering eventually runs out of new things to sell. The job is to hold both at once.
Neither problem on that stage was solved by choosing a product. Both answers run on managed services, and in both the gap was closed by what was built around them. In my conversations with customers the pressure is already pointing back toward engineering: token costs are becoming a line item that draws management attention. Engineering ingenuity is not what the AI era retires. It is what the AI era demands, and there is more of that work ahead than any of us can do alone, which is exactly how a dojo is supposed to work. The problems will keep changing. The way we design against them, I am increasingly convinced, does not have to. Happy architecting!
Session deck (Japanese): 2026
Past session videos (Japanese): 2025 | 2024 | 2023 | 2022
Earlier posts in this series: Architecture Dojo | 2025 | 2024 | 2022
