Search for a command to run...
Short, sweet, and packed with value. Love it!
On stage, you declare a design correct. In production, you find out what that correctness was quietly resting on. In 2022, at the first Architecture Dojo at AWS Summit Japan, I stood on stage and led
An AI coding agent does not bend a productivity curve on its own. A team does. Coding agents, models, and IDE plugins ship new versions every few months, and most of what gets called "best practice" i
Most teams pick the AI agent evaluation tool their framework integrates with, then a quarter later notice that one corner of the evaluation space is covered and the rest is exposed. The market has no
Some incidents look minor on paper. A small single-digit percentage of instances affected, in a single AZ. And yet the user-visible outcome is that more than half of the service's transactions stop wo
Build at Scale — System Design, Distributed Systems, and AI
7 posts
Architecture lessons from real-world systems at scale. I write about the trade-offs, constraints, and design decisions behind production systems, from cloud architecture and distributed systems to AI in production and the organizational choices that shape how systems evolve. Each post digs into why a particular design choice was made, what alternatives were considered, and what you can take away for your own systems.
A proof of concept that passed in 2022 failed its load test in 2025. Nothing in the technology had regressed. The acceptance criteria had grown up.
The project behind that sentence is one of the largest data warehouse migrations Japan has seen: more than an exabyte of data on premises, over three petabytes of it bound for Amazon Redshift, around three thousand users, and roughly 725 million queries a day. Everyone expects the hard part of such a migration to be the move itself: shipping the data, porting the objects around it such as procedures and permissions, rewiring the workloads of every surrounding system. What actually consumed the years was agreeing, across many teams, on what "fast enough" and "correct enough" mean, and then watching the first version of that agreement meet production-scale load. At this scale, no proof of concept can rehearse every query and every access pattern. Some criteria only become visible when the real workload runs on the real system. The acceptance criteria were never a specification fixed at the start. They were something we grew, and this post is about how.
Where I'm coming from. I spent more than six years alongside this migration as a solutions architect, from the first proposal through the PoC to the cutover. The customer, LY Corporation, has told the story from their side at AWS Summit Japan 2026, and every customer-side number in this post comes from that public deck (in Japanese). I refer to the source system only as the previous on-premises warehouse, and I stay inside what the customer has shared publicly. The customer's team reviewed a draft of this post before publication to confirm it stays within that boundary. What follows is the engineering and project judgment I watched up close, not the commercial side. The views are my own.
TL;DR
"Same or better than the previous system" only becomes an acceptance criterion once you decompose it. Three-grade scoring, workaround costs priced in, and real queries instead of standard benchmarks turned a slogan into something a company could decide with.
At national scale, the criteria only fully arrive under load. Hunting down the minority of queries that got slower reads as tuning, and it is also how you find what the first criteria failed to capture.
A missing feature deserves one of three stances: wait for it, accept its absence for good, or keep the workaround cheap. Assign a stance per gap, and never let a single "it can't" hold the whole migration hostage.
The criteria work started with a directive from the customer's side. When the future of the on-premises warehouse came into question, the department's CTO at the time directed the team to compare multiple candidate products thoroughly, against explicit criteria, before committing to a next-generation platform. The customer's data warehouse team turned that directive into a scoring system, and we agreed to be scored by it. I welcomed that. My rule with customers has always been to be confident in the product and neutral in the evaluation, because the neutrality is what earns trust. It is also what gave the result its weight: a strong score on a scorecard we had shaped ourselves would have persuaded nobody, least of all the CTO who had demanded the comparison.
The system had 59 items spanning functional and non-functional requirements, 124 points in total, each item earning a three-grade score against the previous warehouse. More telling than the score was a second mark per requirement: achievable, achievable only through a workaround, or not achievable at all. That distinction is the part I would carry into any large evaluation. A feature checklist tells you what a product has. The workaround mark prices what it lacks: "can it do X" stops being an argument and becomes a line item. And the marks have a second life. Collect every requirement scored as workaround-only, and you are holding the first draft of the migration backlog, long before anyone writes one.
Two more choices shaped the evaluation. First, the team concentrated its testing on performance. Functional requirements and the usual non-functional ones rarely separate mature warehouse products, so that is where the scoring went deep: single-query performance was scored query by query rather than as one requirement. Second, they declined standard benchmarks. TPC-DS models a generic workload rather than yours, and nothing about it models several business units sharing a single cluster as tenants. Instead the team pulled 400-plus real queries from the previous warehouse, cluster by cluster, spread across statement types from SELECT to MERGE and DELETE, across complexity tiers, across data volumes.
Redshift scored 96 of 124, and with total cost of ownership added to the technical result, it was selected as the next-generation platform in the first half of 2023. The finding I keep returning to, though, is the one the customer states plainly in their deck: SELECT-heavy work showed a clear advantage, while very large INSERTs and queries containing certain predicates could get dramatically slower. The decision did not hide that. Those queries would be rewritten, and the rewriting cost was added into the TCO. From the first scoring pass, a slower query was treated as a line item to be priced rather than a reason to reject the platform.
These criteria did what they were built to do, which was to choose a platform using whatever could be reproduced and priced in advance. Defining "good enough to run production on" is a different question. It belongs partly to different people, the ones closest to the heaviest load, and it could only be asked properly two years later, once there was a production-shaped load to ask it with.
The migrated platform runs three Redshift clusters built from ra3.16xlarge nodes and stitched together with data sharing. In 2025, with migration underway, the division serving the heaviest internal users raised a direct concern. The PoC had verified 400-plus representative queries, and "representative" is the word doing all the work in that sentence: whatever you select, it is a thin slice of the 725 million queries a day that production actually asks, and a passing slice did not convince them the platform would hold under real operations. So the team rebuilt the test. They captured the previous warehouse's busiest window and replayed it against Redshift, each query injected at its original timing, and with that the acceptance criterion became concrete in a way it had never been: 95 percent of the queries must complete within 38 minutes of the replay starting, the previous system's mark for that window.
Nobody had been careless in 2022; the criterion was unwritable then, for three reasons I suspect generalize well beyond this project. Peak load drifts with products and seasons, because data is a living thing and so is the business running on it. The divisions generating the load and the team selecting the warehouse sit in different organizations, and a numeric definition of acceptable does not cross that boundary before the platform decision. And some of it is simply unknowable until you run it. An exit criterion is a hypothesis. If you could write perfect hypotheses up front, you would not need the PoC that tests them.
More than seven timed runs followed; the seven runs the customer's deck walks through trace the arc. The first missed the completion bar badly, 83 minutes against the 38-minute target. The configuration, not the size, changed first: the second run, the first on a Multi-AZ deployment, came back worse at 100 minutes, and the third, now on automatic workload management, cleared the completion bar. The deck grades it as a failure anyway: partway through the series a second bar was added, average response time no worse than the previous system's, and every run is scored against the final set. The exit criteria were being revised while the runs that tested them were underway. The fourth run passed both bars on nearly twice the planned size, a standing price nobody wanted to sign up for, so the last three runs asked how small the deployment could get and still hold. Walking the size down broke the response bar again, a step back up was not enough, and the seventh passed both bars roughly a third smaller than the first passing run, still a Multi-AZ deployment, at 80 percent of the target response time.
The final configuration deserves a pause. Multi-AZ is a feature provided to improve availability; the team adopted it to raise concurrent performance at a fixed cost, with both compute layers active and taking queries. The customer's deck is upfront about the trade that comes with that choice: certain query types do not route to the secondary. The choice needed explaining inside the project too. On a resource summary, two active compute layers read as the migration needing more hardware than planned, when the second layer was working capacity rather than standby overhead. So I charted how workload and required resources moved together over time and walked stakeholder after stakeholder through the same chart: engineers weighed the performance results, the people accountable for spend weighed the economics, and one picture had to answer both. Consensus at this scale is built explanation by explanation.
Running and revising until the criteria hold has one boundary worth naming: a few settings can only be decided before data arrives, and no later run will reopen them. The previous warehouse compared strings case-insensitively; Redshift can match that behavior with database-level collation, but only at creation time. Reproduce it inside queries instead, by wrapping join keys in collation functions, and joins quietly degrade: merge joins collapse into hash joins, and intermediate results spill to disk by the terabyte. The last row of the tuning table below shows what that costs. Growing the criteria under load works because most migration decisions are what Amazon calls two-way doors: reversible choices that the cloud makes cheap to walk back through, run after run. A few are one-way doors, and in a migration they rarely announce themselves; this one hid inside a collation setting. Part of the design work is finding those doors before the first table loads.
Most queries got faster after migration. A stubborn minority got slower, and at this scale the minority is not a rounding error; it is where the timeline lives. The hunting method, which the colleague who led performance engineering presented publicly at AWS Summit Tokyo 2023 (deck in Japanese), fits in two moves. First, decompose time before guessing: the SYS_QUERY_HISTORY system view splits elapsed time into queue wait, compile, planning, execution, and lock wait, killing whole categories of speculation in seconds. A query stuck behind a lock, a query stuck in a mis-sized workload-management queue, and a query that spends three minutes being compiled are three different problems, and only one of them will respond to data-layout tuning. Second, classify the slow ones by which layer saturates which physical resource; four patterns covered most of what this project met. The numbers below come from the public deck's benchmark reproduction, not from customer data:
| Bottleneck pattern | What saturates | Representative fix | Benchmark before → after |
|---|---|---|---|
| Work piles onto a few slices (distribution skew) | A few slices, on CPU, memory, and I/O alike, while the rest idle | Redistribute on a high-cardinality column | 91 s → 33 s |
| Sort order misaligned with filters | Storage I/O, excess blocks read | Reorder sort keys, then VACUUM | 2.2 s → 720 ms |
| Joins redistribute both sides across nodes | Node-to-node network | Align join keys with KEY distribution | 4 min → 3 min |
| Collation functions demote merge joins | Memory, intermediates spilling to disk | Set collation at the database level | 26 min → 2 min |
On the customer's own workload the same method paid for itself repeatedly. The example I keep, deliberately rough because it is the customer's operational data: a transformation that used UNPIVOT to reshape columns into rows on its way into another table had been killed after running past the half-day mark. The rewrite, by the same colleague, pre-built a small dictionary table and produced the same rows with a CROSS JOIN against it; the job finished in under an hour. I still consider that the sharpest single rewrite of the effort. Splitting another runaway job into date-range chunks occurred to everyone; seeing a reshape and reaching for a pre-built join did not.
None of this reads to me as tuning in the narrow sense. Every slow query diagnosed and every failed concurrency run rerun was a piece of the acceptance criteria arriving late, because it could not have arrived early. The criteria were being finished in the only place they could be finished: under load.
Not every gap needs load to reveal it. Some are binary and obvious from the first desk check: a statement, a constraint, a console capability the platform simply does not have. The PoC catalogued a set of these in 2022, and finding them was the cheap part. What mattered over the following years was the stance the team assigned to each entry, and in hindsight the stances were three.
1. Wait, with a bridge. Some gaps sat on the vendor's roadmap. The MERGE statement was one: until it shipped, upserts were decomposed into DELETE plus INSERT. QUALIFY was another, covered by window-function rewrites, and longer retention of console query history a third. All three closed within roughly two years of the PoC, MERGE and QUALIFY reaching general availability in 2023. The judgment call is whether your schedule can absorb the wait, and one catch is worth mentioning: general availability does not retire a workaround by itself. A rewrite that is already deployed, tested, and performing well can outlive the gap that forced it, and that is fine.
2. Accept, permanently. Other gaps follow from the product's design philosophy, and the documentation says so. Redshift does not enforce uniqueness constraints: primary and unique keys are informational, used by the planner rather than checked at write time. Distribution is keyed on a single column. If the previous warehouse's constraint enforcement had been catching duplicate loads for you, that responsibility moves into your pipelines, and it never moves back. The judgment call is whether the standing operational cost is one you can pay indefinitely, and pricing exactly that cost is what the workaround mark in the PoC scoring existed to do.
3. Keep it cheap. The subtlest stance, and the one I find myself recommending most often. Some gaps are simply the product's current state, likely to be erased by ordinary engine progress. COUNT(DISTINCT)-heavy queries were a known weak spot at PoC time, and teams rewrote them as COUNT over GROUP BY. In 2023 an engine improvement made the original form fast, and the rewrites, which had deliberately been kept shallow, could simply be dropped. Before investing in a heavy workaround, ask which kind of gap you are looking at: one the vendor will close for everyone, or one that is yours to own. Build accordingly, and keep the removable kind removable.
The three stances share one purpose. They stop any single missing feature from becoming a deal-breaker. More than once during the PoC an item threatened to grow into a knockout requirement, and pulling those conversations back was my job. Each time I made the same argument: the evaluation had already agreed on its deciding criterion, total cost-performance of the platform with workarounds priced in, and a single gap gets a price inside that frame, not the final say. Individual gaps get a stance. The migration gets judged on the portfolio.
The migration project proper took about a year and a half and completed at the end of March 2026, roughly five years after the PoC studies began. The customer's own post-migration measurement covers 1,475 production batch tasks over one week: 26 percent faster on average than the previous warehouse. Extraction-type tasks ran 46 percent faster, aggregations 17 percent, load-and-merge tasks 21 percent. One category, update-style tasks, came out slower than before. I find the deck's willingness to print that last number as valuable as the wins around it. Teams hide regressions when the definition of success is vague, because any single bad number can end up standing for the whole project; a portfolio-level definition of good enough is what makes honesty affordable. Show me a migration report with no regressions in it, and I will wonder what the criteria were.
The gains that changed daily life do not fit in a percentage. Retiring the on-premises hardware also retired the machine-room visits and the long planned outages. Standing up a purpose-built cluster became an exercise in templates rather than a purchasing cycle. And the platform now sits inside an ecosystem that keeps handing it new options, from zero-ETL integrations onward.
The customer's AWS Summit talk closes by pointing forward: MCP servers in front of the warehouse so people can query it in natural language, and AI-assisted first response to operational alerts, read-only at first, with human approval gating any change. I read that final chapter as the next round of the same story. What "useful enough" and "safe enough" mean for AI agents touching a multi-petabyte warehouse is a set of acceptance criteria nobody can fully write in 2026, and they will be grown the way these were, by running the real thing and revising. That story deserves its own post. The same pattern runs through the other problem I write about here, evaluating AI agents: probabilistic systems whose acceptance criteria are even less writable up front, and growable only by running them.
Six years taught me a shorter version of all of this. The 2022 criteria were as good as anyone could have written in 2022, and production still grew beyond them, on schedule and with no one to blame. At national scale, the load, the failure modes, and, above all, the people are never all visible at the start. So the advice I would send back to myself is not "write better criteria" — it is to build, from the very start, a way of working that expects the criteria to grow: score workarounds honestly, replay real load as early as the project allows, and keep every gap assigned to a stance so that no single "it can't" owns your timeline.
