Making Hardware Decisions Before the Hardware Exists

Why I wrote “Should You Split Decode?”

Every accelerator starts as a bet. However the leads bend the story afterwards, it still starts as a bet. Someone intuitively predicts the future workloads and how to deploy them, and every decision after that builds on the bet. By the time the hardware arrives two years later, after countless hours and headaches, the models have changed. That was the story of TPUs, GPUs, MTIAs, and so many other successful or unsuccessful chips. Some were more adaptable, in some the leads were more aggressive, and some were pure luck. The bet is still placed on what we know today.

I keep coming back to the title of Hennessy and Patterson’s Computer Architecture: A Quantitative Approach. As an undergrad, that was my favorite book. However, I never paid attention to its title. Years later, I realized that Patterson was arguing for quantitative reasoning over intuitive or philosophical arguments about design (see his oral history). Back then, a performance decision might mean a better branch predictor or a few more cache sets in the next product generation.

Now it can mean a gigawatt of capacity competing with cities for power, a much larger environmental footprint, and a lot more pressure on the economy. And still, most of the industry operates at the level of tick-tock: slow-paced, committing to hardware before knowing the workloads it will actually meet, that kind of thing. For most of my career, I thought of a performance estimate as an engineering tool. It told us whether a design idea was worth pursuing and partially told others whether the product was competitive. That is no longer its only job. Today the same estimate feeds decisions about data centers, power, supply chains, and a large share of the capital moving through our industry. Architects and designers are no longer the only consumers of our numbers. I think that calls for more organized, more deliberate performance modeling, with assumptions that people outside the architecture community can read and question.

One company’s view is not enough

A lot of hardware decision-making still runs on experience. That worked well for CPUs and microcontrollers. Decades of designs taught people which tradeoffs hold and under what conditions. That knowledge is still valuable. But today every layer of the stack is moving at once: the models, the software stack, the compilers, the memory, the interconnect, and the serving policy. Experience from the last generation says less about a system that does not exist yet, or about models that its designers only recently learned to code in CUDA.

During my PhD and in industry, I have built simulators that represent hardware at many levels of detail. Inside a company, that work sometimes goes deep into the details of one design. You need to know your own product far better than anyone else’s. Still, many companies start even that simulation endeavor much, much later, after a chief architect has laid the cornerstones of the design.

But we are at a different infrastructure level now, and it asks a different question: which of these designs should we build or buy, and dedicate a gigantic amount of resources to, including political capital? Performance is only part of that answer, and it is the part I’ll focus on in this post, even though the modeling does not stop there. For instance, within performance, low latency for one user and high throughput across many users can favor different designs. Answering the question means modeling the competitors with the same care, the same baselines, and the same models. As anyone who has worked on a hardware product team knows, a data sheet or a rough spreadsheet can get you started, but it never answers the questions that really matter. It rarely survives a change in workload, hardware, and deployment at the same time. And each vendor’s numbers come from that vendor’s own environment, with its own workloads and baselines. That comparison can be unfair to everyone else, even when nobody intends it. It also leaves people making infrastructure decisions with numbers whose assumptions are hard to inspect, let alone compare.

We are not even talking about the main challenge yet: the workload the chip will actually meet in deployment. At design time, today’s workloads are usually deconstructed into tiny pieces so we can be efficient in modeling and comparing options. You’d ask, what about the actual workload in two years? Nobody knows it! But we can model several directions and ask which decisions survive, and which ones we might regret.

So, in reality, to make a decision on the future of AI infrastructure, we have to model how workloads change, how chips evolve, how resources at the global level change, and so on. Unfortunately, most simulators I know focus on hardware that exists today. And I did wonder what would happen if I modeled a few of these directions, maybe as a toy example.

Why attention/FFN disaggregation

That is far more than one person can build in their spare time. So I picked one decision to study across a few possible futures. During LLM token generation, attention reads the conversation’s key-value cache. The feed-forward network (FFN) reads dense or expert weights and does most of the arithmetic. The two stress different resources. Attention/FFN disaggregation, or AFD, puts them on separate pools of accelerators. Splitting the work and choosing different hardware for each side are two separate decisions. Several startups argue that the future is heterogeneous: specialized hardware for each phase of the work. The story is easy to follow, or to believe. But I was never a firm believer in it. Different resource demands do not automatically justify different hardware classes. The argument for specialization is that the models are so large that we have to go beyond a single die anyway, so why not specialize each part?

But still, no workload is homogeneous. Architects have handled this inside a single chip for decades. Caches exist because access patterns differ. Trace caches exist because instruction streams differ. A branch predictor learns patterns that each look almost like a small program of their own. Attention and FFN differ at a much larger scale, and I agree they stress a system differently. At data center scale, we can build beyond a single reticle, with the help of all these new fancy scale-up and scale-out networks. So the question becomes much easier to track. Once you can scale out, do attention and FFN still want the same hardware class, say 3D-DRAM? Or is the imbalance large enough to justify different hardware classes on each side? Remember that you pay for every extra hardware class, on top of the extra communication and a second pool to size, not to mention the pain of programming them, deploying them, managing them, caring for their reliability and uptime, and all the other fancy terms.

So that’s how I ended up writing this paper, single-handedly, after a while away from writing papers. It was fun!

The two decisions the paper separates. Rows set where the work is split: prefill/decode disaggregation alone (P/D), or with attention/FFN disaggregation added (P/D+AFD). Columns set what hardware runs each side: one accelerator class for every pool, or a different class per pool. P, D, A, and F stand for prefill, decode, attention, and FFN. The paper varies the two axes separately. From Figure 1 of the paper.

What I found

I built something called SiliconForge, a performance model calibrated against published measurements, and wrote the paper Should You Split Decode? Disaggregation Across Future Models and Accelerators. It compares serving with and without AFD across four model directions, four accelerator classes anchored to current and announced parts, and two fabrics. Every design gets the same power budget and the same rules. Each one picks its own mapping and pool sizes. The performance metric (for now) is requests served within latency targets per kilowatt (SLO goodput per kW).

The main comparisons showed no robust deployment-level advantage for AFD. Most nominal estimates favored keeping attention and FFN together. The largest AFD lead did not come from a more efficient decode pool. It came from fitting one more prefill replica into the same power budget! AFD did lead nominally in one model direction, one with a heavy key-value cache, when the combined design had to spread its experts across several small network domains. Splitting let the FFN pool stay within one domain. Larger domains, or a fast fabric available to both designs, weakened that lead. That makes me cautious about treating the split as a permanent design assumption for the long term.

An example of what SiliconForge covers: 58,310 feasible DeepSeek-V3 deployments (16K input and 16K output tokens, up to 128 units and 1 MW) across six hardware archetypes, built from current, announced, and speculative parts. This catalog is broader than the four classes used in the main comparison. Each dot is one deployment. Moving right means faster tokens for each user; moving up means more tokens per kW. Bands cover the 10th to 90th percentile of deployments, and panel (f) has its own x range. And this is just one model; you can imagine how the design space explodes when you add future model directions. From Figure 3 of the paper.

The result I care about most is about flexibility. A hardware choice that wins today can lose its lead as models, fabrics, and objectives change. Across the tested scenarios, committing to a hardware choice early cost a median 36% in goodput relative to the best deployment for each scenario, not to mention the capital and effort. Re-optimizing the mappings and pool sizes on the same hardware cut that loss to 9%. My takeaway is to keep mappings and pool sizes adjustable. I also varied the hardware specifications to see which conclusions survive. Few comparisons were decisive, but you can read more in the paper.

These are just a few data points, or rather insights backed by some results. You can definitely argue against them for your case, or build a different future. That’s fine. Let’s put the assumptions on the table and discuss them, instead of everyone doing the same thing behind closed doors.

The larger goal

I wrote this paper as a single author, in my own time. I have considered myself part of the architecture community since the start of my PhD. I have submitted countless times to HPCA, MICRO, ISCA, and ASPLOS, and have served on their program committees for the past five years. A month or two before the HPCA deadline, I decided to write this paper. I thought the community would benefit from it; studies like this are rare, and I think they matter more now than they used to.

A few days ago, the paper was rejected. Thankfully, it wasn’t as painful as before; my degree doesn’t depend on it. But still, I put so much effort and time into crafting it. The reviews questioned its novelty and raised some concerns that were hard to pin down. I worked on the revision day and night (it’s customary to submit an updated paper after getting the first round of reviews). I rewrote almost everything and reran the experiments to address the reviews. I was disappointed.

But I understand why. It does not follow the usual shape of an architecture paper: an idea, a simulation, and a comparison against prior work. We have to model the future from assumptions, and the assumptions will change. My goal was a framework flexible enough to absorb those changes and rerun the comparison. I think studies that help people understand a consequential decision deserve room alongside papers that propose a new mechanism.

The rejection felt familiar. All but one of my PhD papers were rejected more than once. Several of them traded accuracy for inference latency. At the time, giving up even 1% of a CNN’s accuracy was hard to defend, when everyone was working to squeeze out 1% or 2% more. Today accuracy versus speed plots, quantization, and similar tradeoffs appear everywhere, in our community and well beyond it. That experience made me wary of reviews that predict which tradeoffs will matter in the future. Modeling work still deserves hard technical criticism. I just want that criticism to ask which conclusions follow from the assumptions, where the model breaks, and what evidence would change the answer.

There is also the question of when this paper is useful. I was just at MICRO, and the volume of published work is already impossible to keep up with. A study like this can go stale within a month or two. One new model or one hardware announcement changes its inputs. A resubmission cycle measured in months or years is a poor fit for that pace. I also no longer need conference papers to finish a degree, and I have limited time to spend on resubmissions. So I decided, why not, let’s try arXiv again, and maybe someone learns something from it.

So I am putting the paper on arXiv. I expect my tool and my results to change as the inputs improve. That is the point of sharing them. Tools like this usually stay inside companies, and they take real effort to maintain. The people who can build them are few, spread across many companies, and usually focused on the current product. Meanwhile, more and more people along the hardware pipeline, from custom chips to infrastructure to investment, care about how these systems perform. I think we have a responsibility to make that knowledge public, beyond any one product, company, or review cycle.