AI is starting to shape architectural mechanisms, workloads, and evaluation. To make sense of it, we need a compact, shared way to preserve enough of that process for other groups to evaluate and build on AI-assisted claims.

At the 53rd ISCA in Raleigh, AI for architecture stopped feeling like a side conversation. In the hallways, the talk kept coming back to one thing. AI is starting to enter the architecture design loop, the repeated process of framing a problem, proposing or editing a design, measuring it, rejecting weak candidates, and deciding what to try next.

In particular, there were two deep-dive workshops coupled with other activities. The MLArchSys workshop added A³, a segment on agentic approaches to architecture, and the full-day Architecture 2.0 workshop focused entirely on agentic design. Both drew well over a hundred people and were standing-room-only by the end. The same shift was visible in the main program, in a plenary panel on research and education in the GenAI era.

 

 

Full rooms on a particular subject matter are undoubtedly a sign of community momentum. The opportunity now is to turn that momentum into a durable engineering practice. That will take shared evidence, reusable tools, and enough agreement for a claim to leave the room where it was born and still be checked, compared, taught, improved, or rejected by someone else. AI is already producing architectural ideas. The question is what must travel with those ideas for them to become engineering knowledge.

Suppose a paper reports an AI-generated memory prefetcher with a 15 percent speedup. The code runs, and the speedup reproduces under the reported setup. But the agent saw some workloads and not others, adapted to simulator feedback, tried many candidates, and picked this one. What, exactly, is the contribution here? The final prefetcher? The speedup? The prompt? The agent? Or the process that connected them?

Once AI chooses workloads, responds to feedback, and selects which candidate to report, the same uncertainty about what counts as the contribution reappears for every such result. As AI gains more influence, we have to decide which evidence should accompany a result, so that another group can tell which part actually holds. This blog post is about the evidence that should accompany them if they are to become engineering knowledge.

The Scale of the Shift

The workshops reflect a broader rise in AI-mediated systems research. A recent cross-stack survey of more than 7,800 arXiv papers found that the annual AI-for-systems publication volume grew roughly 23× from 2017 to 2025, and even faster in hardware and chip design (GenAI for Systems). These categories reach beyond architecture, and not every paper in them runs an adaptive design loop. But where AI adapts to workloads, simulator feedback, or selection criteria, the final artifact can obscure how the result emerged. As this body of work grows, leaving that process implicit makes it harder to compare results or carry a finding from one group to the next.

Recent SIGARCH posts show the field working this out in public from different angles. Karu Sankaralingam asks whether architecture has reached an AlphaZero moment, with evaluation, not idea generation, as the real bottleneck. Dimitrios Skarlatos argues that agentic co-design is already reshaping the architect’s role and the hardware-software contract. Jeff Dean and David Patterson project a 10,000× future built on compounding gains, one of them AI automating hardware design itself. Together, their arguments point toward a common question about what should count as evidence when AI helps produce a design. Answering it requires being precise about what changes when AI moves from a bounded tool to an actor in the design process.

What Changes With Agentic Design

AI for architecture means using learned or agentic systems to help shape architectural designs and the evidence used to evaluate them, rather than building hardware optimized to run AI workloads. This direction, framed in recent work on the foundations of AI agents for modern computer system design, overlaps with software generation and electronic design automation (EDA), but it is distinct from both. An open-ended architecture agent can influence mechanisms, abstractions, workloads, simulator configurations, and interfaces whose effects propagate through many downstream programs and tools. That reach is what makes both its designs and its decisions worth scrutinizing.

Machine learning has entered architecture before. Perceptron branch predictors, reinforcement-learning memory controllers, learned prefetchers such as Pythia, surrogate models that approximate expensive simulations, and autotuners that automatically search configuration choices all used statistical learning to sharpen a mechanism or search a design space. In much of that work, ML was part of the artifact or a bounded optimizer, while the workloads, evaluator, and rules governing the search were set outside the model. The agentic shift is not a clean break from autotuning. It expands the scope and authority of the adaptive process. When a system can propose or edit mechanisms, call tools, choose workloads, adapt to simulator feedback, and influence which candidate survives, ML is no longer only inside the design. It starts to shape the claim we make about the design.

This concern predates AI. Human researchers explore design spaces, tune systems, and discard candidates, too, and research has always run on authors disclosing what others need to judge the work, backed by a degree of trust. What changes with an agent is how we scale. An adaptive system can make and revise these choices at machine speed across mechanism code, simulator configurations, workloads, tool calls, and selection criteria, often in response to the same evaluator that later supports the claim. The issue is not that a choice made by an AI system is inherently less trustworthy. It is that a large, tool-mediated search collapses into a final mechanism and a score, and the path that produced them disappears unless someone deliberately records it. The end goal is not to eliminate trust, but to keep that part of the methodology visible enough for others to assess the claim.

Gupta and colleagues’ ArchAgent makes this concrete by designing and implementing cache-replacement policies, not just their parameters. Starting from Mockingjay, a prior state-of-the-art policy, ArchAgent generated Policy31 for the single-core SPEC CPU 2006 suite, with a usage-intensity mechanism that its authors could inspect and test feature by feature. It also generated Policy12, which appeared to beat Mockingjay through what the paper calls a simulator escape, a higher score won by exploiting the simulator rather than the architecture. In ChampSim, unsupported bypassing of last-level cache writes was protected only by an assertion that optimized builds removed, so Policy12 looked faster because the bypassed writes vanished rather than being handled correctly.

The same agentic process produced both a genuine mechanism and a broken measurement, and the reported scores alone would not tell a reviewer which was which. The authors, to their full credit, caught the escape through manual inspection and reported it, exactly the kind of evidence future studies should preserve. A fuller record would not have found the bug automatically, but preserving the build configuration, the rejected policy, and the check that disqualified it would let others see why Policy12 failed and reuse that check in the next study.

ArchAgent also helps show where Karu Sankaralingam’s AlphaZero comparison holds and where architecture departs from it. AlphaZero discovered powerful Go strategies through self-play, but the board and the rules stayed fixed. Only the strategy could change. Depending on its permissions, an architecture agent can influence the strategy, the board, the rules, and the score used to judge it. Benchmarks can become data the agent adapts to, simulators can become environments it acts on through tool calls, metrics can become optimization targets, and interfaces define what actions it can take. That is why the claim must carry a record of the “design loop,” not just the artifact that emerged from it.

What Evidence to Preserve

Architecture papers already describe mechanisms, baselines, workloads, simulators, and evaluation procedures, so this is not a call for longer methods sections. What they rarely preserve is how the search reached the reported design. A compact record would make a few things visible:

• Bounds: what the agent could see and change, and what stayed fixed
• Feedback: how much simulator feedback it drew on, and how it steered the search
• Evidence and failures: what supported the reported result, and which candidates were rejected and why
• The decision: who could reject a candidate, who made the final call, and what would overturn the result

None of it is exotic. It is the part of the process that an adaptive search tends to erase.

Machine learning has already faced a version of this gap. A released model or dataset often lacked sufficient context to assess its intended use, evaluation, or provenance. In response, model cards and datasheets for datasets provided the field with compact records that accompany the artifact, short enough to read yet specific enough to state what the work does and does not support. Architecture needs the same kind of record, extended from a finished artifact to the search that produced it.

The reporting burden should scale with how much authority the agent had. If AI only helped implement a mechanism specified by a human, ordinary artifact disclosure is probably enough. If it chose workloads, edited the design, adapted to simulator feedback, or determined which candidate was reported, some account of that process should accompany the result. A simple test is whether AI materially shaped the mechanism, workload, evaluator, stopping rule, rejection rule, or reported result. That record need not become a universal checklist, expose the model’s private reasoning, or promise an exact replay of a randomized search. Its format should emerge through use and revision rather than being settled in advance. What matters is how much of an AI-shaped process must stay visible for another group to see why a result survived and whether it holds under different assumptions.

Existing practice offers only partial precedents. Declaring the setup before a search resembles preregistration, keeping the evidence trail resembles artifact evaluation, and preserving failed alternatives resembles ablation studies, which test the effect of changing one part of a design, as well as negative result reporting. The individual practices are not new. The change is that a single adaptive system can operate continuously across the design, workload, evaluator, and stopping rule, which are usually documented separately.

A record like this is a good-faith disclosure, not proof. An author can omit an inconveniently rejected candidate, and a reviewer cannot rerun an adaptive search to catch the omission, especially when the agent relies on a proprietary model that shifts over time and never repeats a run exactly. The record cannot stand on its own. What keeps it honest is disclosure scaled to the agent’s authority, read by reviewers rather than filed as a badge, and confirmed against evidence the search did not produce.

A result selected through adaptive evaluation should face at least one confirmation check outside the search, using held-out workloads, a second simulator, or a targeted test of the claimed mechanism. If the same agent tunes against the simulator that scores it, selects its evaluation workloads, and stops once the metric looks good, the evaluator has become part of the optimization loop, the architecture equivalent of training on the test set, or evaluation leakage. The check must also use measurements appropriate to the claim, and as a recent SIGARCH post on full-system timing simulation argues, simulation speed and fidelity are already in tension before an agent begins optimizing. Agent feedback makes the measurement window and metric part of the search surface, so authors should explain why the confirmation is credible and what evidence would overturn the result.

The same shift that put agents into the design loop is now putting them into the review loop. In systems research, agents already drive the design loop end-to-end, and elsewhere they draft and review their own papers, with a language model serving as the judge. An automated judge can share the blind spots of the system it reviews, so it does not replace the independent check. But it does raise the value of a record built to be machine-readable as well as human-readable, one that the next agent can use to rebuild the setup, rerun the disqualifying check, and test the claim rather than take a summary on faith.

A Shared Layer for the Loop

A record inside a single paper is a good start. It becomes a shared convention when authors use common fields and present supporting evidence in a form others can inspect. A reviewer can then challenge the record, a student can learn why the reported design survived, and another group can revisit a rejected candidate under the same conditions. Some variation in this scaffolding is healthy, but shared infrastructure gives groups a common base without requiring them to pursue the same research questions.

The computer architecture community has previously built shared responses to analogous coordination problems. SPEC provided common workloads, while simulators such as SimpleScalar and gem5 provided researchers with reusable experimental platforms. MLPerf and the long line of prediction and prefetching championships showed that we can agree on workloads, rules, and scoreboards. These shared objects did not settle every question, but they gave the field durable things to run, dispute, teach, and improve. Benchmarks do not capture the path through an adaptive search, but they show how common boundaries make comparisons meaningful. Agentic design now needs a similar layer for search state, allowed actions, failures, and independent checks.

When Amir Yazdanbakhsh and I first articulated the Architecture 2.0 vision in a SIGARCH blog post in 2023, it was conceived as a data-centric AI gymnasium, a shared ecosystem of data, benchmarks, and tools for ML-assisted architecture research. Three years later, many of those building blocks have emerged, including knowledge benchmarks such as QuArch, assembled with the help of more than 140 contributors across 40 institutions, capability evaluations such as ArchEval, and design-space infrastructure such as ArchGym. Building them has taught us something more important. The remaining challenge is not simply another benchmark, evaluation, or piece of infrastructure. A benchmark tests what a model knows, an evaluation tests what an agent can do, and infrastructure runs the search. Some of these systems log a run in detail, but that record stays inside the tool. What no published result yet carries with it is a portable account of how a study bounded its search, rejected candidates, and chose what to report, and that is the part we cannot supply by building one more tool.

Making It Routine

Making this kind of evidence part of everyday research practice will take deliberate community effort. Machine-learning communities have shown one path. Benchmarks and competitions provide participants with shared tasks and rules, while model cards and datasheets establish shared reporting expectations. NeurIPS requires submissions to include a paper checklist addressing reproducibility, transparency, limitations, and experimental details, a step it introduced to help authors document the completeness and limits of their work. ICML has likewise published paper guidelines, based on the NeurIPS checklist, that ask authors to document claims, limitations, code, data, and experimental details. A complementary perspective appears in the 2022 FAccT paper by Smith and colleagues on REAL ML, which argues that responsible machine learning depends not only on models and metrics, but also on documenting the broader research process.

Architecture conferences and workshops could experiment with a few concrete practices. Artifact-evaluation tracks can request versioned configurations, failures that affected the result, and at least one confirmation check that was not used to select the reported result. Competitions can specify workloads, allowed actions, limits on evaluator queries, stopping rules, and held-out tests to ensure scores remain comparable. We do not need to harden these practices into permanent rules at once, and venues can learn what helps reviewers and discard what does not. For proprietary work, the detailed record may remain internal, but a public claim still requires sufficient disclosure for outside groups to assess it. The simplest shared form for that disclosure is a single page. Authors could include or link to a one-page design-loop card summarizing the process in a consistent format.

These conventions also need a public home alongside tools, benchmarks, failure cases, and examples. The Architecture 2.0 hub is one possible starting point. Its value will depend on whether multiple groups use, challenge, revise, and help govern its contents.

No single convention will make AI for architecture an engineering discipline. The value of a shared record is that it lets a result move beyond the group that produced it so someone who was not there can check it, build on it, or challenge it when the evidence does not hold. The full rooms at ISCA were the momentum. Making the evidence travel with the design is what turns momentum into a discipline.

About the Author

Vijay Janapa Reddi is the Gordon McKay Professor of Electrical and Computer Engineering at Harvard University and a visiting professor at ETH Zurich. His work spans computer architecture, machine learning systems, and autonomous agents. He is Vice President and a board member of MLCommons and the author of the open-source Machine Learning Systems book.

Disclaimer: These posts are written by individual contributors to share their thoughts on the Computer Architecture Today blog for the benefit of the community. Any views or opinions represented in this blog are personal, belong solely to the blog author and do not represent those of ACM SIGARCH or its parent organization, ACM.