The First AAI4Science Developer Summit

Scientific results are still published much as they were when journals traveled by post: months of work compressed into a monolithic PDF which is “a digital representation of a bit of printed paper.” This session asked what scientific publication could become in an agentic era and how it might serve both human and machine readers.

Beginning with an ambitious vision for the future, we considered what a scientific corpus might look like as a knowledge graph composed of atomic units such as questions, claims, and evidence. In such a system, claims could be connected and reused across projects, sparing every reader or agent from repeatedly mining the same PDFs. Yet research rarely fits neatly into a fixed schema, and it remains unclear whether formal concepts and knowledge graphs can capture the reality of scientific investigation. The significance of a result is also contextual: something not recognized as a worthwhile claim in one project may prove fundamental to another when viewed in a different light. How to handle contradictory evidence and claims also raises further questions about verification, and hints at the fact that a machine-readable scientific record must preserve uncertainty and disagreement, not merely organize facts.

But maybe more fundamentally, even a very well-constructed repository of structured knowledge cannot replace narrative. Humans understand complex ideas through stories that establish context, explain significance, and guide readers through an argument.

The emerging consensus was therefore to separate scientific outputs into two layers: a machine-readable record of claims, evidence, and code beneath a concise narrative for human readers.

Crucially, the structured layer should be extracted from the papers, repositories, and results that scientists already produce, rather than imposed as another authoring burden. As an important side effect, if machines can inspect and work on the full record, researchers might reserve human-facing publications for fewer, more meaningful contributions: “Write less, so we can read more.”

The hardest obstacle to the emergence of this shared compendium of research is that scientists need reasons to share intermediate results, failed attempts, and abandoned ideas. Strong provenance and attribution of atomic units of research can ensure that credit follows reused work, but attribution only becomes a real incentive when funding institutions recognize it.

The discussion explored two related topics: (1) how projects should preserve and organize the agent memory needed for long-horizon, AI-assisted scientific work, and (2) where the agents conducting that work should run.

For the first, we defined “long-running” as AI-assisted scientific workflows that may span several weeks and many short-lived agentic sessions, making persistent project memory and context—rather than prolonged autonomy within a single session—the central requirement. To that end, we started by reviewing Neurodesk, a neuroimaging environment in production. Neurodesk demonstrated how a stable, containerized environment, discoverable tools, machine-readable data standards, and a short file of universal instructions can prevent each new agent from having to rediscover the project.

While Neurodesk showed how to reduce what each agent needs to rediscover, several independent setups in the room addressed the complementary problem of carrying project-specific context across sessions. Despite evolving separately, they mostly converged on the same answer: a directory of cross-linked Markdown files that the agent writes to before compaction, committed to Git, and handed to the next session or the next model release. Nobody had adopted a database or a dedicated knowledge-graph framework; both were judged to be harder to inspect and maintain than plain text. But everyone was clearly reaching for the same missing piece: a shared way to record which decisions were made, what evidence backed them, and which alternatives were rejected.

For the second, we discussed where the agent should run. Regulations in place at many European computing facilities currently prevent scientists from installing agents directly on clusters. Alternatives involving a local agent sending cluster jobs through SSH were discussed, but this kind of interaction partially blinds the agent by preventing it from accessing its own outputs live, necessarily reducing its capabilities. Others mentioned running agents directly on the cluster under the assumption that “if you can do it, you’re probably allowed to.” Everyone agreed on the use of containerized environments to limit the potential damage when running many unmonitored agents in parallel. Finally, a third approach, HPC Application Service (HAPS), would expose to agents running off the cluster a limited set of pre-approved HPC workloads—such as simulation or imaging codes—that can be executed on HPC resources via API calls. Such an architecture is already under development as part of France’s exascale-preparedness NumPEx program.

The session focused on two questions: (1) when designing tooling, how can one build a feedback loop that surfaces scientists’ real needs? And (2) what would make an agentic system for scientific research genuinely enjoyable to use?

For the first, the group noted that scientific software teams often lack clear signals about how to improve the user experience. User research is essential for understanding what scientists actually need and want from a software product. Direct observation—which can be accomplished by embedding with scientists and watching them use early prototypes—is therefore an important process for revealing real workflows and points of friction. Technical implementation matters, but so do the product’s touchpoints. When it comes to a completely new technology and new interaction patterns, as with AI, even technically sophisticated scientists may be unsure how to start using these tools, so testing should aim for concrete, familiar tasks. One example might be reproducing a scientist’s paper and letting them quickly explore how changing an assumption affects the result through interactions with an agent.

For the second, the group argued that in the process of producing a scientific result, AI should reduce implementation burden without diminishing scientists’ understanding, agency, or joy of discovery. Moreover, it is important that scientists remain accountable for the work they publish under their name, even if it is assisted by AI agents. To that end, agents must produce inspectable results by default, so that scientists are able to vet them before publication. Heavily agent-generated or anonymous research may still be useful if clearly labelled and held to strong reproducibility and verification standards. The ideal product should ultimately free scientists to focus on the scientific questions and choices that matter while allowing implementation details to be handled by agents, but still ensuring that those details are accessible for inspection.

This session began by asking what “access to frontier AI resources” meant. Two distinct needs emerged: the capacity to train and fine-tune frontier AI models, and unrestricted inference access to state-of-the-art models.

On both fronts, it was clear that researchers do not compete on equal footing. Access often depends on the country in which they work, which determines their eligibility for national computing facilities, or on their country of origin, as illustrated by recent U.S. government restrictions limiting access to Anthropic Fable. Because current AI capabilities allow researchers to move from an idea to a result at unprecedented speed, such unequal access to frontier models creates an imbalance that compounds every cycle. Privileged access attracts more funding, collaborators, and computing resources, further concentrating capabilities in a small number of already well-resourced institutions and countries. Subscription prices are also likely to increase over the coming year, which would further affect the ability of researchers around the world to access frontier models.

To address these inequalities, the discussion highlighted several ideas and existing initiatives. Recent open-weight models appear to be on par with commercial models and well suited to research. The primary bottleneck is not their quality, but the specialized hardware required to run them at scale, which remains scarce.

One of the most promising proposals was a federated academic computing platform: a single gateway providing access to open models deployed across academic computing facilities around the world. A federation of this kind is already operating in France and has expanded over the past year to pool the resources of 18 universities. Extending this approach across continents could reduce dependence on commercial providers, maximize the use of academic inference infrastructure by keeping it active around the clock, and lower the overall cost of access.

As AI agents take on increasingly autonomous scientific work, verification becomes a more difficult question. Formal proof systems such as Lean can provide strong guarantees in pure theory, but most empirical and computational science lacks a complete formal specification or accessible ground truth. The session therefore distinguished correctness—reaching the right conclusion—from soundness—following a defensible scientific process. The central question became: which parts of that process can be verified automatically, and which still require human judgment?

A thought experiment helped sharpen the discussion: if an agent reported evidence for evolving dark energy after reviewing its own work for several weeks, should physicists believe it? The answer depended less on the agent’s diligence than on the analysis history. An established, preregistered, and blinded pipeline applied to new data might be persuasive, but a new analysis developed alongside the data would not. The danger lies in the many defensible choices and discarded paths through which an analysis can drift toward a preferred result.

Capturing those choices could therefore be essential. Immutable event logs and causal decision graphs might record how each attempt followed from earlier decisions, including the dead ends that shaped the final result. But complete traceability creates its own problem: a record containing millions of events may be technically comprehensive while remaining impossible for a human to inspect. Verification will require ways to compress these traces into meaningful summaries, perhaps through causal structure and visualization.

Agents may also help review one another. Models from different families could provide an initial layer of scrutiny, reducing the risk of shared failures, while human experts could focus on uncertain or consequential cases. Such a system would not eliminate the human bottleneck, and agent self-critique cannot be assumed reliable. Evidence from interpretability research suggests that a model may internally register uncertainty while still producing a confident answer.

The deeper challenge is social. Scientific trust depends not only on evidence but on a tacit contract between authors and readers. Human researchers have limited output, reputations to protect, and reasons to avoid embarrassment; agents have none of these constraints. A vast archive of agent-generated papers could therefore undermine the expectation that a randomly selected paper is likely to be sound, making every result costly to verify.

The discussion ultimately pointed toward verification as a layered system designed to make human oversight more manageable, including inspectable decision traces and automated review techniques. Agents may perform much of the work, but they remain directed and funded by people. Preserving trust in agentic science will therefore require clear human accountability for the questions asked.

Scientific provenance has traditionally meant tracing a result back through its data and code. In agentic science, that chain must extend further: what did an agent do, who authorized it, how did decisions evolve, and who remains accountable when something goes wrong?

One motivation for creating this richer record is to make scientific work more reusable. Negative results and abandoned approaches are routinely lost, leaving other researchers to repeat the same costly experiments. A gap in the literature may suggest that an idea has been tried and failed, rather than that no one has thought of it before. A shared record of failed attempts could reduce duplication, save resources, and provide valuable information by capturing the long tail of what does not work. From a technical standpoint, such a shared record would require multiple elements for discoverability: static analysis could connect outputs to their underlying code and data; runtime logging could capture execution traces; linting could warn when traceability is broken; and signed “content credentials” could package the resulting record.

The harder problem is convincing researchers to share their work, including unfinished results and failed experiments. Anonymization and temporary embargoes may alleviate some concerns. Funding-agency mandates often produce compliance theatre rather than meaningful openness. A more promising approach to building effective incentives may be to ask where meaningful scientific contributions will reside in the future and how researchers can receive credit for them. Indeed, as agents assume more of the scientific workload, authorship may move “up the cognitive stack.” Human contributions will lie less in writing every line of code or prose than in defining questions, steering exploration, recognizing important turns, and validating claims. From this perspective, a paper is merely a lossy summary of a deeper scientific session, whose real value often lies in the failed attempts and consequential decisions that never reach the final publication. Scientists can still receive credit and recognition for the judgment required to guide a project to completion—for knowing what step to take when confronted with negative results, for example. This would require sharing and making transparent the context in which such decisions are made.

On a separate but related dimension of provenance, institutions must be able to trace agent actions back to accountable humans, particularly when those actions consume resources, affect shared data, or violate policy. Technical safeguards such as workload isolation can reduce operational risks, but they cannot fully prevent the illegal use of resources, including violations of export-control regulations. Human responsibility must therefore remain central. Provenance ultimately serves two inseparable roles: preserving the decisions, failures, and discoveries that constitute scientific memory, while ensuring human accountability as agents become active participants in science.