Generalization, tip-change robustness, and physical interpretation of deep-reinforcement-learning control in autonomous atom manipulation

Deep reinforcement learning (DRL) was shown in 2022 (Chen et al., Nat. Commun. 13, 7499) to drive real scanning-tunneling-microscope lateral manipulation of Ag adatoms on Ag(111) to atomic precision and to autonomously assemble a 42-atom artificial kagome lattice, but the demonstration left open how such agents generalize across substrate/adsorbate systems, survive spontaneous tip-apex changes, and connect to microscopic physics. Case-by-case transfers now exist — RL-driven CO manipulation on Cu(111) with automated assembly of an artificial graphene lattice (2025), and an integrated AI framework manipulating Ag on Si(111)-(7x7) at room temperature for 25 hours (2025) — yet both still identify tip instability and human tip conditioning as limiting, no systematic account of transferability or retraining budgets exists, and no work derives the learned policy's regularities from a microscopic tip–adatom interaction model. The problem — systematic generalization, sample-efficient robustness to tip changes, and physical interpretability of DRL atom manipulation, with quantified assembly efficiency — remains open.

Background

A scanning tunneling microscope (STM) images and modifies conductive surfaces using an atomically sharp metal tip at a fixed tunneling current and bias voltage; the same tip doubles as a nanoscale tool. In lateral atom manipulation, the tip is brought close enough to an adsorbed atom (adatom) that tip–adatom forces drag the atom across the surface to a chosen site without picking it up. This is the workhorse technique for building artificial lattices — adatom arrangements whose confined surface-state electrons realize designer quantum states — and for prototypes of atom-scale circuitry. On the close-packed Ag(111) surface, Ag adatoms occupy two nearly degenerate families of hollow adsorption sites (fcc and hcp, distinguished by the stacking of the underlying atomic layers), so the best achievable placement error for a randomly chosen target is bounded by the site spacing $a/\sqrt{3}\approx 0.166$ nm, where $a=0.288$ nm is the surface lattice constant; placing an atom within this bound is the field's working definition of atomic precision. The physical difficulty is that the correct combination of bias $V$, conductance setpoint $G$ (which fixes the tip height), and tip start/end coordinates is not known a priori, differs for every tip-apex/adsorbate/substrate combination, and changes spontaneously when the tip apex picks up or loses atoms; wrong parameters cause failed moves, tip crashes, or unintended rearrangement of neighboring atoms. Before 2022, autonomous reinforcement learning (RL) — where an agent learns a control policy by trial and error from rewards rather than from a model — had been applied to scanning-probe automation only with discrete action sets or in simulation, e.g. hydrogen-desorption nanofabrication on Si(001) and simulated single-molecule manipulation. Chen et al. (Nat. Commun. 13, 7499 (2022)) trained a deep RL agent, using soft actor-critic (SAC, an off-policy deep RL algorithm learning a stochastic policy and a value function jointly) augmented with hindsight experience replay (a technique that relabels each finished episode as a successful attempt to reach the state the atom actually reached, so that failed attempts still yield training signal) and emphasizing-recent-experience replay (a variant of the standard replay buffer that samples the most recent transitions more heavily) for sample efficiency, to laterally manipulate Ag adatoms on Ag(111) at about 5 K through a continuous six-dimensional action: bias, conductance setpoint, and the tip start/end coordinates. Trained purely on real STM data (about 2000 episodes), the agent reached optimal precision, and combined with the Hungarian algorithm (an optimal assignment routine pairing adatoms to target sites) and a rapidly-exploring random tree (RRT) planner (a sampling-based any-angle path planner that avoids other adatoms but carries no optimality guarantee), it autonomously built a 42-atom artificial kagome lattice, a lattice of corner-sharing triangles known to host flat bands and Dirac-like surface-state electrons. A convolutional neural network (CNN) classifier of tunneling-current traces, with about 80% test accuracy, decided after each manipulation whether the atom had moved, so that STM scans — the most time-consuming step of training — could be skipped when no movement was detected; in the reported training runs scans were still taken after about 90% of manipulations (skipped after only about 10%); applicability to Co adatoms on Ag(111) in two parameter regimes was demonstrated in the Methods. The paper's own reports and its stated expectation of broader applicability (Introduction and Discussion) leave unresolved: transferability of the training procedure to a broader variety of substrate/adsorbate combinations and the retraining budgets this requires; the possible suboptimality of the RRT planner and the unquantified efficiency gains of better planners; the reliability of movement detection and the uncharacterized effect of scan reduction on training bias and ultimate precision; the slow adaptation (continued training) needed after spontaneous tip-apex changes; and the absence of any microscopic explanation of the learned policy, whose observed regularities — a preference for higher bias and conductance, and a tip endpoint overshooting the target along an approximately linear law ($x_{\mathrm{end}}\approx 1.02\,x_{\mathrm{target}}+0.08$) — remain empirical.

Problem Statement

Deep reinforcement learning has achieved atomic-precision lateral manipulation of Ag adatoms on Ag(111) and of Co adatoms on Ag(111) in two parameter regimes. Establish when this capability generalizes and what it means physically. (1) Transferability: determine, systematically across substrate/adsorbate combinations whose adsorption-site energetics and manipulation mechanisms differ qualitatively (not merely case-by-case), when a DRL manipulation agent trained on one system can be reused, fine-tuned, or must be retrained, and quantify the training data (episodes/manipulations) required per new system or manipulation regime to reach precision comparable to the original demonstration. (2) Tip-change robustness: maintain atomic-precision manipulation through spontaneous tip-apex changes during extended autonomous multi-atom assembly so that human intervention and long retraining are not required mid-assembly. (3) Policy-physics connection: explain the learned policy's empirical regularities (the preference for higher bias and conductance, the approximately linear tip-endpoint overshoot, the learned action-variance structure) from a microscopic model of the tip–adatom interaction, so that policy structure on a new system can be at least partially predicted or constrained by physics. (4) Assembly efficiency, as an auxiliary component of the same objective: quantify how total assembly time and manipulation count change with path planning better than the RRT baseline, and characterize how movement-detection reliability and reduced scanning affect training bias and ultimate placement precision.

References

1. I-Ju Chen, Markus Aapro, Abraham Kipnis, Alexander Ilin, Peter Liljeroth, Adam S. Foster. Precise atom manipulation through deep reinforcement learning. Nature Communications 13, 7499 (2022). DOI: 10.1038/s41467-022-35149-w. https://doi.org/10.1038/s41467-022-35149-w (preprint arXiv:2203.06975)
2. Ganesh Narasimha, Mykola Telychko, Wooin Yang, Arthur P. Baddorf, P. Ganesh, An-Ping Li, Rama Vasudevan. Automated Construction of Artificial Lattice Structures with Designer Electronic States. ACS Nano 20, 3393-3401 (2025). DOI: 10.1021/acsnano.5c12469. https://doi.org/10.1021/acsnano.5c12469 (preprint arXiv:2508.02581)
3. Junya Okuyama, Zhuo Diao, Hayato Yamashita, Masayuki Abe. Integrated AI Framework for Room-Temperature Atom Manipulation in Scanning Probe Microscopy. Nano Letters 25, 17771-17777 (2025). DOI: 10.1021/acs.nanolett.5c04982. https://doi.org/10.1021/acs.nanolett.5c04982
4. Ganesh Narasimha, Ralph Bulanadi, Jawad Chowdhury, Rama Vasudevan, Yongtao Liu. Self-Driving Scanning Probe Microscopy: From Acceleration to Discovery and Manipulation. Accounts of Chemical Research 59, 2635-2646 (2026). DOI: 10.1021/acs.accounts.6c00273. https://doi.org/10.1021/acs.accounts.6c00273
5. Tim Dierker, Paul Laubrock, Philipp Rahe. Systematically Navigating Through the Parameter Space During the Lateral Manipulation of PTCDA on the Ag(111) Surface. Small Methods 9, 2500177 (2025). DOI: 10.1002/smtd.202500177. https://doi.org/10.1002/smtd.202500177
6. Bernhard Ramsauer, Grant J. Simpson, Johannes J. Cartus, Andreas Jeindl, Victor García-López, James M. Tour, Leonhard Grill, Oliver T. Hofmann. Autonomous Single-Molecule Manipulation Based on Reinforcement Learning. The Journal of Physical Chemistry A 127, 2041-2050 (2023). DOI: 10.1021/acs.jpca.2c08696. https://doi.org/10.1021/acs.jpca.2c08696
7. Bernhard Ramsauer, Johannes J. Cartus, Oliver T. Hofmann. MAM-STM: A software for autonomous control of single moieties towards specific surface positions. Computer Physics Communications 303, 109264 (2024). DOI: 10.1016/j.cpc.2024.109264. https://doi.org/10.1016/j.cpc.2024.109264
8. Nian Wu, Markus Aapro, Joakim S. Jestilä, Robert Drost, Miguel Martínez García, Tomás Torres, Feifei Xiang, Nan Cao, Zhijie He, Giovanni Bottari, Peter Liljeroth, Adam S. Foster. Precise Large-Scale Chemical Transformations on Surfaces: Deep Learning Meets Scanning Probe Microscopy with Interpretability. Journal of the American Chemical Society 147, 1240-1250 (2024). DOI: 10.1021/jacs.4c14757. https://doi.org/10.1021/jacs.4c14757
9. Philipp Leinen, Malte Esders, Kristof T. Schütt, Christian Wagner, Klaus-Robert Müller, F. Stefan Tautz. Autonomous robotic nanofabrication with reinforcement learning. Science Advances 6, eabb6987 (2020). DOI: 10.1126/sciadv.abb6987. https://doi.org/10.1126/sciadv.abb6987
10. Zhuo Diao, Kouma Matsumoto, Linfeng Hou, Masahiro Ohara, Hayato Yamashita, Masayuki Abe. Integrating Domain-Specialized Language Models with AI Measurement Tools for Deterministic Atomic-Resolution Experimentation. arXiv:2602.20669 (2026). https://arxiv.org/abs/2602.20669