Realizing subgroups as images of Rota–Baxter operators on groups: the free-group instance (Kourovka Notebook Problem 20.124) and Rota–Baxter operators on free non-abelian groups

A Rota–Baxter operator (of weight 1) on a group $G$ is a map $B:G\to G$ satisfying $B(g)B(h)=B(gB(g)hB(g)^{-1})$. Bardakov and Gubarev, building on the definition introduced by Guo, Lang, and Sheng, asked which subgroups $H\le G$ can occur as $\operatorname{Im}B$ for such an operator, and highlighted the instance: does a non-abelian free group $F$ admit a Rota–Baxter operator whose image is exactly the derived subgroup $F'$? This instance is recorded as Problem 20.124 of the Kourovka Notebook (proposed by V. G. Bardakov) and remains unsolved in the 21st edition (2026). Closely linked open questions from the same paper ask for a description of all Rota–Baxter operators on free non-abelian groups, in particular whether every map prescribed on a free basis extends to such an operator, and whether a concrete induced group $(F_2)_B$ is finitely generated. The follow-up literature of 2021–2026 (skew braces, extension theory, classifications on finite simple groups) shows that the bundled question on surjective operators with nontrivial kernel was resolved affirmatively in 2026, while the image-realization and free-group questions above remain open.

Background

A Rota–Baxter operator of weight 1 on an algebra, introduced by G. Baxter in 1960 in probability theory, is a linear map $R$ with $R(x)R(y)=R(xR(y)+R(x)y+xy)$; such operators formalize integration by parts and have deep connections with combinatorics and the classical Yang–Baxter equation. In 2020, Li Guo, Honglei Lang, and Yunhe Sheng, while studying the integration (geometrization) of Rota–Baxter Lie algebras, transferred the notion to groups: a Rota–Baxter operator (RB-operator) on a group $G$ is a map $B:G\to G$ such that $B(g)B(h)=B\bigl(g\,B(g)\,h\,B(g)^{-1}\bigr)\quad\text{for all }g,h\in G.$ A group with such an operator is an RB-group. Two operators exist on every group: $B_0(g)=e$ and $B_{-1}(g)=g^{-1}$. Bardakov and Gubarev developed the basic theory for arbitrary groups and posed the question list treated below.

Three structural facts frame the problem. First, given an RB-group $(G,B)$, the operation $g\circ h=gB(g)hB(g)^{-1}$ makes the underlying set into a new group $G_B$, and $B:(G,\circ)\to(G,\cdot)$ is a group homomorphism; consequently $\operatorname{Im}B$ is always a subgroup of $G$. Moreover the map $B^+(g)=gB(g)$ is a homomorphism $G_B\to G$, the kernels of $B$ and $B^+$ are normal in $G_B$, and $G=\operatorname{Im}(B^+)\operatorname{Im}(B)$. Second, given an exact factorization $G=HL$ (every $g\in G$ uniquely $g=hl$ with $h\in H$, $l\in L$), the map $B(hl)=l^{-1}$ is an RB-operator; such operators are called splitting, and conversely an RB-operator is splitting iff $B(gB(g))=e$ for all $g$. Third, when $G$ is abelian the RB-identity reduces to $B(g)B(h)=B(gh)$, so RB-operators on abelian groups are exactly endomorphisms.

The RB-groups framework is closely tied to skew left braces — sets with two group operations $(G,\cdot,\circ)$ related by $g\circ(h\cdot k)=(g\circ h)\cdot g^{-1}\cdot(g\circ k)$, introduced by Guarnieri and Vendramin because every skew brace yields a set-theoretic solution of the Yang–Baxter equation. Bardakov and Gubarev showed that every RB-group gives rise to a skew left brace and, conversely, every skew left brace embeds into an RB-group; Caranti and Stefanello gave a cohomological characterization of exactly which skew brace structures on a given group come from an RB-operator, with finite $p$-group counterexamples showing the correspondence is not surjective.

Classifications are presently known only for special finite classes: on all sporadic simple groups every RB-operator is splitting (so images are exact-factorization factors); on alternating groups $A_n$ non-splitting operators exist for infinitely many $n$ and are classified; on the finite simple exceptional groups of Lie type only the elementary operators exist, and on $PSL_2(q)$ only splitting ones; on compact simple Lie groups only the two elementary operators exist. No analogous description exists for any infinite non-abelian group beyond constructions from free products: if $G=K*L$ is a free product with $L$ abelian, then $B(g)=\chi(\pi(g))$ (projection onto $L$ followed by an endomorphism $\chi$ of $L$) is an RB-operator — on a free group $F=\langle x_1,\dots,x_r\rangle$ this yields operators with cyclic images.

Within this landscape, Bardakov and Gubarev posed (Question 5.10): given a group $G$ and a proper subgroup $H\le G$, under which conditions does there exist an RB-operator $B$ with $B(G)=H$? In particular: if $F$ is a non-abelian free group with derived (commutator) subgroup $F'=[F,F]$ — the normal subgroup generated by all commutators $[g,h]=ghg^{-1}h^{-1}$, with $F/F'$ the free abelianization — is there an RB-operator on $F$ with image $F'$? This instance is now Kourovka Notebook Problem 20.124. It is a genuine non-splitting question: a splitting operator with image $F'$ would require an exact factorization $F=H\cdot F'$, which would force $H\cong F/F'\cong\mathbb{Z}^r$ with $r\ge2$; but $\mathbb{Z}^r$ is not free for $r\ge2$ and, by the Nielsen–Schreier theorem, every subgroup of a free group is free. So realizing $F'$ demands an operator of the kind for which no construction machinery currently exists on torsion-free groups. Two adjacent questions about free groups accompany it in the same paper: whether every map prescribed on a basis of a free group extends to an RB-operator (Question 7.10; the paper gives a general extension criterion and shows that a specific basis map on $S_3$ does not extend, while for $F_2=\langle a,b\rangle$ the map $\beta(a)=a$, $\beta(b)=e$ does extend, via the projection operator of Example 7.11), and whether the induced group $(F_2)_B$ of that example — the set $F_2$ with the operation $g\circ h=gB(g)hB(g)^{-1}$ — is not finitely generated (Question 7.12; the paper shows the $\circ$-closure of $\{a,b\}$ is a proper subset of $F_2$).

Problem Statement

Determine which subgroups of a group arise as images of Rota–Baxter operators, taking free non-abelian groups as the decisive test case. Concretely, with an RB-operator on a group $G$ defined as a map $B:G\to G$ satisfying $B(g)B(h)=B(gB(g)hB(g)^{-1})$ for all $g,h\in G$:

1. (Realization criterion; Question 5.10 of Bardakov–Gubarev.) Given a group $G$ and a proper subgroup $H\le G$, give necessary and sufficient conditions for the existence of an RB-operator $B$ on $G$ with $\operatorname{Im}B=H$. The criterion must at least reproduce the known instances: images of splitting operators (exact-factorization factors) and endomorphism images on abelian groups.
2. (Kourovka Notebook Problem 20.124.) Let $F$ be a non-abelian free group and $F'=[F,F]$ its derived subgroup. Does there exist an RB-operator $B$ on $F$ with $\operatorname{Im}B=F'$?
3. (Question 7.10.) Describe the Rota–Baxter operators on free non-abelian groups. In particular, is it true that any map $\beta$ defined on a basis of a free group extends to an RB-operator on this group? A negative answer requires an explicit basis map together with a proof that no RB-operator extends it; a positive answer requires a construction producing, for every prescribed basis map, an extending operator. A description counts as complete when it determines, for an arbitrary RB-operator on a free non-abelian group, which elements lie in its image, and decides, for a prescribed map on a free basis, whether that map extends to an operator on the group; settling only the yes/no basis-extension sub-question above does not by itself provide such a description.
4. (Question 7.12.) In the concrete situation of Example 7.11 of Bardakov–Gubarev — $F_2=\langle a,b\rangle$ with the RB-operator $B$ extending $\beta(a)=a$, $\beta(b)=e$, i.e. the projection homomorphism $B:F_2\to\langle a\rangle$ with $B(a)=a$, $B(b)=e$ — decide whether the induced group $(F_2)_B$ — the set $F_2$ under the operation $g\circ h=gB(g)hB(g)^{-1}$ — is finitely generated.

The four items are linked by the single objective of understanding Rota–Baxter operators and their images on free non-abelian groups: items 2–4 are its concrete open instances on free non-abelian groups, and item 1 is the general frame they test. A criterion from item 1, or a description from item 3, that decides whether $\operatorname{Im}B$ is the derived subgroup settles item 2 as well; conversely, answering item 2 alone leaves the description sought in item 3 and the finite-generation decision of item 4 open. Results restricted to a fixed finite-rank free group $F_r$ are restrictions of the unrestricted problem, which concerns arbitrary non-abelian free groups.

References

1. Valeriy G. Bardakov and Vsevolod Gubarev, Rota–Baxter operators on groups, Proc. Indian Acad. Sci. Math. Sci. 133 (2023), No. 1, Art. 4, DOI 10.1007/s12044-023-00723-9, arXiv:2103.01848, https://arxiv.org/abs/2103.01848
2. Li Guo, Honglei Lang, and Yunhe Sheng, Integration and geometrization of Rota–Baxter Lie algebras, Adv. Math. 387 (2021), 107834, DOI 10.1016/j.aim.2021.107834, arXiv:2009.03492, https://arxiv.org/abs/2009.03492
3. Valeriy G. Bardakov and Vsevolod Gubarev, Rota–Baxter groups, skew left braces, and the Yang–Baxter equation, J. Algebra 596 (2022), 328–351, DOI 10.1016/j.jalgebra.2021.12.036, arXiv:2105.00428, https://arxiv.org/abs/2105.00428
4. A. Caranti and L. Stefanello, Skew braces from Rota–Baxter operators: a cohomological characterisation and some examples, Ann. Mat. Pura Appl. (4) 202 (2023), 1–13, DOI 10.1007/s10231-022-01230-w, arXiv:2201.03936, https://arxiv.org/abs/2201.03936
5. Apurba Das and Nishant Rathee, Extensions and automorphisms of Rota–Baxter groups, J. Algebra 636 (2023), 626–665, DOI 10.1016/j.jalgebra.2023.09.006, arXiv:2212.06429, https://arxiv.org/abs/2212.06429
6. Nishant Rathee and Mahender Singh, Relative Rota–Baxter groups and skew left braces, Forum Math. 37 (2025), 919–935, DOI 10.1515/forum-2024-0020, arXiv:2305.00922, https://arxiv.org/abs/2305.00922
7. L. Guarnieri and L. Vendramin, Skew braces and the Yang–Baxter equation, Math. Comp. 86 (2017), 2519–2534, DOI 10.1090/mcom/3161, https://doi.org/10.1090/mcom/3161
8. E. I. Khukhro and V. D. Mazurov (eds.), Unsolved Problems in Group Theory. The Kourovka Notebook, No. 21 (21st edition, 2026), arXiv:1401.0300, https://arxiv.org/abs/1401.0300
9. Wouter van Doorn, Elias Judin, Pietro Monticone, and Daniel Morrison, On Some Problems from the Kourovka Notebook, arXiv:2607.17477, https://arxiv.org/abs/2607.17477
10. Pietro Monticone, Kourovka Notebook Problem 20.125, solution note, Kourovka Notebook (23 March 2026), https://kourovkanotebookorg.wordpress.com/wp-content/uploads/2026/03/20_125-1.pdf
11. Alexey Galt and Vsevolod Gubarev, Rota–Baxter operators on dihedral and alternating groups, arXiv:2404.14078, https://arxiv.org/abs/2404.14078
12. Alexey Galt and Vsevolod Gubarev, On Rota–Baxter operators on finite simple groups of Lie type, arXiv:2509.04961, https://arxiv.org/abs/2509.04961
13. Saveliy V. Skresanov, Rota–Baxter operators on compact simple Lie groups and algebras, arXiv:2506.14324, https://arxiv.org/abs/2506.14324
14. Xing Gao, Li Guo, Yanjun Liu, and Zhi-Cheng Zhu, Operated groups, differential groups and Rota-Baxter groups with an emphasis on the free objects, Comm. Algebra 51 (2023), 4481–4500, DOI 10.1080/00927872.2023.2212775, arXiv:2209.04922, https://arxiv.org/abs/2209.04922