Abstract
Tits has shown that a finitely generated matrix group either contains a nonabelian free group or has a solvable subgroup of finite index. We give a polynomial time algorithm for deciding which of these two conditions holds for a given finitely generated matrix group over an algebraic number field. Noting that many computational problems are undecidable for groups with nonabelian free subgroups, we investigate the complexity of problems relating to matrix groups with solvable subgroups of finite index. For such a groupG, we are able in polynomial time to compute a homomorphismφsuch thatφ(G) is a finite matrix group and the kernel ofφis solvable. If, in addition,Ghas a nilpotent subgroup of finite index, we obtain much stronger results. For such groups, we show how to effectively compute an encoding of elements ofGsuch that the encoding length of an element obtained as a product of length ⩽ℓ over the generators isO(logℓ) times a polynomial in the input length. This result is the best possible; it has been shown by Tits and Wolf that if a finitely generated matrix group does not have a nilpotent subgroup of finite index, then the number of group elements expressible as words of length ℓ over the generators grows ascℓfor some constantc>1 depending onG. For groups with abelian subgroups of finite index, we obtain a Las Vegas algorithm for several basic computational tasks, including membership testing and computing a presentation. This generalizes recent work of Beals and Babai, who give a Las Vegas algorithm for the case of finite groups, as well as recent work of Babai, Beals, Cai, Ivanyos, and Luks, who give a deterministic algorithm for the case of abelian groups.