Multi-Head mHC: No Gain at Equal Parameters
Marcos Hernanz
Preprint, 16 October 2026
Abstract
Manifold-constrained hyper-connections (mHC) widen the residual stream into several copies that each layer reads from, writes to and mixes, with coefficients shared by all channels. Multi-head mHC gives each group of channels its own read, write and doubly stochastic mix while keeping the guarantees of mHC. In 629 training runs at 27M and 112M parameters, we tune the learning rate of every main variant and compare against parameter-matched controls. Multi-head mHC gives no gain at equal parameters. With the parameter count of mHC it is no better than mHC, and when it adds parameters it is no better than a wider feed-forward layer. At 27M this holds with QK-norm and, at one learning rate, with four times the tokens. At the best learning rate of mHC it also holds for each tested variant that makes only some of the read, write and mix per group. At a shared learning rate tuned for mHC in a pilot, heads that add parameters beat mHC, but a wider feed-forward layer with the same parameters gains as much. Without QK-norm the heads degrade more gently at too high a learning rate, where at 27M every variant fails through attention-logit growth. QK-norm removes this growth and with it the advantage of the heads. Apart from a smoother gradient norm that does not show in the loss, we find no evidence that multi-head mHC would make larger models more stable.