Adapting mHC for a Small, On-Device Indic Translation Model: What Worked and What Didn’t

Vaibhav and I spent about 5 months running this project on an RTX 4070 with 12GB VRAM, trying to understand whether mHC could materially improve small-model performance in a realistic local setup.

To test this, we focused our target model (Qwen3-0.6B) on practical applications, asking : Can Manifold-Constrained Hyper-Connections (mHC) improve the performance of a small model enough to make it more useful for local Indic translation workloads ?

This project began based on a fairly straightforward question :

While our specifications for the target model (Qwen3-0.6B) were primarily driven by practical applications, we also sought to ensure the model would be able to run on device, have a low hardware impact, and still exceed performance expectations when working with Indian-language datasets. The main approach we examine here is whether an architectural idea from the large-model era — mHC — can produce similar benefits in a much smaller design under tighter hardware constraints.

Figure: The architecture sketch of the model (CC : Vaibhav)

Why we picked Qwen3-0.6B ?

One of the reasons Qwen3-0.6B was interesting is that it is a relatively deep model for its size : it has 28 decoder layers despite having only around 0.6B parameters.

That depth stood out to us.

A lot of the motivation behind mHC comes from the idea that instability can emerge from the compound residual mapping across layers. In other words, even if each layer is individually well-behaved, the repeated composition of residual transformations across many layers can create optimization and stability issues. mHC is designed to constrain that behavior.

So our working intuition was :

• Qwen3-0.6B is small enough to be attractive for on-device use
• but still deep enough that residual-path dynamics might matter
• which makes it a good candidate to test whether mHC can help at small scale

This made the project a nice meeting point between theory and deployment constraints.

Figure: The actual setup we used for training and evaluation

The goal

Our end goal was not just to reproduce a paper result.

We wanted to explore whether mHC could help us build a better small language model for Indic translation, ideally one that would still be practical for local/mobile deployment.

Concretely, we were aiming for:

• better translation quality on Indic-focused data
• improved stability or representational flexibility over the base model
• minimizing the resource requirements associated with using a standard SLM
• developing ways to infer locally on constrained devices

We also used broader evaluation signals like MILU as a sanity check, while the main practical focus remained Indic language use cases.

What mHC changes in the model ?

At a high level, mHC replaces the standard single residual stream with multiple parallel residual streams and learns how information should move between them.

In this repository, the main training path on the current branch uses mHC V2, which is implemented in a more hyper-connections-style form. For intuition, though, it is still useful to describe the mechanism using the familiar $H^{\text{pre}}_l$ / $H^{\text{post}}_l$ / $H^{\text{res}}_l$ view, because that captures the main stream-mixing idea.

In practice, we expanded the model into 4 parallel streams. Instead of a single hidden stream moving through attention and MLP blocks, the model carries multiple streams in parallel and learns three types of mappings:

• $H^{\text{pre}}_l$: squeezes the multiple streams into one stream before the sublayer
• $H^{\text{post}}_l$: broadcasts the sublayer output back into multiple streams
• $H^{\text{res}}_l$: mixes information between streams along the residual path

The critical part is $H^{\text{res}}_l$. In the mHC formulation, this residual mixing matrix is constrained to be doubly stochastic, which helps prevent unstable growth or collapse across many composed residual mappings.

That constraint is enforced using Sinkhorn-Knopp projection, which was a central piece of the implementation.

How we implemented it ?

We adapted Qwen3-0.6B into an mHC version by wrapping both major sublayers in every decoder block:

• the self-attention sub-layer
• the MLP/FFN sublayer

Since Qwen3 has 28 decoder blocks, this effectively gave us 56 mHC-wrapped sublayers in total.

Main architectural decisions

• 4 residual streams
• stream expansion after token embeddings
• stream collapse before the final norm and LM head
• separate mHC wrappers for attention and FFN
• static + dynamic mixing coefficients
• RMSNorm-based coefficient computation
• Sinkhorn projection with 20 iterations for residual mixing

The model starts from the original Qwen weights and is converted in a way that is designed to preserve initial behavior as closely as possible. That was important, because we didn’t want the model to begin training from a completely different function. In the V2 path, this is best thought of as a warm start that stays close to the base model, rather than a guarantee of exact output equality.

Warm-start / equivalence initialization

To make the converted model begin close to the base model at initialization:

• $H^{\text{pre}}_l$ starts as an equal weighting across streams
• $H^{\text{post}}_l$ starts as an equal broadcast
• $H^{\text{res}}_l$ starts close to the identity mapping
• dynamic projections are initially gated down with small learned scalars

This lets the model begin as something very close to the original network, and then gradually learn whether multi-stream routing is useful. For the repository's default V2 path, the practical goal is bounded initial divergence with a stable warm start, not strict mathematical identity at every logit.

Stabilizing the residual mixing

A big part of the project was translating the paper idea into something trainable in practice.

The main problem mHC is trying to address is that repeated residual mappings across depth can become unstable when composed many times. The paper’s key insight is that if the residual mixing matrices are constrained to lie on a well-behaved manifold — specifically the set of doubly stochastic matrices — then their composition remains much more controlled.

In the paper's formulation, this is written as a projection of the residual mixing matrix onto the doubly stochastic manifold:

In practice, we parameterize these matrices in an unconstrained space and project them into valid forms:

In our implementation, we used Sinkhorn-style projection to enforce that structure on $H^{\text{res}}_l$.

We also kept a small initial gating value for the dynamic component, so early training would stay close to the base model instead of immediately introducing aggressive stream mixing.

Additional practical details

Some of the implementation and training ideas we followed included :

• 20 Sinkhorn iterations
• small initial gating values
• separate static and dynamic mixing terms
• residual mix temperature kept simple
• final stream collapse before final RMSNorm and LM head
• RMSNorm with epsilon 1e-6 for compatibility with the base architecture

We also considered the stability implications of projection in different formulations. Conceptually, log-domain Sinkhorn is relevant when numerical stability becomes a concern, but the core implemented path used a standard Sinkhorn-Knopp projection workflow for the residual mixer.

Training plan

The training recipe was built around the idea that the mHC parameters should first remain conservative, then gradually be allowed to adapt more freely.

The intuition was:

1. start near the base model’s behavior
2. avoid early instability from the new routing parameters
3. then let the model learn whether stream mixing is actually helpful

A key part of this was the idea of an initial freeze phase, followed by unfreezing, so the new structure would not immediately destabilize the original pretrained behavior.

The hardware mattered here too. Working on an RTX 4070 with 12GB VRAM meant keeping the setup practical: modest batch sizes, gradient accumulation, and a training recipe that stayed within a realistic local budget rather than assuming multi-GPU infrastructure.

Figure: The training pipeline

We also monitored:

• training loss
• gradient norms
• how far $H^{\text{res}}_l$ moved away from its initialization behavior
• how much the learned gates moved away from their initialization

Figure: Training loss stayed noisy throughout the long run instead of settling into a clean convergence curve.

Figure: Gradient norms were mostly controlled, but still showed intermittent spikes, including a more noticeable late-stage disturbance.

What we observed ?

This is where the project became especially interesting.

1. We did not get strong improvement over the base model

Our original hope was that mHC would improve the small model’s ability to represent difficult multilingual behavior, especially for Indic translation settings. In practice, the gains over the base model were small or negligible.

That doesn’t make the experiment a failure — but it does make it informative.

It suggests that simply transplanting an architecture that helps at larger scales does not automatically produce benefits at smaller scales.

2. Internally, the model often remained conservative

One of the strongest signals came from the internal analysis.

At a high level:

• the monitored residual-mixing statistics stayed remarkably flat over training
• the routing behavior did not appear to evolve much across the long run
• earlier parameter analysis also suggested that the learned coefficients stayed fairly conservative

That means the model often behaved as if it never really committed to using the extra routing flexibility that mHC introduced.

In other words, the architecture was present, but the training dynamics did not strongly move into that larger hypothesis space.

Figure: A parameter-level summary view of the learned mHC coefficients. The overall picture is still conservative rather than strongly specialized.

Figure: The residual-mixing identity-distance statistic stays almost flat across the long run, which suggests the mixer is not evolving dramatically.

Figure: The residual-mixing entropy metric is also nearly flat, reinforcing the same story: the stream-routing behavior changes much less than we initially hoped.

3. Training was noisy across a long run

The training diagnostics showed noisiness not just at the beginning, but across the broader training window:

• repeated loss spikes
• intermittent gradient norm spikes
• a later-stage regime shift where loss rises rather than clearly improving
• limited evidence that the extra routing parameters were being used in a richer way over time

This matters because mHC is supposed to help with stability through constrained residual mixing. If the model is not actually learning useful mixing patterns — or if the optimization budget is too small to get there — then the extra machinery may add complexity without enough payoff.

4. Composite residual mappings did not become meaningfully richer

Another useful analysis was looking at single-layer and composite mapping behavior across depth. The core story there was also conservative: we did not see strong evidence that the model learned a substantially richer residual transport pattern across layers. The composite transport remained very bounded and close to a conservative regime across almost the entire stack.

Figure: The composite residual transport stays extremely close to a bounded, almost unchanged regime across depth instead of developing a strongly differentiated structure.

Our current hypothesis

Our current interpretation — and we'd frame this as a hypothesis, not a hard conclusion — is that mHC may simply not be as useful for a model this small.

More specifically, one or more of these may be true:

• mHC benefits may emerge more clearly at larger parameter scales
• in smaller models, the extra routing capacity may not be the bottleneck
• the optimization cost of learning useful stream interactions may be too high relative to the model size
• for an SLM, the base model may already be using its limited capacity efficiently enough that mHC adds overhead more than benefit
• the training budget or data regime may have been insufficient for the new structure to “turn on”

So the statement we’d make publicly is not:

“mHC does not work for small models.”

It is closer to :

“In our Qwen3-0.6B setting, we did not observe meaningful improvement over the base model, and one plausible explanation is that mHC’s advantages may be more scale-dependent than we initially expected.”

That is a much safer and more honest claim.

Benchmarks and evaluation

For evaluation, we focused on tasks that mattered to the intended use case.

For training data, we worked with multilingual corpora such as Sangraha and also used tooling around datasets like IndicCorpV2.

For evaluation, the repository's benchmark tooling is centered around Indic tasks such as:

• IndicNLG
• IndicQA
• IndicGLUE
• broader capability sanity checks such as MILU

The practical question was not only whether the model became more expressive, but whether that translated into better downstream behavior for Indic language workloads.

At least in our current runs, the answer was: not enough to justify the added architectural complexity.

Figure: Qualitative translation examples — base vs mHC outputs

Figure: Performance comparison

Figure: Latency evaluation

Figure: Accuracy distribution

Why this still matters ?

Even though we did not get the improvement we wanted, we still think the project was valuable for three reasons.

1. It tested a real research transfer question

A lot of techniques look promising in papers, especially when reported at large scale. But for practical deployment, the real question is whether they survive transfer into :

• smaller models
• tighter memory budgets
• mobile or local-device constraints
• multilingual settings with specific target use cases

That is exactly the kind of question worth testing.

2. It gave us a reusable implementation base

We now have a working mHC-style adaptation stack for Qwen3-like models, including :

• model conversion
• equivalence validation
• training utilities
• diagnostics for stream mixing
• plotting and internal analysis

That makes future experiments much easier.

3. Negative results are still useful

A clean negative or weak-positive result is still a result.

In this case, it tells us that if the goal is better on-device Indic translation, then simply adding mHC to a 0.6B model may not be the best next step.

That helps narrow future work.

What we would try next :

If we continue this line of work, we'd explore a few directions:

• test the same idea on a somewhat larger base model
• use a more translation-specific fine-tuning setup
• compare against cheaper alternatives such as LoRA or lightweight adapters
• let the model train longer and verify whether stream specialization eventually emerges
• ablate the number of streams and projection strength
• test whether the gains are more visible on reasoning-heavy tasks than on translation

Another useful direction would be to compare architectural gains per added parameter. For an on-device system, raw quality matters, but so does efficiency per watt, per MB, and per millisecond.

Final takeaway

This project was an attempt to bring a large-model architectural idea — Manifold-Constrained Hyper-Connections — into a small, practical, on-device multilingual setting.

We implemented mHC on top of Qwen3-0.6B, targeted Indic language use cases, and evaluated whether the added residual-stream flexibility could improve a small model enough to matter in a practical local setup on RTX 4070-class hardware with 12GB VRAM.

The main takeaway is straightforward :

We were able to implement mHC successfully, but we did not see substantial improvement over the base model in this small-model setting.

Our current hypothesis is that mHC may be more effective at larger scales, where the model has enough capacity to benefit from richer inter-stream routing. But that remains a hypothesis, not a conclusion.

For now, the project stands as a useful engineering and research exploration: a grounded attempt to answer whether a promising architecture for very large models can also help small models intended for local Indic translation.

At least in this case, the answer seems to be: not yet, or not in this form.

Paper Link : https://arxiv.org/pdf/2512.24880

You can find the code for this at :

[Repository Needs to be cleaned up a little bit, sorry for the mess, do ignore AGENTS.md]

Authors :

• Vaibhav Verma
• Ch-Kumar-Kartik