Experiments

Introduces Delta Attention Residuals and Delta Block to improve cross-layer information flow in Qwen3-0.6B while preserving the standard residual stream, evaluated on Hindi FineWeb-2 fine-tuning and downstream benchmarks. All training, validation, routing analysis, and downstream evaluation were conducted on an NVIDIA DGX Spark, with Delta Block achieving the best validation loss/perplexity and the highest overall downstream average.

Adapting mHC for a Small On-Device Indic Translation Model : What Worked and What Didn’t

LLMs · NLP · Benchmark · Jun 2026

Testing whether Manifold-Constrained Hyper-Connections can improve Qwen3-0.6B for small, on-device Indic translation workloads. Roughly five months on a single RTX 4070.