Conventional scaling laws suggest that sub-billion-parameter models cannot compete with 7B baselines on composite benchmarks such as MMLU and GSM8K. Our latest architecture challenge that assumption — not through brute-force scaling, but through deliberate structural sparsity and compute routing.
Architecture Overview
The model combines state-space layers for long-context sequence modeling with sparse transformer blocks activated via a learned router. Only 34% of attention heads participate in any given forward pass, yielding effective FLOPs comparable to a 400M dense model while retaining 780M total parameters.
Key Results
- MMLU (5-shot): 58.3% vs. 7B baseline 59.1%
- GSM8K: 71.2% vs. 7B baseline 72.8%
- Inference latency (A100, batch=1): 12ms vs. 7B baseline 89ms
- Memory footprint (INT4): 412 MB vs. 7B baseline 3.8 GB
Engineering Implications
For edge deployment and high-throughput API serving, the latency and memory advantages compound. We are integrating this architecture into our production fine-tuning pipeline as the default backbone for domain-specific adapters.
Full reproducibility artifacts — training configs, eval scripts, and weight initialization seeds — are available to partners under our engineering collaboration agreement.