EdgeXpert: An Edge Device for Memory-Efficient LLM Inference with Mixture-of-Experts and Speculative Decoding

The paper addresses the bottleneck of external memory access (EMA) in feed-forward network (FFN) layers during on-device LLM inference. Speculative decoding and mixture-of-experts (MoE) are both promising but incompatible when combined. EdgeXpert introduces two key mechanisms: in the prefill stage, prompt-wise expert reuse reformulates routing as prompt-level expert reuse rather than per-token selection, identifying important tokens via a lightweight encoder and constructing a shared expert set to reduce expert EMA for less important tokens. In the decode stage, depth-aware expert coalescing exploits contextual similarity and mutual exclusivity of same-depth candidate tokens, loading only salient channels and using computational calibration to recover accuracy without extra memory access. Synthesized in Samsung 28nm technology at 800 MHz, EdgeXpert achieves up to 56.3% latency reduction and 44.1% energy reduction compared to prior works while maintaining near-baseline accuracy.