Moe Latency Insights

MoE reduces FLOPs but latency can increase because routing expert dispatch/gather, memory traffic communication, smaller/less efficient GEMM and expert load imbalance reduce hardware utilization. So fewer FLOPs do not necessarily mean lower wall clock latency