DisTrO, introduced by Nous Research, is an architecture‑agnostic distributed optimizer that reduces inter‑GPU communication by 1000x‑10,000x compared to prior methods.
I am eager to investigate the algorithms behind DisTrO, its benchmark setups, and any trade‑offs in latency or memory.