New GUM Method Cuts Memory Costs in Large Language Model Training

Researchers have unveiled a new optimisation method called Gradient Unbiased Meta-Update (GUM) to tackle the high memory demands of training large language models (LLMs). The approach aims to cut resource costs without sacrificing accuracy, offering a more efficient alternative to existing techniques like GaLore and Muon.

GUM builds on the GaLore mechanism and the Muon algorithm but introduces an unbiased low-rank optimisation strategy. Unlike GaLore, which struggles with convergence in noisy environments, GUM selectively updates a subset of model parameters each iteration. This method ensures a more representative and unbiased estimate of the gradient, improving both performance and memory efficiency.

Testing showed GUM matched the accuracy of the full-parameter Muon baseline in a synthetic noisy linear regression task, while GaLore failed to converge. Further experiments in fine-tuning LLMs revealed GUM achieved higher accuracy on instruction-following and mathematical reasoning tasks compared to GaLore. The method also maintains a comparable or smaller memory footprint than GaLore, making it more scalable for large-scale training. The team has released the code publicly, allowing other researchers to replicate the results and explore further applications. GUM's novel sampling strategy minimises bias during updates, leading to faster and more reliable training cycles.

GUM provides a way to train large language models with lower memory requirements while keeping accuracy intact. The open-source release of the code enables wider adoption and testing across different models and datasets. Early results suggest it could become a practical solution for reducing computational costs in LLM development.