Dust: Pretraining Transformers Without Backpropagation

Researchers at QLabs have developed a pretraining method for transformers called Dust, which allows for efficient pretraining without the need for backpropagation. Dust uses a combination of random noise and a novel masking strategy to train the model, resulting in state-of-the-art performance on several benchmarks. The method is particularly useful for large-scale pretraining, as it reduces computational resources and training time. The QLabs team has open-sourced the code and a paper describing the method, which has sparked interest in the research community.