Janas: Run LLMs larger than RAM by streaming experts from SSD

Researchers have developed Janas, an open-source tool that enables running Large Language Models (LLMs) larger than the available RAM by streaming the models from an SSD. This is achieved by breaking down the model into smaller chunks and processing them in a streaming manner, allowing for the processing of models that would otherwise not fit in memory. The tool is designed to support models up to 1TB in size, and the developers claim it can be used to run models on a machine with as little as 16GB of RAM.