Context Language Models

The paper discusses the context language models, specifically focusing on the limitations of current models in handling context and the potential benefits of using contextualized models. The authors propose a new model, called Context Language Model (CLM), which is based on the Transformer architecture and incorporates contextualized word representations. The CLM is trained on a large corpus of text and achieves state-of-the-art results on several benchmark tasks, including question answering and text classification. The authors also propose several variants of the CLM, including a smaller version called CLM-Base and a larger version called CLM-Large. The results show that the CLM outperforms other contextualized models, such as BERT and RoBERTa, on several tasks.