How good are frontier models at physics?

Researchers tested the ability of large language models to solve physics problems and found that they performed well on certain types of problems but struggled with others. The study used a dataset of 1,000 physics problems and tested five different language models, including BERT, RoBERTa, and a transformer model. The models performed well on problems that involved simple calculations and concepts, but struggled with problems that required more complex reasoning and understanding of underlying physics principles. The study found that the models were able to solve 60% of the problems in the dataset, but were often unable to provide a correct explanation for their answers. The researchers suggest that the models are able to perform well on certain types of problems because they have been trained on a large corpus of text data that includes many physics-related concepts, but are limited by their lack of understanding of the underlying physics principles.