When LLM judges agree, should we believe them?

Researchers at Amazon Web Services (AWS) have been exploring the concept of Large Language Models (LLMs) as judges in a simulated court setting. They found that when multiple LLMs agree on a verdict, it significantly increases the confidence in the verdict's accuracy. However, the study also highlights the limitations of relying solely on LLMs as judges, as they can still be misled by biased or incomplete information. The researchers conclude that a combination of human and LLM judges could lead to more accurate and reliable verdicts.