Multi-agent systems and credibility-based advanced scoring mechanism in fact-checking

Fact-checking is crucial as rumours and misinformation negatively impact social networking services (SNS) and online discussions, often leading to the spread of misinformation. Meanwhile, fact-checking with large language models (LLMs) is becoming increasingly popular with the increase in the performance of LLMs. However, the previous works have issues, including overconfidence in the judgment results of LLM and the insufficiency of binary fact-checking due to the text’s complexity. On the other hand, using multiple information sources to make judgments reveals another obstacle: the lack of proper scoring mechanisms. Thus, we propose a framework called multi-agent fact-checking (MAFC), which includes multiple agents with unique information sources to measure the text’s credibility. Specifically, a brand-new scoring mechanism is also used to calculate credibility according to each agent’s judgment results and confidence. We tested our proposed method through several comparative experiments. The results of the experiments prove that the proposed method performs better than other baselines in both the binary fact-checking task and the multi-label fact-checking task. Finally, the challenges and obstacles existing in fact-checking fields, such as the definition standards and dataset creation, are discussed.