Traditional news outlets as carriers and distributors of information have been challenged by online social networks with regards to their gate-keeping function. We believe that only a combined effort of people and machines will be able to curb so-called “fake news” at scale in a decentralized Web. In this position paper, we propose an approach to design social machines that coordinate human- and machine-driven credibility assessment of information on a decentralized Web. To this end, we defined a fact-checking process that draws upon ongoing efforts for tackling disinformation on the Web, and we formalized this process as a multi-agent organisation for curating W3C Web Annotations. We present the current state of our prototypical implementation in the form of a browser plugin that builds on the Hypothesis annotation platform and the JaCaMo multi-agent platform. Our social machines would span across the Web to enable collaboration in form of public discourse, thereby increasing the transparency and accountability of information.