Don’t Trust the Process: When Verifiability Undermines AI Accountability

How do we know if Artificial Intelligence (AI) systems are as performant and responsibly designed as the AI companies claim them to be? In a race-driven innovation climate where responsive development requires time and resources, AI developers and providers may be tempted to misrepresent system performance or overstate their commitment to responsible AI principles. Such circumvention is further enabled by limited access to system components and information by external stakeholders, a restriction commonly justified on the grounds of trade secret protection, privacy and security considerations, among others. In response, a growing community of scholars has been developing cryptographic and statistical solutions that aim to enable robust verification of specific claims under constrained access. However, the construction of these solutions rely on a set of shared, yet unexamined, assumptions required to abstract complex real-world governance challenges into computational representations. In this article, we examine the validity of these assumptions. After detailing the conceptual foundation and analytical lens we used to interrogate these abstraction processes, we show that existing technical approaches to developing verifiable AI commit systematic fallacies that compromise the validity of these approaches. While the existing technical verification processes aim to solve critical AI governance problems, we argue that these fallacies create loopholes that can be exploited by dishonest developers and providers, and therefore lead to misplaced trust in these processes. Finally, we discuss how the field of verifiability could be reoriented towards a more nuanced and interdisciplinary approach to develop rigorous verification processes, both technical and non-technical, that support effective AI governance.