The Illusion of Oversight - GenAI Decision Security and Moral Crumple Zone

About this event

"There's a human in the loop" has become the default reassurance for putting GenAI into consequential decisions - SOC triage, AML/KYC scoring, HR screening, customer support, legal. This talk argues that, as commonly implemented, that human is oversight in appearance only, and shows why the problem is neither purely technical nor purely human. We open with three real-world failures spanning three countries, three decades, and three technologies — from a SQL script to an LLM: Australia's Robodebt (500,000+ wrongful debts), the Dutch childcare-benefits scandal (\~26,000 families wrongly accused), and Moffatt v. Air Canada (2024), the first ruling holding a company liable for its chatbot's hallucination. In every case a human was meant to catch the machine's error. In none did it work. The trap is a double bind. Remove the human and accountability evaporates - the responsibility gap, the accountability sink, the moral crumple zone: the harm is real, but no one is answerable. Add the human (HITL, AI Act Art. 14) and cognitive science says it still won't hold: half a century of evidence on automation bias, vigilance decrement, and Bainbridge's ironies of automation - plus fresh 2025 findings on GenAI-driven operator deskilling - shows the human becomes an interface for blame, not a source of control. The result is accountability theater. Ultimately, this talk reframes human oversight from a compliance checkbox into a security property — something you design, measure, and can get wrong. The question every organization deploying GenAI will eventually have to answer is not "is there a human in the loop?" but "who is accountable when the machine is wrong, and did we design for that on purpose?”