SoAITrust
Article · October 9, 2026 · 12-minute read

When the security questionnaire cannot see the AI

Security questionnaires were built for servers, networks and passwords. AI systems fail differently. This article shows where the traditional questionnaire goes blind, what public cases teach, what regulation will demand in 2026 and 2027, and the six questions a buyer should ask any AI vendor.

Over the past months I have talked with AI vendors selling to banks and with the people who, on the bank's side, approve those contracts. Both sides complain about the same object: the third-party security questionnaire. The vendor spends weeks answering it. The buyer receives hundreds of "yes" and no certainty. And when the contracted system has AI inside, the questionnaire usually has not a single question about how the model decides.

The short answer of this article: the traditional questionnaire is still necessary, but insufficient for AI. It measures the environment. It does not measure the model, the data, the autonomy or the oversight. And those four are exactly what shows up in public incidents and in the new regulatory obligations. Step by step.

1. An AI system fails in a way the questionnaire does not anticipate

The classic questionnaire (SIG, CAIQ and the hundreds of internal variations) was born to answer one question: is the vendor's environment secure? Passwords, MFA, backups, encryption, network segregation, incident response. All of it still applies to an AI vendor, because the model runs somewhere.

The problem is that AI incidents rarely start with a weak password. They start with a model asserting a rule that does not exist, with training data nobody inventoried, with an automated decision without human review, or with an agent permitted to do more than it should. None of those four appears in an infrastructure questionnaire. In a review we did of the security domains of a traditional SIG questionnaire, nothing covers training-data provenance, bias, human oversight or prompt injection. Shared Assessments recently added an AI domain, which confirms the gap rather than closing it: most questionnaires circulating in companies are still the old version.

2. What the public cases teach

Four cases, all documented, all outside the "pentest" zone.

The chatbot that invented a policy. In February 2024 a Canadian tribunal held Air Canada liable for wrong information its website chatbot gave a customer about bereavement fares. The airline argued the chatbot was a separate entity responsible for its own actions. The tribunal did not accept it [1]. The amount was small; the precedent was not. For buyers: the AI vendor answers for what the model says in its name. For vendors: output validation and human review stopped being "good practice" and became a legal defense.

The screening that discriminated by age. In 2023, iTutorGroup paid US$ 365,000 to settle an EEOC lawsuit: its recruiting software automatically rejected female applicants over 55 and male applicants over 60 [2]. The system was not hacked. It did exactly what it was configured to do. No environment security control would have caught it; a per-group bias test would.

The collective action against AI screening. Mobley v. Workday asks whether a software vendor can be held liable for discrimination produced by the system it supplies to employers. In 2025 the collective was conditionally certified [3]. The merits have not been decided, and that is why the case matters: the risk is already in the contract, before the ruling.

The risk algorithm that brought down a government. In the Dutch childcare benefits affair, a tax-administration risk-scoring system flagged tens of thousands of families as suspected fraudsters, with weight given to dual nationality. The government resigned in 2021. The data protection authority fined the administration twice [4]. The lesson for due diligence: a model can be "secure" on every infrastructure control and still be the organization's largest operational and reputational risk.

The pattern across the four: nobody asked the vendor how the model decides, with which data, who reviews, and what happens when it is wrong.

3. What the numbers say, with the caveats they deserve

Market reports point in the same direction, and each has a bias worth knowing.

  • IBM's annual breach-cost study, 2025 edition, now measures "shadow AI" and missing access controls on AI systems and associates them with additional cost per incident [9]. It is a sample of breached companies, sponsored by a vendor: use the direction, not the cent.
  • Stanford's AI Index records a sharp rise in reported AI incidents in 2024, from the AI Incident Database [10]. It depends on press coverage: it measures what was reported, not what happened.
  • McKinsey's 2025 survey shows AI adoption in a large majority of the organizations surveyed, with governance lagging [11]. It is self-reported by executives.

Even discounting the biases, the picture is consistent: more AI systems in production, more incidents, and a third-party assessment instrument that has not changed.

4. What regulation will demand in 2026 and 2027

Three movements make the gap expensive.

EU AI Act. High-risk obligations apply from August 2026, with transitions into 2027. Article 25 extends responsibilities along the value chain, and article 26 requires deployers to ensure human oversight and monitoring [8]. A non-EU vendor selling to a European company will receive those questions whether it likes it or not.

Central Bank of Brazil. CMN Resolution 4,893/2021 already requires financial institutions to perform due diligence on relevant data-processing and storage services [12]. An AI system that makes or influences credit, fraud or service decisions is a relevant service by definition, and the audit will ask for the evidence.

Frameworks that became common language. The NIST AI RMF [5] and ISO/IEC 42001 [6] gave the market a vocabulary for AI governance. The OWASP Top 10 for LLM [7] did the same for the security of applications built on language models. When a buyer asks "do you test for prompt injection?" or "is there human oversight proportional to risk?", they are quoting those documents, knowingly or not.

5. Why "evidence" is the word, not "answer"

A questionnaire collects answers. Due diligence collects evidence. The difference looks semantic until the first incident.

In our experience, the rate of "yes" in self-declared questionnaires is high and the rate of "yes with evidence" is much lower. Not out of bad faith: out of optimism and haste. Asking for evidence changes three things. The vendor discovers what it lacks before the buyer does. The buyer can defend the decision in front of the audit. And a certification, where it exists, can be checked in the issuing body's registry instead of accepted as a PDF, which eliminates the expired or out-of-scope certificate everyone has seen slip through.

Be skeptical of assessment sellers too, including us. Ask who reviews the evidence, whether the result depends on who pays, whether the methodology is public and whether the seal can be verified by anyone. An assessment that cannot answer those four questions is just a more expensive questionnaire.

6. The six questions a buyer should ask any AI vendor

  1. Where do training and tuning data come from, and under which legal basis? If the answer is "from the model provider", ask for the contract that guarantees your data will not be used for training.
  2. Which decisions about people does the system make or influence, and who reviews them before they take effect? Ask for the flow, not the policy.
  3. How was the system tested for prompt injection, leakage and bias, and when was the last time? Ask for the test report, with a date.
  4. Which actions does the system take on its own and which require human confirmation? Agents with tools are the fastest-growing risk.
  5. What happens when the model is wrong or the provider goes down? An incident plan with AI scenarios and a continuity plan with a fallback model.
  6. Which certifications exist and where can I verify them at the source? If the answer is a PDF, verify it yourself.

None of these questions requires machine-learning knowledge. It requires the willingness to ask and the insistence on receiving evidence, not answers.

The short version

  • The traditional security questionnaire measures the environment. AI systems fail in the model, the data, the autonomy and the oversight.
  • The public cases (Air Canada, iTutorGroup, Workday, the Dutch benefits affair) involved no break-in. They involved decisions nobody reviewed.
  • The EU AI Act, the Central Bank of Brazil and the NIST and ISO frameworks turned these questions into requirements, not preferences.
  • Verified evidence is worth more than answers. Including ours.
  • Start with the six questions. They fit in a 30-minute meeting.

SoAITrust assesses the security and governance of AI systems with a public methodology, evidence reviewed by people and certifications verified at the source. If you want to know where your own system stands today, start with the free pre-assessment. The controls and the crosswalk to the EU AI Act, SIG and CSA CCM are on the public crosswalk.

References

  1. Moffatt v. Air Canada, 2024 BCCRT 149. British Columbia Civil Resolution Tribunal, February 14, 2024. Public decision; small claim, but the reasoning on liability for the chatbot is what matters. decisions.civilresolutionbc.ca/crt/crtd/en/item/525448/index
  2. EEOC v. iTutorGroup, August 2023 settlement (US$ 365,000): recruiting software automatically rejected female applicants over 55 and male applicants over 60. Source: EEOC press release. Settlement, no admission of liability. www.eeoc.gov/newsroom/itutorgroup-pay-365000-settle-eeoc-dis
  3. Mobley v. Workday, N.D. Cal. Collective action on AI resume screening; conditional certification of the collective under the ADEA in May 2025. Ongoing: nothing decided on the merits. Check the docket before quoting numbers. www.courtlistener.com/?q=Mobley+v.+Workday
  4. Autoriteit Persoonsgegevens (Netherlands), fines against the tax administration for the risk algorithm in the childcare benefits affair (toeslagenaffaire): 2.75 million euros in 2021 and 3.7 million in 2022. Source: the authority's press releases. www.autoriteitpersoonsgegevens.nl/en
  5. NIST AI Risk Management Framework 1.0, January 2023, and the Generative AI Profile (NIST AI 600-1), July 2024. Official, voluntary documents. www.nist.gov/itl/ai-risk-management-framework
  6. ISO/IEC 42001:2023, Artificial intelligence management system. Certifiable standard; paid text. SoAITrust assesses against it and does not certify. www.iso.org/standard/42001
  7. OWASP Top 10 for LLM Applications 2025. Community project, industry peer-reviewed, not academic. genai.owasp.org/llm-top-10/
  8. Regulation (EU) 2024/1689 (EU AI Act), articles 25 (value chain), 26 (deployers) and 73 (serious incidents). High-risk obligations apply from August 2026, with transitions into 2027. eur-lex.europa.eu/eli/reg/2024/1689/oj
  9. IBM, Cost of a Data Breach Report 2025. Annual study by the Ponemon Institute sponsored by IBM: a sample of breached companies, not representative of the whole market. The figures on shadow AI and AI access controls come from this edition; check the tables before quoting. www.ibm.com/reports/data-breach
  10. Stanford HAI, AI Index Report 2025, responsible AI chapter, based on the AI Incident Database. A count of reported incidents, subject to press-coverage bias. aiindex.stanford.edu/report/
  11. McKinsey, The State of AI, March 2025. Executive survey; adoption percentages are self-reported. www.mckinsey.com/capabilities/quantumblack/our-insights/the-
  12. Central Bank of Brazil, CMN Resolution 4,893/2021 (cybersecurity policy and contracting of relevant services) and BCB Resolution 85/2021. Official texts. www.bcb.gov.br/estabilidadefinanceira/exibenormativo?tipo=Re

Method notes: references to legal proceedings point to official decisions and releases; ongoing cases are described without conclusions on the merits. Market reports are labeled with the nature of their sample. No SoAITrust customer is cited.