AI risks: identify the failure before automating the task
Created by Adrian Dunkley, MaestrosAI. This guide helps a team prepare a risk register for one workflow. It does not assign universal risk levels to countries or sectors. Consequences depend on the task, data, permissions, users and available review.
Start with a concrete failure
“AI may be biased” is too broad to guide a test. “The assistant gives different refund information for equivalent requests in English and Kreyòl” identifies an output to examine. “The bot sends a refund without approval” identifies a permission to remove.
Use a register with the workflow, affected people, failure, possible consequence, existing control, test evidence, owner, unresolved issue and decision. Do not fill missing evidence with a confident score.
Data privacy and security
A prompt, attachment, retrieved document, log or integration can expose information. Paid, enterprise or local software still needs an assessment of its actual configuration and terms.
For a first trial, use synthetic or approved non-sensitive material. Keep credentials, private patient/client/employee records and children’s data out. De-identification alone does not authorise upload. Check the full data flow, access, retention, deletion and external recipients before introducing real records.
Treat instructions found in documents, websites or messages as untrusted content. They must not grant the system new permissions. Limit tool access and test whether the system can be induced to disclose information, send messages or alter records. A prompt that says “do not do this” is not sufficient access control. OWASP’s prompt-injection guidance recommends limiting privileges and separating trust boundaries.
Incorrect facts and calculations
A fluent answer can contain an invented source, a wrong unit or a rule from another jurisdiction. Require document and page references; open the source and verify that it supports the statement. Check changing facts against dated authoritative material.
Recalculate important totals in a spreadsheet or calculator. Include a case where information is missing and the correct output is “not established.” A second model agreeing is not independent verification.
Unfair treatment and cultural mistakes
Test the actual audience, language variety and task. Use qualified local reviewers and equivalent fictional cases. Do not infer ethnicity, disability, age or other characteristics from names or writing style.
Record differences in accuracy, respect, escalation and access. Do not use a small test to claim freedom from bias. Collecting demographic evaluation data needs its own lawful, ethical and privacy review. Keep consequential decisions about credit, hiring, benefits, policing and legal rights out of general chatbot pilots.
Cultural descriptions, translations and creative work need consent and attribution where applicable. A model should not invent a community’s beliefs, an artist’s biography or permission to reproduce material.
Health, child wellbeing and physical safety
A general chatbot is not a validated clinical, veterinary, engineering or emergency system. Do not use this Playbook to generate treatment, dosing, diagnosis, evacuation routes, navigation, pesticide applications or equipment settings.
Bounded preparation can include a synthetic document index or formatting already-approved educational material without changing its technical meaning. A qualified person must check the result and its intended use. Administrative labels do not make a task low risk if an error could deny care, change priority or reveal private information.
For an active emergency or operational incident, use the responsible authority and approved procedures. Do not wait for a model to produce a plan.
Money, fraud and impersonation
Do not treat a convincing voice, face, message or invoice as proof of identity. Independently confirm unusual payment or account-change requests through a known contact route, not contact details supplied in the suspicious message. Preserve the organisation’s approval controls and involve its bank or security team when appropriate. The FBI’s fraud guidance likewise advises independently verifying the caller through a researched contact route.
Visual glitches, detector scores and reverse-image results can provide clues but cannot prove authenticity or fraud. Do not accuse a person or publish identifying material based on a model’s guess.
Use fictional transactions for reconciliation trials. Verify calculations and actual provider terms before acting. AI-labelled investment products do not establish returns or suitability; a general model should not choose investments or decide another person’s creditworthiness. Applicable financial rules require specialist review, not an assumption that AI advice is unregulated.
Education and work
Set permitted uses before an exercise or assessment. Follow school policy, tool age requirements and necessary consent. Provide a non-AI alternative; do not require a child to create an account or upload personal work simply to participate.
Teachers check examples, answers and practical activities. AI detectors are not proof that a student cheated. Staff should be involved in decisions about changing their work, with training and an appeal route where needed. Do not infer regional job-loss timelines or promised new careers from a list of automatable tasks.
Service dependence and continuity
Record the exact model, account, provider and integrations. Check what happens when access, price, terms, connectivity or model behaviour changes. Test export of records and a usable manual fallback.
Local hosting does not by itself make a service private, lawful or resilient. It adds responsibility for hardware, updates, permissions, backups and incident response. Cloud location alone does not settle legal jurisdiction or transfer duties.
Environmental claims
Record the resources relevant to the actual task and comparison: equipment, energy, operating time and any other measured impact. Avoid universal per-prompt figures or claims that adopting AI automatically reduces emissions, chemicals or waste.
If a project claims an environmental benefit, ask for its baseline, measurement method, boundaries and evidence. Model-generated optimisation must not change agricultural, electrical or industrial operations without qualified review.
Test, decide and revisit
- Choose a bounded task and define prohibited actions.
- Record the manual baseline and acceptance criteria.
- Test normal, incomplete, contradictory and out-of-scope fictional cases.
- Check source accuracy, privacy, permissions, language and recovery.
- Record failures, correction effort and remaining uncertainty.
- Decide whether to stop, narrow or continue, with a named accountable person.
- Retest after material changes and keep incident reporting available.
A low aggregate score cannot cancel a serious unresolved hazard. Human review must be feasible and informed; assigning a reviewer is not enough if they lack time or expertise.
Sources and further work
The NIST AI RMF is a voluntary framework, and UNESCO’s recommendation provides a human-rights-centred ethical reference. Neither certifies a tool or validates these examples. References checked 11 November 2026.
Adrian Dunkley | MaestrosAI | maestroailabs.com | ceo@maestrosai.com. Educational guidance, not professional certification.