AI & Ethics
26 tools you can use for nothing to check an AI system rather than read about it: bias detectors, dataset audits, model documentation, red-teaming scanners, explainability libraries, and checklists that need no code at all.
Every entry says who makes it, what it does, and — the part most lists leave out — what it will not tell you. None of these produce a verdict. A clean report from any of them means “it passed these particular checks”, never “this is fair”. Checked September 19, 2026.
Start with the principles
A tool is only useful once you know which question you’re asking. The principles, the documented risks and the questions to ask before you adopt anything are on the main ethics guide.
These measure whether a model's errors fall unevenly across groups of people, and let you compare fixes.
Microsoft and open-source contributors (MIT license)
Measures how a model's accuracy and error rates differ between groups, then offers mitigation algorithms that trade a little overall accuracy for a smaller gap.
What it won’t tell you: It can only measure gaps for the groups you give it data about. If you never recorded the attribute, the gap stays invisible.
IBM, now under the Linux Foundation AI & Data project
A large library of fairness metrics and bias-mitigation algorithms for datasets and models, with tutorials on credit scoring and medical data.
What it won’t tell you: It gives you dozens of metrics that can disagree with each other. Choosing which definition of fairness applies is still a human judgment, not an output.
Center for Data Science and Public Policy, University of Chicago
A bias audit toolkit aimed at policy and public-sector use: you feed it predictions and it produces a group-by-group fairness report.
What it won’t tell you: Built for classification decisions about people. It has nothing to say about a generative model's outputs.
Google PAIR
An interactive visual tool: change one person's data and watch the prediction move, or slice performance by group without writing analysis code.
What it won’t tell you: It shows you behavior, not causes. A visible disparity still needs investigating in the training data.
Google, part of TensorFlow Extended
Computes and charts fairness metrics across slices of data at scale, designed to be run repeatedly as a model is retrained.
What it won’t tell you: Assumes a TensorFlow-shaped pipeline. Useful mainly if you already build models that way.
Holistic AI (open-source library)
Bias, robustness and explainability measurements in one package, including some support for generative models rather than only classifiers.
What it won’t tell you: The company behind it also sells audits. The library is open, but read its framing with that in mind.
Most bias arrives with the data. These help you look at what is actually in a dataset, and write down what you found.
Google Research
Browse widely used public image and text datasets in the browser: what is in them, how the labels are distributed, and which correlations look suspicious.
What it won’t tell you: Only covers the datasets Google has loaded. Your own data has to be examined with something else.
Timnit Gebru, Kate Crawford and colleagues
The original question set for documenting a dataset: how it was collected, who is in it, who consented, and what it should not be used for.
What it won’t tell you: A template, not a check. It surfaces problems only if you answer honestly — including the answers you'd rather not write down.
Google Research
A practical kit — worksheets, examples and a starter template — for producing structured dataset documentation with a team rather than alone.
What it won’t tell you: Process guidance. It will not tell you whether your dataset is fit for purpose.
The Data Nutrition Project
A short, readable label format for a dataset, modeled on food packaging, so a non-specialist can see provenance and known gaps at a glance.
What it won’t tell you: Deliberately brief. It summarizes; it does not audit.
Hugging Face
Every dataset on the hub can be previewed row by row in the browser, alongside a card describing its source, license and known limitations.
What it won’t tell you: Card quality is entirely down to whoever uploaded it. A blank card is common and means nobody has audited anything.
Cleanlab (open-source core)
Finds label errors, duplicates and outliers in your own dataset automatically — the mundane data problems that get mistaken for model bias.
What it won’t tell you: The open-source library is free; the company's hosted studio product is not. Stick to the library.
For writing down what a model is for, and for checking what a vendor has disclosed.
Google, TensorFlow project
Generates a model card — intended use, evaluation data, performance by group — from your own evaluation results.
What it won’t tell you: Produces a document. It does not verify that the numbers in it are good ones.
Stanford Center for Research on Foundation Models
Scores major model developers on 100 disclosure indicators — data, labor, compute, risk — so you can see which questions a lab refuses to answer.
What it won’t tell you: It scores disclosure, not behavior. A high score means a lab tells you a lot, not that what it tells you is reassuring.
Hugging Face
The de facto public record for open models: license, training data, evaluations and stated limitations, on the same page as the download.
What it won’t tell you: Self-reported by the publisher, and frequently thin on exactly the sections that matter most.
For finding out how a language model fails when someone is actively trying to make it fail.
NVIDIA (originally an independent project by Leon Derczynski)
A vulnerability scanner for language models: runs hundreds of probes for prompt injection, jailbreaks, data leakage and toxic output, then reports what landed.
What it won’t tell you: Known attacks only. A clean report means it survived the probes in the suite, not that it is safe.
Microsoft AI Red Team
A framework for automating red-team campaigns against generative systems, including multi-turn attacks and scoring of the responses.
What it won’t tell you: A harness, not a verdict. You still have to decide what counts as a harmful response for your use.
Giskard (open-source library)
Scans models — including LLM applications — for bias, hallucination, prompt injection and robustness problems, and generates a test suite you can re-run.
What it won’t tell you: The open-source scanner is free; the hosted hub has paid tiers. The library alone is enough to get a report.
For asking why a model produced a particular answer — with the caveat that these are approximations, not confessions.
Scott Lundberg and open-source contributors
Attributes a prediction to the individual input features, with charts that work for a non-specialist audience.
What it won’t tell you: An approximation of the model's behavior. Two explanation methods can disagree about the same prediction.
Marco Tulio Ribeiro and colleagues
Explains one prediction at a time by building a simple local model around it — the original paper behind most explainability tooling.
What it won’t tell you: Local by design. It tells you nothing reliable about the model overall.
Meta, for PyTorch
Attribution methods for neural networks, including vision and text models, integrated into PyTorch training code.
What it won’t tell you: Aimed at model developers; not much use if you only consume an API.
No code, no model access required. These are the ones to reach for if you're deciding whether to adopt a tool at all.
DrivenData
A short, practical ethics checklist for data projects — data collection, storage, analysis, deployment — that drops straight into a repository or a project doc.
What it won’t tell you: Prompts, not policy. It raises the questions; your team still has to answer and record them.
US National Institute of Standards and Technology
The reference framework for identifying, measuring and managing AI risk, with a companion playbook of concrete suggested actions per function.
What it won’t tell you: Voluntary and deliberately generic. It describes a process, not thresholds.
European Commission High-Level Expert Group on AI
A question-by-question self-assessment across seven requirements — human oversight, robustness, privacy, transparency, fairness, wellbeing, accountability.
What it won’t tell you: Self-assessment. Nobody checks your answers, and it predates current generative systems.
US White House Office of Science and Technology Policy (2022), led by Alondra Nelson
Five plain-language protections people should expect from automated systems, each with a section on what companies should actually do.
What it won’t tell you: Never binding, and the current administration has removed it from whitehouse.gov — the link goes to the official archive.
Doteveryone
A one-hour team workshop format: list the intended and unintended consequences of what you're building, then decide which you'll act on.
What it won’t tell you: Doteveryone closed in 2020, so the method is no longer maintained. It still works as a meeting structure.
