Raji and colleagues examine the ethics of the audits themselves: who is photographed, who consented, and what an auditor owes the people in the test set.
From the site: Although essential to revealing biased performance, well intentioned attempts at algorithmic auditing can have effects that may harm the very populations these measures are meant to protect. This concern is even more salient while auditing biometric systems such as facial recognition, where the data is sensitive and t…
Why I recommend it: A rare paper about the ethics of doing ethics work. Free on arXiv.
Raji and colleagues argue many deployed AI systems fail on their own stated terms — they simply do not work — and that this belongs in the harm conversation alongside bias.
From the site: Deployed AI systems often do not work. They can be constructed haphazardly, deployed indiscriminately, and promoted deceptively. However, despite this reality, scholars, the press, and policymakers pay too little attention to functionality. This leads to technical and policy solutions focused on "ethical" or value-ali…
Why I recommend it: The first question is not "is it fair" but "does it work at all". Free in full on arXiv.
Deborah Raji and co-authors set out a practical, stage-by-stage internal audit process for AI systems, from scoping through to post-deployment review.
From the site: Rising concern for the societal implications of artificial intelligence systems has inspired a wave of academic and journalistic literature in which deployed systems are audited for harm by investigators from outside the organizations deploying the algorithms. However, it remains challenging for practitioners to ident…
Why I recommend it: The closest thing to a step-by-step audit template you can use inside an organization. Free on arXiv.
An open-source library of metrics and algorithms for finding and reducing unwanted bias in datasets and models, in Python and R. Free.
From the site: A comprehensive set of fairness metrics for datasets and machine learning models, explanations for these metrics, and algorithms to mitigate bias in datasets and models. - Trusted-AI/AIF360
Why I recommend it: For the harm that shows up in ordinary systems long before anything dramatic does — hiring screens, lending, scoring. Measuring bias is the easy half; deciding what fair means is yours.
