What’s Covered?
The report focuses on a fundamental challenge in AI governance: we talk about “audits,” “evaluations,” “impact assessments,” and “assurance” as if they’re interchangeable—but they’re not. Each serves different purposes, has different strengths and limitations, and supports different accountability goals. This document breaks it down into a matrix of goals and dimensions, creating a structure to help align expectations and improve policy design.
The authors start by outlining four key goals for AI assessments:
- Inform: to surface system characteristics and risks
- Evaluate: to judge adequacy or compliance with standards
- Communicate: to make impacts understandable to others
- Change: to motivate improvement or course correction
From there, the report maps two major axes:
- Scope of Inquiry (ranging from exploratory to highly specific)
- Independence of Evaluation (from self-assessment to fully external)
The body of the report explains these categories in depth:
- Exploratory assessments probe for unknown risks in complex systems and contexts.
- Structured reviews use predefined taxonomies to ensure coverage of known issues.
- Focused evaluations zero in on a specific harm or risk.
- Specific tests benchmark performance against a defined standard or requirement.
On the independence axis:
- Low-independence reviews (like internal documentation efforts) help map risk and set a foundation.
- Medium-independence efforts (e.g., contracted third-party audits) build trust and help verify processes.
- High-independence approaches (e.g., adversarial testing, public interest research) are critical for pressure and accountability.
The report emphasizes that these categories aren’t fixed or exclusive. An assessment can be highly specific yet still independent, or exploratory and still useful—even if internally conducted. What matters is matching the structure and independence level to the goals at hand.
The final section proposes design principles for more effective assessments:
- Be clear about purpose and thresholds.
- Disclose scope, methods, assumptions, and system versions.
- Support pluralistic, sociotechnical approaches that reflect real-world use and impact.
- Fund the work and value public interest contributions—not just corporate-driven oversight.
💡 Why it matters?
Accountability is only as strong as the assessments behind it. This report clarifies what’s often lost in buzzwords: not all “audits” are equal, and not every test is built to reveal harm or trigger change. The matrix helps policymakers and practitioners design more thoughtful, well-matched evaluation processes—and avoid false confidence in lightweight reviews.
What’s Missing?
The report avoids prescribing specific tools or assessment templates, instead offering a high-level conceptual model. This limits its immediate usability for those looking to operationalize assessment programs. There’s also little detail on enforcement—how the findings of these evaluations translate into mandatory changes or penalties. Finally, while the framework is strong on conceptual clarity, it leaves out discussion of AI system types (e.g., recommender systems vs. generative AI), which could help contextualize different risk profiles.
Best For:
Policymakers designing AI risk frameworks, audit teams, civil society watchdogs, and legal advisors seeking clarity on assessment approaches. Also valuable for companies building internal compliance programs that need to align with upcoming regulatory expectations.
Source Details:
Title: Assessing AI: Surveying the Spectrum of Approaches to Understanding and Auditing AI Systems
Authors: Miranda Bogen (CDT AI Governance Lab), with contributions from Chinmay Deshpande, Ruchika Joshi, Evani Radiya-Dixit, Amy Winecoff, and Kevin Bankston
Publisher: Center for Democracy & Technology (CDT), January 2025
Context: CDT is one of the longest-running digital rights advocacy groups in the U.S., with deep policy expertise and a strong presence in both Washington and Brussels. The AI Governance Lab at CDT focuses on developing technically informed governance strategies for AI that protect civil rights and democratic values. This report draws on that dual expertise in law and systems design to offer a policy-facing, cross-disciplinary toolkit.
