What’s Covered?
The report begins with a literature review of AI-enabled recommendation systems, aiming to clarify how explainability and interpretability are described and evaluated in academic research. It zeroes in on two key questions:
- Do researchers distinguish between explainability and interpretability in a meaningful way?
- What methods are used to evaluate claims about explainable AI?
The findings are clear but concerning:
- Most papers use explainability and interpretability interchangeably or define them inconsistently.
- Descriptions typically fall into four themes: references to other AI principles (like fairness or transparency), technical implementation, system purpose (e.g. justifying a recommendation), and intended user outcomes (e.g. building trust).
- Researchers primarily use five evaluation methods: case studies, comparative evaluations, parameter tuning, user surveys, and operational evaluations.
- There is a strong bias toward system correctness (i.e. whether the system was built as intended) over system effectiveness (i.e. whether it actually works for users in practice).
These results are important because they suggest a disconnect between the way researchers conceptualize explainability and how those ideas are measured. The authors highlight that if research practices are misaligned or unclear, downstream policies may end up incentivizing box-ticking exercises rather than meaningful improvements in AI transparency.
💡 Why it matters?
Explainability is often cited in AI policy frameworks — from NIST to the EU AI Act — but this report shows that the concept is still slippery in practice. Without a shared understanding or standard for how to evaluate it, explainability risks becoming an empty buzzword. Worse, poor evaluation practices could result in false confidence about system safety or user comprehension. If AI governance relies on explainability, we need evaluation methods that can actually tell us whether systems are understandable and helpful — not just technically compliant.
What’s Missing?
The study is thorough in its descriptive analysis, but doesn’t engage deeply with end-user outcomes. While it criticizes the overuse of correctness evaluations, it stops short of offering robust alternatives or concrete metrics for evaluating effectiveness. Additionally, the focus on recommendation systems, while justified, leaves open the question of whether these findings generalize to other high-risk domains like healthcare, hiring, or criminal justice. Some readers may also wish for more engagement with cross-cultural or regulatory perspectives outside the U.S. and EU.
Best For:
Policymakers, AI auditors, and research leads working on AI explainability or user-facing systems. It’s particularly relevant for teams preparing to meet transparency requirements under upcoming regulations. It also offers useful insight for anyone designing evaluation frameworks or building internal standards for AI safety and performance validation.
Source Details:
Narayanan, M., Schoeberl, C., & Rudner, T. (2025). Putting Explainable AI to the Test: A Critical Look at AI Evaluation Approaches. Center for Security and Emerging Technology (CSET).
All three authors are affiliated with CSET, a research organization at Georgetown University focused on AI policy, security, and emerging technology. Their backgrounds bridge technical AI research and policy analysis, and the work is part of a broader CSET initiative to inform governance through empirical, evidence-based insights. The brief draws on peer-reviewed academic literature, but it’s written for a policy audience — accessible, pointed, and pragmatic.
