Partnering with Accenture on embedded evaluation
Anthropic announced a partnership with Accenture on independent evaluation of frontier AI, calling it an important step toward the commitment in CEO's essay “We Must Pace the Frontier” to embed evaluators within Anthropic. The work will be led by Faculty, Accenture's specialist AI business, and will include evaluating and red-teaming models, conducting alignment assessments, and testing model safeguards. Accenture helps businesses and governments deploy AI across many industries, and Anthropic says its understanding of enterprise AI use in practice informs its safety approach and will bring that perspective to evaluating Anthropic's models.
Anthropic and Accenture each expect to invest at least $1 billion over the next five years to build capacity in this area. Embedded evaluation is new, and many operational details are still being worked out. Unlike today's external evaluators, embedded evaluators will work inside AI companies with access comparable to an employee's. That access lets them watch models take shape during training, follow the decisions that govern how models are built and deployed, and speak directly to employees. From that vantage point, they can assess how a company operates, verify that it is keeping safety commitments, identify blind spots, report incidents, and give the public a more informed account of benefits and risks.
Anthropic stresses that independent embedded evaluators do not reduce its accountability but help make it more verifiable, and that model safety remains its responsibility. There are no standards yet for what information embedded evaluators should access or how they should report findings, and there is no settled system for funding independent evaluation. Long term, Anthropic thinks funding should come from pooled or government sources, as it called for in its Advanced AI Framework in June. Because neither exists today, it plans to work with different evaluators under different funding arrangements.
Given the work's importance and urgency, Anthropic will fund Accenture's work directly. It is also in dialogue with METR and other nonprofit evaluators to pilot elements of embedded evaluation using their own funding. Ultimately, Anthropic believes frontier AI needs an ecosystem of evaluators operating with shared standards. It expects frontier labs to work with several organizations at once: the Accenture partnership is non-exclusive, Anthropic will work with other evaluators to be announced in coming weeks, and Accenture will work with other AI developers in similar capacities.
Anthropic says it will continue to train and release frontier models and wants independent evaluators working alongside it. It is sharing these early efforts so people and other AI developers can see its process, expects its approach to evolve as the field matures, and will share more as work begins and additional evaluators are brought on.