Anthropic cannot accede to Pentagon's request in AI safeguards dispute, CEO says

Anthropic said it would not accede to the Pentagon’s request to eliminate safeguards from its AI systems, despite threats to deem the company a “supply chain risk” and remove it from the Department of Defense’s systems, putting a $200 million contract at risk.

The Pentagon’s dispute with Anthropic stems from the AI startup’s refusal to remove safeguards that would prevent its technology from being used to target weapons autonomously and conduct surveillance in the United States.

Earlier in the day, Pentagon spokesperson Sean Parnell said on X that the department has no interest in using AI to conduct mass surveillance of Americans nor does it want to use AI to develop autonomous weapons that operate without human involvement.

“Here’s what we’re asking: Allow the Pentagon to use Anthropic’s model for all lawful purposes,” Parnell said, adding that the company had until 5:01 pm ET on Friday to decide.

“Otherwise, we will terminate our partnership with Anthropic and deem them a supply chain risk.”

In a statement on Thursday, Anthropic CEO Dario Amodei underscored the company’s opposition to the Pentagon using its AI models for mass domestic surveillance or to power fully autonomous weapons, the latter being because “frontier AI systems are simply not reliable enough.”

A source close to the company said Anthropic was not accusing the Pentagon of planning to engage AI for either use, but was offering a product safety judgment.

The source said AI systems were not reliable enough for “life-or-death targeting” because they behave unpredictably in novel scenarios, which could lead to “friendly fire, mission failure or unintended escalation” in a weapons context.

Using AI for mass domestic surveillance was problematic because current laws do not restrict the conclusions that AI can draw by aggregating large amounts of data, the source said. That meant AI could build population-level profiles that “no law explicitly prohibits but that clearly violate the spirit of constitutional protections,” they added.

Amodei said he hoped the Pentagon would reconsider but that the company “will work to enable a smooth transition to another provider,” if the Pentagon decided to cancel the contract.