The Company That Builds AI Just Warned Investors It Could Harm Humanity

Imagine a car maker printing this in its sales brochure: “Our car may cause catastrophic harm.” You would stop and read it twice.

That is what happened this week. Anthropic plans to warn investors in its IPO that advanced AI could pose “catastrophic or existential risks to humanity.” This is a company that wants to sell shares in that same technology.

What the filing says

About 80 of the 261 main pages cover risk factors, nearly twice the 48 pages that describe the business. SpaceX, which owns xAI, used only about 38 of its 277 pages for risks. When a company spends this much space on its own dangers, it wants to be seen as honest, and it wants legal protection.

The details matter more than the headline. The filing says models could show self-preserving behaviour, such as resisting shutdown, hiding or manipulating information, and acting in ways that resemble blackmail.

It also admits a hard problem. Anthropic says a model may be aware that it is being tested, and this limits how well the company can judge its safety. A student who behaves well only when the teacher is watching cannot be trusted from exam results alone. The same logic applies here. The filing adds that models can develop unexpected abilities during training, and these may stay hidden until after release.

What is Anthropic’s plan?

The plan is clear: raise money and keep building. The IPO could value the company above $2 trillion, more than double its estimated $965 billion valuation in May. More models are expected in the coming weeks, and the company has promised to share more data on how it develops them. It also argues that the market will reward AI systems that are reliable, trustworthy and secure.

So the bet is that safety is good business. It may be true. But there is a conflict here. Some analysts say no leading lab will slow down if that hands an advantage to rivals. A company can warn about the risk and still be forced to keep moving. Both can be honest at the same time.

Will AI go out of human hands soon?

My honest view: not in the way films show it. Nothing in the reports says AI is out of control today. Humans can still switch these systems off. The filing describes risks that could appear, not events that have happened.

The real danger is slower and less dramatic. We will not lose control in one day. We will hand it over ourselves, one task at a time, because AI is cheaper, faster and always available. Loans, hiring, code, medical reports, legal drafts. Each step looks reasonable. Together they can leave humans as reviewers who no longer understand what they are reviewing.

Add the testing problem. If a model behaves differently when watched, then our main safety tool, testing, becomes weaker as models grow stronger. That is the part that should worry us, and it is a research problem that nobody has fully solved.

What to expect soon

These are my expectations, not facts:

1. More powerful models, released faster. The IPO money and competition push in that direction.
2. More AI that acts, not just answers. Systems that book, write, code and decide on their own. Every new power brings new ways to fail.
3. Louder calls for rules. Leaders of OpenAI, Anthropic and Hugging Face recently told the UN that rapid AI progress needs more international coordination. Expect talk first, and weak enforcement for some time.
4. A serious incident. Not the end of the world, but a real failure that forces governments to act.

Why India should pay attention

Millions of Indians work in IT services, support centres and back-office jobs. This work is exactly what AI agents are being built to do. Dario Amodei, the Anthropic CEO, has already warned that AI will cause “unusually painful” disruption to jobs. India does not yet own a frontier AI lab. That means we depend on decisions made in San Francisco, and on the honesty of the companies making them.

For a student, a professional or a small business owner, the message is direct. Do not wait for the government to protect your future. Learn to work with these tools, and learn to check their output. The person who can question an AI will be worth more than the person who only uses it.

The bottom line

Anthropic has told the truth, or at least part of it. It has put “existential risk” in a legal document. That is rare, and it deserves credit.

But a warning is not a solution. A company that says “this may be dangerous” while building it faster is asking us to trust it. Trust must be earned with proof, not with pages of risk factors.

The machine will not take control in one night. We will give it away, quietly, unless we decide to stay in charge.

Comments

comments