Quitting your job isn’t enough....

· 7 min read

Jacob Coxon quit Anthropic because he fears development of AI, more specifically RSI may end humanity in the next 3 years and warning that frontier labs are “racing straight to self-improving superintelligence” and gambling with humanity’s future.

AI takeover warnings surface the problem to masses, but it doesn’t do anything to advance AI safety.

The overall sentiment is quite negative about AI and specifically RSI(Recursive self-improvement). This will only result in more fearmongering among people not in tech.

For the past 3 days, it’s all anyone is talking about on any platform. whether RSI will end humanity and it’s true or not only time will tell, but why is no one talking about what to do, rather than only speculating?

On one hand, these concerns should be taken seriously given he worked directly on pretraining research at OpenAI and Anthropic, and reportedly walked away before his Anthropic equity vested, but also need to be taken seriously with a grain of salt as this stance doesn’t give us much of an action plan.

I work closely in AI research, and I think the more useful question is not whether one person’s timeline is correct. It is this:

If recursive self-improvement is a serious possibility, what should we be doing now to catch up?

What RSI actually means

The non-tech version of RSI might seem like: An AI wakes up, copies its own weights, finds thousands of GPUs, hosts itself somewhere on the internet and starts creating increasingly intelligent versions of itself.

That is not the most plausible near-to-medium-term version of recursive self-improvement at all.

RSI can happen inside infrastructure that humans have already built and provisioned. An AI system could:

  1. Design a research experiment.
  2. Write the necessary training or evaluation code.
  3. Run the experiment using an existing compute cluster.
  4. Analyze the results.
  5. Modify the model, training process or inference system.
  6. Produce a more capable successor.
  7. Use that successor to accelerate the next iteration.

The model does not, and can not secretly purchase GPUs or construct a data center. The lab operating it may have already provided the compute, data, permissions and research environment. We are not at fully autonomous RSI today, and it is not inevitable. But we are already automating portions of the loop.

Anthropic reports that Claude now authors a significant proportion of the code merged into its codebase and that its engineers are producing considerably more code than before. The company also explicitly states that full recursive self-improvement has not arrived and may not arrive but that institutions should prepare for the possibility. ref: [Anthropic’s RSI analysis] This is why the fear particularly gets attached to people who don’t know how this technology works.

But looking at the lens of RSI that automates an existing research pipeline makes it a systems problem, and not replicating itself whenever it pleases so.

AI is not completely unknowable

It is common to hear that nobody knows how AI works.

What we do not fully understand is how billions or trillions of learned parameters collectively represent concepts, produce reasoning and generate unexpected behavior. We understand how the system is constructed. We cannot yet reliably explain everything it learns internally.

Mechanistic interpretability is attempting to reduce this gap by studying internal model representations. The field is still early, but it is making real progress. Researchers are developing methods to inspect activations, identify features and translate portions of internal model computation into human-readable interpretations.

For example, Anthropic’s recent work on natural-language autoencoders attempts to convert model activations into textual descriptions. The method remains imperfect, the researchers explicitly warn that its explanations can hallucinate but it demonstrates that the “black box” is not necessarily impenetrable. ref: [Anthropic’s interpretability research]

We need to acknowledge that while progress is slow, it’s not zero. and while such progress catches up, what we do know concretely is that we understand the scaffolding around the AI systems, such as transformer architectures, gradient-based training, tokenization, inference systems and the infrastructure used to train and serve models.

Engineers can measure activations, modify model weights, inspect attention patterns and observe behavior across evaluations. So, while attack surface of misalignment is larger, a model control surface in architecture design, model hosting, infrastructure, and inference is equally comparable that we can curb it’s harmful effects in near-term.

RSI can advance AI safety too

The central concern behind RSI is that AI-assisted research could accelerate AI capabilities faster than humans can understand or control them. That concern is super valid.

But the same acceleration can also be applied to safety research.

AI systems can help researchers:

This does not mean AI will automatically solve AI safety. A model evaluating another model can introduce its biases to the process, blind spots or evaluation awareness that we are trying to detect. But we need to acknowledge that model capability is not the only process that RSI can accelerate.

The relevant question to figure out here, is whether alignment, interpretability and governance can improve quickly enough and whether we deliberately allocate compute, researchers and institutional attention toward AI safety.

Compute is one surface in our control

Frontier models require enormous physical infrastructure, and have dependency on accelerators, high-bandwidth networking, storage, energy, cooling systems and specialized inference software. That makes autonomous harmful replication for AI incredibly difficult.

Such compute constraints for hosting AI should instead be construed as a potential control surface we can leverage in our favour.

Training runs and large-scale inference workloads leave behind infrastructure-level signals, and organizations can intervene here to decide which models access particular clusters, what code is executed, which network destinations are reachable and whether an agent can modify its own research environment, translating AI safety into partially a system problem with following areas for safety work:

By no means will addressing these count as a completely secure and safe AI system, but they’re the interventions we can implement that a purely philosophical discussion can’t address.

What an actionable RSI agenda could look like

If recursive self-improvement is a real concern, we need to translate it into measurable engineering and governance problems.

1. Evaluate the self-improvement loop

We need evaluations that measure whether an AI can conduct meaningful AI research, not merely whether it can solve long-horizon coding problems, which seems to be every lab’s focus these days.

Can it form hypotheses, design experiments, interpret ambiguous results, improve training pipelines and produce changes that hold its validity across iterations?

2. Invest in interpretability and alignment alongside capabilities

Safety research should be integrated throughout the entire model training pipeline from data collection, training, evaluation, deployment and monitoring, and not merely a final test it passes when model is done training.

3. Secure automated research environments

If agents increasingly conduct experiments, their sandboxes, tools, permissions and communication channels become part of the safety interventions. so, it’s not just about the model, but all the scaffolding around it as well.

4. Create independent audits and deployment thresholds

Labs should not be the only organizations deciding whether their systems are safe to deploy. Although OpenAI and Anthropic have been transparent about their process, progress and internal opinions around this, having external researchers and auditors enough access to validate important safety claims would make the results unbiased. Governments and industry bodies also need to establish technically meaningful standards rather than broad statements about “responsible AI.”

5. Expand research participation carefully

Open-weight models allow universities, independent researchers and smaller organizations to study model behavior without relying entirely on frontier labs. That can accelerate interpretability and alignment research.

But, it also introduces legitimate proliferation and misuse risks. Open-weight development is therefore neither automatically safe nor automatically dangerous. The trade-offs need to be examined very carefully here.

Open-weights model is a highly polarized topic that I would rather skip addressing here.

All to say, Coxon’s resignation was not meaningless. Walking away from a frontier lab with unvested equity is a principled decision most people wouldn’t take very easily. this warning also brought AI safety into a public discussion at a scale that most technical papers never reach, but awareness is only the beginning.

If the conversation stops at “AI might kill everyone,” the public is left with fear and no way to evaluate the risk.

If the technical community responds only by mocking such predictions, it avoids the harder questions about model capability control, oversight and institutional readiness.

We do not need absolute certainty about an AI takeover timeline before taking reasonable precautions.

We need Researchers converting abstract risks into evaluations.

We need Engineers building secure research environments.

We need Interpretability teams making model internals more observable.

We need Independent organizations auditing frontier model’s safety claims.

Finally, What ties everything is Policymakers creating enforceable deployment standards.

Humankind has survived thousands of years, and evolved through it. We will survive, nay thrive in the era of AI development.