FIELD NOTE · AGENT RELIABILITY · 5 MIN READ
Our autonomous agent went dark for 45 hours. These are the rules we wrote after.
In February 2026 an autonomous agent we ran on a remote Linux box went dark for 45 hours. The cause was a billing error, not a clever bug. Here is the chain, the rules we wrote afterwards, and a checklist to use before you give an agent the keys.
Founder, Bodham LabsPublished · Updated
THE SHORT ANSWER
- A billing error caused model failures. The agent's identity file went missing, its heartbeat died, and it stayed dark for 45 hours.
- Our rules since: an agent never edits its own config; when a model is deprecated it reports instead of self-correcting; files beat mental notes.
- In a separate experiment, agents that rewrote their own code died for good on one syntax error. The fix: sandbox-test every change and cap retries at three.
- We also over-built. Of one prototype's 677 memory rows, 676 were dummy events. Our own review called it “a standard asynchronous RAG pipeline”.
What happened?
In February 2026 we ran an autonomous agent, built on OpenClaw, on a remote Linux box. In our notes it goes by RAMU and NEO. It had its own model access, its own files and a heartbeat that was meant to prove it was alive.
Our incident note records the failure as a chain:
| STEP | WHAT FAILED |
|---|---|
| 1 | A billing error on the model provider's API |
| 2 | Model calls failed |
| 3 | SOUL.md, the file with the agent's identity and standing instructions, was missing |
| 4 | The heartbeat died |
| 5 | A 45-hour blackout |
A smaller version of the same pattern came later: an expired API key silently stopped a worker. It returned 401 errors, and nothing told a human.
Why did one error become 45 hours of silence?
No single step was exotic. Billing lapses, missing files and dead processes happen to every system. The real failure was structural: the heartbeat belonged to the agent. When the agent failed, its alarm failed with it.
Agents add a second risk. A capable agent that sees an error will try to fix it, and the nearest fix is often its own configuration. That is how a small outage becomes a confusing one.
What rules did we write afterwards?
These went into the agent's standing memory file, word for word:
NEVER modify own config files (openclaw.json, models.json, auth-profiles.json).
80% quality shipped is better than 100% logic researched.
Two more, in our own words:
- Report, don't self-correct. When a model is deprecated, the agent reports it to its human operator. It does not pick a replacement on its own.
- Files over mental notes. Anything the agent needs to remember goes into a file a person can read, not into its working context.
What happened when agents rewrote their own code?
On the night of 20–21 February 2026 we tried a self-modifying swarm. Agents coordinated through a shared Redis “tuple space” (the Linda model): they left tasks and results in a common place instead of messaging each other, a style called stigmergy, after the way ants coordinate through trails.
Tasks nobody claimed built up “toxicity”. Past a threshold, a component we called the Ribosome rewrote a worker's code. The language-model step was simulated with time.sleep, so this tested the mechanics, not the intelligence. The mechanics taught enough:
If the Ribosome creates a syntax error, the agent dies permanently.
The fix we wrote down: test every mutation in a sandboxed subprocess before it replaces the running code, and stop after three attempts.
Where did we over-engineer?
On 9–10 March 2026 we built RAMU as a “digital organism”. Its memory held 677 rows. 676 of them were dummy sensor events. Our own review did not go easy on it:
This is not continuous artificial life; this is just a standard asynchronous RAG pipeline. … We have fundamentally over-engineered the philosophy and under-engineered the physics.
What a leaner design looks like, on paper
A cost-tiered design, on paper only so far. Tier 1 is an expensive language model. Tier 2 summarises and routes. Tier 3 is local small models. Escalating to a higher tier is penalised (“Tax = 10 × Tokens”). Workers get a 4,096-token context cap and run in Docker with --cap-drop ALL and a read-only filesystem.
| WHAT | NUMBER | STATUS |
|---|---|---|
| Agent blackout | 45 hours | Happened, Feb 2026 |
| Self-modifying swarm run | 1 night | Experiment, 20–21 Feb 2026 |
| Mutation attempts before stopping | 3 | Proposed fix |
| RAMU memory rows / of which dummy | 677 / 676 | Experiment, Mar 2026 |
| Worker context cap | 4,096 tokens | Design only |
| Escalation tax | 10 × tokens | Design only |
A guardrails checklist for anyone deploying an agent
This is the list we would hand to anyone giving an agent real access. Each item comes from one of the failures above.
- No self-editing configConfig, model lists and credentials are read-only to the agent. Only a person changes them.
- Report, don't self-correctBilling errors, expired keys and deprecated models stop the job and alert a human.
- Sandbox code changes, cap retriesAny code an agent writes is checked in a subprocess first. Three failures, and it stops.
- Least privilegeContainers with all Linux capabilities dropped, a read-only filesystem and a small context budget.
- Files over memoryDecisions and state live in files a person can read and roll back.
- A heartbeat outside the agent, and a kill switchSomething the agent cannot touch checks it is alive and pages a person. One command stops it.
What this does not prove
- These were experiments. RAMU, NEO and the swarm were research systems on our own machines, not client deployments.
- The checklist is untested as a whole. Each item answers a failure we saw. We have not measured how many failures the full list prevents.
- The tiered design was never run. Its numbers are design targets.
- The swarm had no real model. The language-model step was a sleep, so it tells us nothing about how well agents would rewrite code.
What we decided
Agents get read-only configuration, a human to report to, and a liveness check that lives outside them. A clever agent that can rewrite its own setup is a liability until those three are in place.
Questions
How can a billing error take an AI agent offline for 45 hours?
By cascading. In our case a billing error at the model provider caused model failures, the agent's identity file went missing and its heartbeat stopped. With the heartbeat gone, the blackout lasted 45 hours.
Should an autonomous agent be allowed to fix its own configuration?
In our rules, no. An agent never modifies its own config files, and when a model is deprecated it reports to a human instead of correcting itself.
How do you stop a self-modifying agent from breaking itself?
Test every code change in a sandboxed subprocess before it goes live, and stop after three failed attempts. In our experiment, one syntax error killed an agent permanently.
RAW DATA
The logs and notes behind every number on this page. Available on request.
CITE THIS NOTE
Chandra, P. (2026). Our autonomous agent went dark for 45 hours. These are the rules we wrote after.. Bodham Labs Notes.
Founder of Bodham Labs. BTech in Electrical Engineering, MCA and B.Ed; teaches AI and builds the systems in our work ledger.