Postmortem · Open Source

Fixing a deadlock that kept OpenClaw Gateway from restarting after upgrade

An ESM dynamic import, a signal handler, and a race condition — the gateway broke silently for six days.

2026-05-22 myps6415 PR #84890 · openclaw/openclaw

374k

GitHub Stars

+28/-1

Diff size

Issue priority

Background

OpenClaw is a 374k-star open source AI assistant framework. I use it day-to-day in my homelab to run a Telegram bot. Its gateway is a Node.js process running under macOS launchd, exposing a REST API that various agents call.

One day I clicked the Update button in the control panel. The update reported status=ok and everything looked normal. A few hours later I noticed the gateway throwing errors:

13:35:15  request handler failed: ERR_MODULE_NOT_FOUND
        task-registry.maintenance-B-jsfe-3.js
13:36:55  ERR_MODULE_NOT_FOUND  status.link-channel-BK3OG460.js
14:16:52  ERR_MODULE_NOT_FOUND  task-registry.maintenance-B-jsfe-3.js
14:16:54  [restart] request coalesced (already in-flight) reason=update.run
14:31:31  ERR_MODULE_NOT_FOUND  hooks-DBKknpfW.js
      production log · PID 30106

The stranger part: the last entry in gateway-restart.log was six days old. The gateway was running, but would never restart. The only workaround was to manually run launchctl kickstart -k gui/$(id -u)/ai.openclaw.gateway.

Root cause

Reading the source, I traced it to a chicken-and-egg deadlock caused by three independent design choices stacking together.

Problem 1: the SIGUSR1 listener uses dynamic import

const onSigusr1 = () => {
gatewayLog.info("signal SIGUSR1 received");
void (async () => {
  const {
    markGatewaySigusr1RestartHandled,
    ...
  } = await loadGatewayLifecycleRuntimeModule();  // ← dynamic import
  request("restart", "SIGUSR1", restartReason);
  markGatewaySigusr1RestartHandled();              // ← never called
})();
// Note: no .catch() here
};
      src/cli/gateway-cli/run-loop.ts (before)

Problem 2: chunk hashes rotate after an upgrade

After npm install -g openclaw@newer swaps the package, the chunk filenames in dist/ (each suffixed with a content hash) are all replaced. The old process is still running, but the paths it tries to import no longer exist. Result:

loadGatewayLifecycleRuntimeModule()
→ import("./lifecycle.runtime-OLDHASH.js")
→ ERR_MODULE_NOT_FOUND  ← silent reject, no .catch()

Problem 3: the restart token gets stuck

OpenClaw’s restart system uses a pair of counters to prevent duplicate triggers:

function hasUnconsumedRestartSignal(): boolean {
return emittedRestartToken > consumedRestartToken;
}

export function scheduleGatewaySigusr1Restart(opts?) {
if (hasUnconsumedRestartSignal()) {
  restartLog.warn(`restart request coalesced (already in-flight)`);
  return { coalesced: true };  // ← bail out, do nothing
}
// ...
}
      src/infra/restart.ts

Because the SIGUSR1 handler rejects at await loadGatewayLifecycleRuntimeModule(), markGatewaySigusr1RestartHandled() — i.e. consumedRestartToken++ — is never called. emittedRestartToken stays permanently above consumedRestartToken, and every subsequent restart request gets coalesced away.

Timeline

2026-05-15

Last successful restart

Final valid entry in gateway-restart.log
2026-05-18 → 05-19

Version upgrade triggers the bug

npm install -g openclaw@newer completes; chunk hashes rotate
14:16:54

Gateway enters coalesce loop

[restart] request coalesced (already in-flight) reason=update.run
Six days running

ERR_MODULE_NOT_FOUND keeps firing

PID 30106 uptime > 36h, process never replaced
2026-05-21

Filed PR #84890

Attached production log + live terminal proof
2026-05-22

PR merged by vincentkoc

+28/-1, single-file change

The fix

The core idea is simple: load the lifecycle module into the ESM cache before installing the signal listener. After that, every await loadGatewayLifecycleRuntimeModule() hits the cache instead of disk, so chunk rotation can’t affect it.

Eager-load: import before installing the signal listener

First line of runGatewayLoop, before acquireGatewayLock

Defence in depth: add .catch() to the SIGUSR1 IIFE

Use the cached reference; guarantee the token is consumed

Fail-fast contract

If the lifecycle module isn't found at startup → fail loudly, let the supervisor handle it, no half-dead process

src/cli/gateway-cli/run-loop.ts

@@ -120,6 +120,9 @@ export async function runGatewayLoop(opts) {

+ const eagerLifecycleRuntime = await loadGatewayLifecycleRuntimeModule();

await acquireGatewayLock();

// ... rest of init ...

@@ -600,8 +603,15 @@ const onSigusr1 = () => {

void (async () => {

const { ... } = await loadGatewayLifecycleRuntimeModule();

request("restart", "SIGUSR1", restartReason);

- markGatewaySigusr1RestartHandled();

+ eagerLifecycleRuntime.markGatewaySigusr1RestartHandled();

+ }).catch((err) => {

+ gatewayLog.error("SIGUSR1 handler failed", err);

+ eagerLifecycleRuntime.markGatewaySigusr1RestartHandled();

+ });

Alternatives considered

I considered three alternatives in the PR before settling on this:

Option A

Replace SIGUSR1 with process.exit(0) on package swap — too aggressive, loses in-process drain, and is an orthogonal concern. It can stack on top of this fix but doesn’t replace it.

Option B

Make update.run always go through launchd kickstart — treats only the symptom. Config watcher, manual SIGUSR1, and other paths still hit the same bug.

Option C

Wrap the dynamic import in try/catch inside the listener — only fixes one callsite. run-loop.ts has ~10 lifecycle dynamic-import callsites; one eager-load fixes them all.

Verification

After a local build I triggered the previously-deadlocked path with kill -USR1 <pid>:

$ kill -USR1 47154
$ tail -n 8 /tmp/pr-gw-test.log
2026-05-21T19:52:27.296+08:00 [gateway] signal SIGUSR1 received
2026-05-21T19:52:27.310+08:00 [gateway] received SIGUSR1; restarting
2026-05-21T19:52:27.335+08:00 [shutdown] started: gateway restarting
2026-05-21T19:52:28.274+08:00 [gmail-watcher] gmail watcher stopped
2026-05-21T19:52:28.279+08:00 [shutdown] completed cleanly in 943ms
2026-05-21T19:52:28.285+08:00 [gateway] restart mode: in-process restart
      terminal output after the patch

Before: signal SIGUSR1 received appears once, then total silence — every restart returns coalesced: true. After: the full shutdown sequence runs and exits cleanly in 943ms.

I also ran the five related vitest suites:

$ pnpm exec vitest run src/cli/gateway-cli/run-loop.test.ts \
  src/infra/restart.test.ts src/infra/restart-coordinator.test.ts \
  src/gateway/server-methods/update.test.ts \
  src/cli/gateway-cli/run.supervised-lock.test.ts

Test Files  5 passed (5)
    Tests  56 passed (56)

Takeaways

On ESM lazy loading

Dynamic import inside a signal handler is dangerous. Signal handlers fire at unpredictable times, and anything that touches disk can fail if the environment has changed underneath it. The recovery path shouldn’t depend on the thing it’s recovering.

On async IIFE

void (async () => {...})() without a .catch() is a quiet time bomb. Particularly dangerous in a signal handler, where failure is silent and leaves no observable trace.

On open source contribution

The most convincing PR is “this happened to me in production.” A slice of production logs plus a live terminal proof beat any unit test. ClawSweeper bot’s final grade was platinum hermit — the jump from the initial silver shellfish came from that terminal screenshot.