Description

After upgrading NemoClaw via curl|bash install.sh (e.g. v0.0.55 → v0.0.67), the installer auto-runs Checking for sandboxes that need upgrading... and tries to rebuild every existing sandbox to match the new agent image. The rebuild flow is DESTRUCTIVE-then-CREATE NON-ATOMIC: it backs up state, then deletes the old sandbox container and image, then tries to recreate the new one. When the recreate step uses ambient shell env (NEMOCLAW_PROVIDER_KEY, NEMOCLAW_AGENT) or stale onboard-session state from a freshly-completed unrelated onboard, that ambient context routinely does NOT match the original sandbox's recorded provider / model / credential format, and recreate fails. The sandbox is then gone — only the backup remains.

Related fixed bug: NVB 6103453 / GitHub Issue #2273 (nemoclaw rebuild is not atomic — sandbox destroyed before recreate step can fail) was verified-fixed on 2026-04-29. The earlier fix covered the nemoclaw rebuild CLI command path. The current observation is on a separate code path — the installer's auto-rebuild step that fires after a new sandbox is onboarded. The same destructive-then-non-atomic-create pattern is still present in that path.

Environment

Device:        Ubuntu Linux server, 16 vCPU / 62.5 GiB RAM
OS:            Ubuntu (cloudshqa test machine)
Architecture:  x86_64
Node.js:       v22.23.1
npm:           10.9.8
Docker:        server 29.2.1
OpenShell CLI: 0.0.44
NemoClaw:      v0.0.55 (installed) -> v0.0.67 (target via NEMOCLAW_INSTALL_TAG)
OpenClaw:      Existing sandboxes test-slack at v2026.5.27 and oc-clean at v2026.5.22 (both OpenClaw); new sandbox deep-agent created during this run.

Steps to Reproduce

1. On a host running NemoClaw v0.0.55, have at least one existing OpenClaw sandbox with provider nvidia-prod (i.e. configured with an nvapi-* key). In this repro the host had two such sandboxes: test-slack and oc-clean.

2. Export env that targets a DIFFERENT agent + provider for the upcoming onboard step:

   export NEMOCLAW_AGENT=langchain-deepagents-code
   export NEMOCLAW_PROVIDER_KEY=sk-<inference-hub-key>

3. Run the v0.0.67 installer:

   curl -fsSL .../install.sh | NEMOCLAW_INSTALL_TAG=v0.0.67 bash

4. In the wizard, pick "Other OpenAI-compatible endpoint", give the new sandbox a name (e.g. deep-agent), let onboard complete cleanly.

5. After the new sandbox banner OK LangChain Deep Agents Code terminal runtime is ready, the installer prints [INFO] Checking for sandboxes that need upgrading... and starts auto-rebuilding the existing sandboxes (test-slack, oc-clean).

6. Observe: for each existing sandbox, the rebuild step backs up state, destroys the container + image (untags, removes layers), then fails during recreate because the ambient env / onboard-session does not match the sandbox's recorded provider / model / credential format. The destroyed sandboxes are not recreated; only the snapshot backups remain.

Expected Result

Auto-rebuild should be atomic, OR perform a recreate-precheck BEFORE destroying anything. Required behavior:

• Validate the recreate config first: ensure the credentials / provider / model that the recreate step will use are compatible with the existing sandbox's recorded config. If not, ABORT the rebuild with a clear error BEFORE destroying anything.
• Each sandbox's rebuild should source its own recorded config (provider, model, API key reference) from registry state, NOT the ambient NEMOCLAW_PROVIDER_KEY env or the just-completed onboard session.
• If recreate has to happen, build the new image and verify it healthy BEFORE destroying the old container/image. Only destroy the old one after the new one is up.
• If any rebuild fails (or is going to fail by precheck), report it as a clearly-flagged fatal failure with recovery steps, not a non-fatal [WARN] tucked at the bottom of Installation complete.

Actual Result

For each existing sandbox the installer runs:

1. Backup sandbox state → /home/cloudshqa/.nemoclaw/rebuild-backups/<name>/<timestamp>
2. Deleting old sandbox... → docker image untagged + ~70 layer SHAs deleted → Old sandbox deleted (POINT OF NO RETURN)
3. Creating new sandbox with current image...
4. Recreate fails non-interactive precheck:
   • test-slack: ambient env has NEMOCLAW_PROVIDER_KEY=sk-* (Inference Hub key, set for the DA onboard). Non-interactive rebuild mapped test-slack to agent LangChain Deep Agents Code and provider build (NVIDIA Endpoints), which requires an nvapi-* key. Validation rejects the sk-* key. Sandbox already destroyed.
   • oc-clean: onboard-session state was reused from the deep-agent onboard run minutes earlier and recorded provider=compatible-endpoint + model=openai/openai/gpt-5.4. oc-clean's registry record said provider=nvidia-prod + model=nvidia/nemotron-3-super-120b-a12b. Precheck detected the drift, refused to recreate. Sandbox already destroyed.

Final installer line: OK 2 sandbox(es) failed -- see errors above. But the closing banner still says OK === Installation complete === and only emits [WARN] Sandbox upgrade check failed (non-fatal). — this makes the destruction easy to miss.

Recovery requires the user to run nemoclaw onboard --resume per sandbox to recreate it with correct config, then nemoclaw <name> snapshot restore <timestamp> to put workspace state back. Documented in the inline error, but easy to miss given the success-style "Installation complete" banner.

Logs

test-slack rebuild — DESTROY succeeded, RECREATE failed

Rebuild sandbox 'test-slack'
  Current:  OpenClaw v2026.5.27
  Target:   OpenClaw v2026.5.27

Note: onboard session belongs to 'deep-agent', not 'test-slack'. Using the 'test-slack' registry entry for credential preflight.
OK Active gateway set to 'nemoclaw'
Backing up sandbox state...
OK State backed up (13 directories, 1 files)
  Backup: /home/cloudshqa/.nemoclaw/rebuild-backups/test-slack/2026-06-24T10-02-11-399Z
Deleting old sandbox...
Untagged: openshell/sandbox-from:1782257594
Deleted: sha256:554edcca4e79... (~70 layer SHAs deleted)
...
Removed Docker image openshell/sandbox-from:1782257594
OK Old sandbox deleted              <-- POINT OF NO RETURN

Creating new sandbox with current image...
[non-interactive] Agent: LangChain Deep Agents Code
Agent changed from OpenClaw to LangChain Deep Agents Code; refreshing provider selection.

  NemoClaw Onboarding
  (non-interactive mode)
  (resume mode)
  [3/8] Configuring inference provider
  [non-interactive] Provider: build
  Invalid NVIDIA API key. Must start with nvapi-
  Get a key from https://build.nvidia.com/settings/api-keys

Recreate failed after sandbox was destroyed.
Backup is preserved at: /home/cloudshqa/.nemoclaw/rebuild-backups/test-slack/2026-06-24T10-02-11-399Z

To recover manually:
  1. Fix the issue above (missing credential, Docker problem, etc.)
  2. Run: nemoclaw onboard --resume
     This will recreate sandbox 'test-slack'.
  3. Then restore your workspace state:
     nemoclaw test-slack snapshot restore "2026-06-24T10-02-11-399Z"

WARN Failed to rebuild 'test-slack': Recreate failed (sandbox destroyed).

oc-clean rebuild — DESTROY succeeded, RECREATE failed

Rebuild sandbox 'oc-clean'
  Current:  OpenClaw v2026.5.22
  Target:   OpenClaw v2026.5.27

Backing up sandbox state...
OK State backed up (13 directories, 1 files)
  Backup: /home/cloudshqa/.nemoclaw/rebuild-backups/oc-clean/2026-06-24T10-02-54-807Z
Deleting old sandbox...
Untagged: openshell/sandbox-from:1782293828
Deleted: sha256:5b6b4aaaee... (~70 layer SHAs deleted)
...
Removed Docker image openshell/sandbox-from:1782293828
OK Old sandbox deleted              <-- POINT OF NO RETURN

Creating new sandbox with current image...
Resumable state recorded provider 'nvidia-prod', not 'compatible-endpoint'.
Resumable state recorded model 'nvidia/nemotron-3-super-120b-a12b', not 'nvidia/nvidia/nemotron-3-super-v3'.
Run: nemoclaw onboard              # start a fresh onboarding session
Or rerun with the original settings to continue that session.

Recreate failed after sandbox was destroyed.
Backup is preserved at: /home/cloudshqa/.nemoclaw/rebuild-backups/oc-clean/2026-06-24T10-02-54-807Z

WARN Failed to rebuild 'oc-clean': Recreate failed (sandbox destroyed).

Installer summary (note the misleading "complete" banner)

WARN 2 sandbox(es) failed -- see errors above.
[WARN]  Sandbox upgrade check failed (non-fatal).
[INFO]  === Installation complete ===

  NemoClaw  (421s)
  Your OpenClaw Sandbox is live.
  ...

The user is told "Installation complete" but TWO of their existing sandboxes have been destroyed and the backup-only restoration path is buried above in the [WARN] block. Easy to miss the destruction.

Root-cause pattern: the rebuild's recreate step pulls from

• (a) ambient shell env (NEMOCLAW_PROVIDER_KEY, NEMOCLAW_AGENT)
• (b) the most-recent onboard-session state on disk

instead of from the destroyed sandbox's own recorded config. Cross-contamination from the DA onboard (sk-* key, compatible-endpoint provider, openai/openai/gpt-5.4 model) was applied to unrelated openclaw sandboxes that originally used nvapi-* + nvidia-prod + nemotron-3-super-120b-a12b.

NVB#6370188