Ansible/scripts/addons/webgui/ADDON-INSTALLATION.md

# AIM WebGUI Add-on Installation

**Applies to:** AIM WebGUI 2.1.0rc9  
**Required AIM Core:** 3.3.0rc8  
**Core service / wire / event API:** 1.0  
**Canonical Ansible Core:** 2.19.11, provided by the separately installed Core environment  
**WebGUI HTTP API / database:** v2 / SQLite schema 5  
**WebGUI Python:** Python 3.11+ with virtual-environment support  
**Managed deployment:** Linux with systemd  
**Prerequisite:** AIM Core 3.3.0rc8 `scripts/docs/INSTALLATION.md` (separately supplied Core guide)

This guide continues after the Core installation guide. It installs a **new AIM WebGUI add-on on an already configured AIM controller**; it does not install AIM Core again. Complete Core installation, establish approved customer data, and accept a controlled terminal run before enabling WebGUI execution. Core remains usable without this add-on. [C1][C2]

For an existing managed WebGUI installation, use [section 15](#15-updating-an-existing-add-on), not the fresh-install command. For a controller already running this exact add-on version, use the verification and acceptance sections; the deployer rejects a same-version reinstall. [W2]

**Reading order:** fresh installations follow sections 1-13; backup, updates and recovery are in sections 14-17. The completion checklist is at the end.

The commands below use `/etc/ansible`, the existing executor account `svc_bf-ansible`, and the default web account `aim-web`. These are the current release profile, not requirements to rename an established controller account. This document describes a release candidate and a source-checked procedure, not evidence that a new controller has passed live acceptance.

## 1. Complete the Core prerequisite first

Use Core's `INSTALLATION.md` for the source installation, dependency environments, canonical Ansible runtime, collections, controller configuration, inventories, Vaults and keys. Do not substitute the WebGUI virtual environment for either the AIM or Ansible environment. [C1, sections 1-12]

Before continuing, establish:

- `aim` and `aimctl` refer to the intended Core 3.3.0rc8 installation.
- The AIM Python environment contains its declared dependencies, including the documented `rich>=13,<15` range.
- The separately configured Ansible runtime contains exactly Core 2.19.11 and the approved collection/connection dependencies needed by the selected playbooks.
- Core 3.3.0rc8 requires `ansible.windows >=3.8.0,<4.0.0` for Windows playbooks. Upgrade that collection deliberately in the canonical Core/Ansible collection path before accepting rc8; WebGUI does not install or upgrade it.
- At least one approved test customer/host is available for the terminal and add-on acceptance run.
- The non-root execution account is authorized by Core and has the required filesystem access.
- Terminal AIM has completed an approved low-risk test independently of WebGUI.

Check the installed commands:

```bash
command -v aim
command -v aimctl
aim --version
aimctl --version
```

Both Core commands must report `3.3.0rc8`. A generic API version of `1.0` alone is not enough: this WebGUI candidate also checks its explicitly supported Core product version and capabilities. [W2][W3]

For a fresh setup, leave this existing Core setting disabled while provisioning the add-on:

```yaml
addons:
  execution_enabled: false
```

Merge configuration changes into `/etc/ansible/scripts/aim.yml`; do not replace the entire file. The final opt-in is covered in section 10. Discovery and installation of the add-on do not grant permission to execute jobs. [C1, sections 8 and 13]

## 2. Confirm the two local service identities

The managed topology uses two different non-root accounts: [W2][W4]

| Component | Default account | Responsibility |
|---|---|---|
| `aim-web.service` | `aim-web` | HTTP, sessions, UI and private workflow database |
| `aim-web-worker.service` | `aim-web` | Queue, authorization rechecks, journal/report persistence and credential handoff |
| `aim-web-executor.service` | `svc_bf-ansible` | Fixed public `aimctl` calls and Core-owned native execution under that same UID |

The WebGUI installer can create the local `aim-web` system account and group when absent. It **does not create the Core executor account or change its persistent group memberships**. The executor must already exist, must not be root, and must differ from the web account. [W2]

Inspect the approved executor:

```bash
getent passwd svc_bf-ansible
id svc_bf-ansible
```

The passwd/NSS home must be a reviewed, usable account home. The installer will create or normalize its `.ansible` and `.ansible/tmp` directories to executor-owned `0700`, in addition to the separate `/var/lib/aim-web-executor` tree. Review pre-existing shared uses or symlinks before applying; do not use installation as a general home-directory repair procedure. [W2][W4]

Review Core's configured `required_group` and the account's active local/NSS memberships. Do not assume the example directory group in the Core installation guide or the local `aim-operators` group exists at every site. Provision authorization through the approved local/directory administration process, not through an add-on workaround. [C1][C2]

The account needs access to the configured Core launcher/runtime, customer inventory and encrypted Vault, required role payloads and collections, and Core's cooperating customer-lock paths. In customer-key mode, the canonical private key must be an owner-only `0600` file owned by the execution UID. A key filename matching the account name does not establish ownership. [C2][C3]

`service_user` is not a local UID switch. `runtime.private_key_owner` controls newly generated keys; it does not migrate an existing root-owned key or grant access. Existing-key migration is a separate, explicit Core/operator decision. Do not recursively change `/etc/ansible`, make private keys group-readable, or add the web account to root/operator groups to bypass this prerequisite. [C1, section 8][C2]

The managed executor preserves its native primary group and receives the `aim-web` group for this service only. A normal interactive `sudo -u svc_bf-ansible` session therefore need not be able to read WebGUI's `root:aim-web 0640` configuration. That is not a reason to make it world-readable. [W4]

## 3. Verify Core access as the executor

First check metadata through the installed public launcher:

```bash
sudo -u svc_bf-ansible \
  /usr/local/bin/aimctl \
  --config /etc/ansible/scripts/aim.yml \
  capabilities
```

Then check a protected operation:

```bash
printf '%s\n' '{"api_version":"1.0","operation":"list_customers"}' \
  | sudo -u svc_bf-ansible \
      /usr/local/bin/aimctl \
      --config /etc/ansible/scripts/aim.yml \
      request
```

`capabilities` is unguarded metadata; success there does not prove authorization. The customer-list response must also succeed. An empty list can be a valid discovery result, but it is not a completed managed-host acceptance test. [C3]

The WebGUI integration expects the detailed progress, inventory hierarchy, target outcome, dual-home staging and operation-result contracts described by the current Core documentation. The installer probes live Core capabilities and protected customer discovery; do not edit a support snapshot or disable version checks to make an incompatible pair install. [C2][W2][W3]

### Confirm runtime and collection visibility

Use the executable configured in Core's `runtime.ansible_playbook`. For the separate environment shown in Core's fresh-install guide, the checks are:

```bash
sudo -u svc_bf-ansible \
  /opt/ansible/venv/bin/ansible-playbook --version

sudo -u svc_bf-ansible \
  /opt/ansible/venv/bin/ansible-galaxy collection list
  /opt/ansible/venv/bin/ansible-galaxy collection list ansible.windows
```

If Core was accepted with `/usr/bin/ansible-playbook`, use its corresponding runtime instead; do not create a replacement just for WebGUI. Sibling Vault/Galaxy tools and collection discovery must match the approved native runtime. [C1, sections 4-5][C3]

For rc8, confirm the discovered `ansible.windows` version is at least 3.8.0 and below 4.0.0. Core advertises this floor in `capabilities.collection_baselines` and its readiness checks reject older versions before an affected playbook launches. Do not satisfy the check by installing a private collection visible only to root or to the WebGUI virtual environment.

The collection-install command in the Core guide uses `sudo`. Verify discovery under the executor as well as root. Collections available only in root's private home do not establish executor access. Use Core's documented `runtime.ansible_collections_path` when an approved shared/nonstandard collection location is needed; WebGUI does not install collections or expose root's home. [C1, section 5][C3]

These shell checks do not reproduce systemd restrictions. The installed executor's startup preflight and the real one-host test are still required.

## 4. Review HTTPS, SSH trust and the shipped site profile

### HTTPS and reverse proxy are prerequisites

The WebGUI deployer does not install Nginx, Nginx Proxy Manager, certificates, CA trust or DNS. It serves HTTP on its configured backend listener; the approved reverse proxy provides the browser's HTTPS endpoint. [W1][W2]

The bundled `deploy/webgui.example.toml` contains this **controller-specific** profile:

| Setting | Shipped value |
|---|---|
| Backend | `127.0.0.1:8080` |
| Browser origin | `https://aim.desq-gaming.de` |
| Immediate trusted proxy | `127.0.0.1` |
| Referenced upstream topology | NPM `192.168.20.3` -> verified HTTPS controller `192.168.20.46:8443` -> local Nginx -> loopback HTTP |
| WebGUI execution / credential entry | Enabled |
| Independent approval | Disabled |
| Eligible playbooks | The 14 explicit catalog keys in the shipped profile |
| Maximum requested hosts / execution timeout | 25 / 1,800 seconds |
| `transport_verified` | `true`, representing this site's operator attestation |

An allowlist does not establish that every playbook is executable or appropriate at the site. `transport_verified=true` does not provision or test TLS. Review the actual proxy path and trust before using these values. [W1][W5]

The `public_url` must be the exact browser origin: scheme, hostname and optional port, with no path or trailing slash. The trusted-proxy list identifies the immediate backend proxy, not the browser's remote address. Execution rejects wildcard proxy trust. [W5]

`deploy/nginx.example.conf` is a header reference, not an installable complete TLS server configuration. Preserve the browser Host and forwarded scheme through the approved proxy chain and verify streaming. WebGUI uses authenticated SSE, not WebSockets. No certificate setup or unverified replacement proxy configuration is supplied by this guide. [W1][W5][W9]

### Other sites: an explicit configuration gate

Do not treat the shipped domain, IP addresses or transport attestation as generic defaults.

The current deployer can select the Core launcher/configuration and account names, but **has no `--profile`, `--config-file` or public-origin override argument**. It replaces the whole installed TOML from its release template during install/update and restores the checkpoint's configuration on rollback. Editing the extracted template invalidates its manifest. [W2]

For repeatable release-managed deployment, obtain a reviewed release profile matching the new site. An operator can edit the installed `/etc/ansible/scripts/config/webgui.toml` before exposing it, but that is a **local override, not a durable site-profile mechanism**; the next managed update replaces it. Keep Core execution disabled and public ingress closed during any such adaptation. Preserve the managed state/socket paths and review account/path choices rather than copying a complete unrelated configuration.

If credentials or execution are disabled locally, disable `[credentials].enabled` before or together with `[execution].enabled`. The configuration validator does not allow credential entry without enabled execution and verified HTTPS. Run `sudo aim-web config-check` after editing; it validates settings, not certificates. Configuration is loaded by the service processes: apply edits in a quiesced window and restart the appropriate services before exposing the site. Leave the worker stopped/disabled when execution is intentionally disabled. [W5]

### SSH host trust is a separate prerequisite

For Linux/SSH targets, the executor needs independently verified host trust for the effective connection destination. The inventory's logical hostname may differ from the actual `ansible_host` address.

The existing approved controller-wide `/etc/ssh/ssh_known_hosts` arrangement can be retained. WebGUI does not enroll keys, overwrite that file or automatically copy root's personal trust decisions. Do not infer OpenSSH's effective user known-hosts path solely from the executor's substituted `HOME`. Review the effective SSH configuration and account context instead. [W4]

Windows WinRM connectivity, credentials, transport dependencies and certificate policy remain the independently accepted Core/operator configuration. Do not disable TLS or SSH verification to pass add-on acceptance. [C1][C3]

## 5. Obtain and verify the WebGUI release

Transfer both files through a trusted channel:

```text
AIM-WebGUI-2.1.0rc9.zip
AIM-WebGUI-2.1.0rc9.zip.sha256
```

Stage outside the active add-on and Core source directories:

```bash
cd /var/tmp
sha256sum -c AIM-WebGUI-2.1.0rc9.zip.sha256
unzip AIM-WebGUI-2.1.0rc9.zip
cd aim-web-2.1.0rc9
sha256sum -c MANIFEST.sha256
```

The expected ZIP SHA-256 is:

```text
bf0a6977d2672477bcb48c78b49085ff79e70195e780813b934d87ad334d7556
```

Use a new extraction directory, not a directory containing an older release. Never extract over `/etc/ansible/scripts/addons/webgui` or `/opt/aim-web/current`. A checksum verifies integrity, not publisher identity. The deployer additionally verifies manifest hashes, safe paths and required-file coverage. [W1][W2]

Do not edit manifest-covered files, remove integrity checks or regenerate a manifest just to bypass a deployment error. This documentation companion is not an updated release ZIP and should be kept outside the verified extraction.

## 6. Prepare WebGUI dependencies and inspect the deployer

The controller must provide Linux/systemd, Python 3.11+ with `venv` support and administrative permission to perform installation. Inspect the local Python before running the installer:

```bash
python3 --version
python3 -m venv --help
python3 deploy/deploy.py --help
```

The WebGUI deployer creates its **own versioned virtual environment** and installs the declared WebGUI/build dependencies using the supplied `constraints.txt`. It does not run pip in AIM's environment, install Ansible, or install the test dependency extra. There is no need to pre-install FastAPI or pytest globally. [W2]

If the deployment interpreter is not the Python intended for WebGUI, use the supported `--python /absolute/path/to/python3` option. Do not point it at an interpreter with an unsupported version. Separate environments can use the same Python installation without sharing package environments.

The package is not a complete offline dependency bundle. A new installation needs approved package/asset access or prepared offline artifacts. Browser assets are the exact pinned Bootstrap 5.3.8 CSS and HTMX 2.0.10 JavaScript; the installer verifies their SHA-384 pins and serves them locally. Do not substitute fixture assets or a newer version. [W1][W8]

For an offline installation, prepare wheels compatible with the chosen Python/OS/architecture, including constrained build dependencies, and an asset directory containing:

```text
bootstrap.min.css
htmx.min.js
```

Pass the existing `--wheelhouse` and `--assets-dir` options in section 7. The detailed wheelhouse preparation depends on the operator's build environment; the supplied guide does not establish a universal offline-build recipe. [W1][W2]

### Core and WebGUI deployers have different commands

| Operation | Core guide | WebGUI 2.1.0rc9 |
|---|---|---|
| Preview | `install --dry-run` | No equivalent dry-run mode |
| Fresh apply | `install --apply --quiesced` | `install` performs installation |
| Existing add-on update | Not applicable | `update` |
| Rollback selector | `--from-backup /path` | `--backup SNAPSHOT_ID` |

Do **not** append Core's `--dry-run`, `--apply`, `--quiesced` or `--aim-python` options to the WebGUI deployer. Review its help, profile and source/documentation before applying. Its internal checks are not a whole-system no-write preview: preparatory add-on directories, dependencies or the web account may be created before a later failure. [C1][W2]

An existing unrecognized installation, unrelated CLI link or unmanaged systemd drop-in stops deployment. Review and back up such files; do not delete overrides or disable guard checks blindly.

## 7. Apply a fresh add-on installation

Proceed only after the preceding Core, identity, profile and transport gates are satisfied. Do not run a Core source update or change its inventory/configuration concurrently.

From the verified fresh extraction:

```bash
sudo python3 deploy/deploy.py install \
  --aim-scripts /etc/ansible/scripts \
  --service-user aim-web \
  --executor-user svc_bf-ansible \
  --aimctl /usr/local/bin/aimctl \
  --core-config /etc/ansible/scripts/aim.yml
```

These options make the default binding explicit. The Core account and launcher must already exist. [W2]

For the same installation using prepared offline artifacts, add:

```bash
sudo python3 deploy/deploy.py install \
  --aim-scripts /etc/ansible/scripts \
  --service-user aim-web \
  --executor-user svc_bf-ansible \
  --aimctl /usr/local/bin/aimctl \
  --core-config /etc/ansible/scripts/aim.yml \
  --wheelhouse /secure/wheelhouse \
  --assets-dir /secure/aim-assets
```

The successful fresh-install sequence: [W2]

1. Verify the release, stage dependencies/assets and probe public Core metadata as the executor.
2. Create the web account if needed; create private WebGUI state and a protected deployment checkpoint.
3. Install the reviewed TOML and provision the managed executor staging directories.
4. Initialize schema 5 and the initial local administrator without resetting existing initialized accounts.
5. Activate the full source/virtualenv and the `aim-web` launcher; write the three managed service units.
6. Run executor startup checks, wait for its correctly permissioned socket, start HTTP and its local readiness check, and start the worker when the WebGUI execution setting is enabled.

The deployer does not change Core's `addons.execution_enabled`, Core source, customer keys/Vaults, persistent executor group memberships, SSH trust or proxy certificates. The shipped profile starts the queue worker even when Core's independent execution opt-in is still false; that does not bypass the Core gate.

Record the printed add-on source, configuration, checkpoint identifier and initial-credential location. They are needed for verification and recovery.

### Nondefault layouts

`--aim-scripts`, `--aimctl` and `--core-config` can bind the add-on to reviewed locations. They do not create an independent second WebGUI instance: `/opt/aim-web`, state paths, service names and the executor socket are fixed in this managed profile. The executor's inventory write exception also names `/etc/ansible/inventories`. A different inventory root or parallel installation needs a reviewed deployment profile; changing the CLI paths alone is insufficient. [W2]

## 8. Verify the installed services and permissions

Refresh command discovery:

```bash
hash -r
command -v aim-web
aim-web --version
sudo aim-web config-check
```

Expected version:

```text
AIM WebGUI 2.1.0rc9 (AIM compatibility: 3.3.0rc8 / service API 1.0)
```

The installer supplies `/usr/local/bin/aim-web -> /opt/aim-web/current/bin/aim-web`; do not manually copy a versioned launcher over it. `config-check` validates TOML only and can run as root without opening the workflow database. [W2][W6]

Check services and the actual startup preflight:

```bash
sudo systemctl status \
  aim-web-executor.service \
  aim-web.service \
  aim-web-worker.service \
  --no-pager

sudo journalctl -u aim-web-executor.service -n 80 --no-pager

systemctl show aim-web-executor.service \
  -p User -p Group -p SupplementaryGroups
```

The intended executor identity is:

```ini
User=svc_bf-ansible
Group=svc_bf-ansible
SupplementaryGroups=aim-web
```

Numeric group IDs in `systemctl show` are acceptable when they resolve to the intended groups. Systemd may also initialize other account memberships; an empty explicit `SupplementaryGroups` field is not proof that the process has no supplementary groups. [W4]

Inspect the managed paths, substituting the actual passwd/NSS home if different:

```bash
sudo stat -c '%U:%G %a %n' \
  /var/lib/aim/webgui \
  /var/lib/aim/webgui/webgui.sqlite3 \
  /var/lib/aim-web-executor \
  /var/lib/aim-web-executor/.ansible \
  /var/lib/aim-web-executor/.ansible/tmp \
  /home/svc_bf-ansible/.ansible \
  /home/svc_bf-ansible/.ansible/tmp \
  /run/aim-web-executor \
  /run/aim-web-executor/core.sock \
  /etc/ansible/scripts/config/webgui.toml
```

| Resource | Expected owner/group | Mode |
|---|---|---|
| WebGUI state directory | `aim-web:aim-web` | `0700` |
| Workflow database | `aim-web:aim-web` | `0600` |
| Executor state and process-home `.ansible/tmp` hierarchy | Executor:native primary group | `0700` |
| Executor passwd-home `.ansible` and `.ansible/tmp` | Executor:native primary group | `0700` |
| `/run/aim-web-executor` | Executor:native primary group | `0711` |
| `core.sock` | Executor:`aim-web` | `0660` |
| `webgui.toml` | `root:aim-web` | `0640` |
| Managed unit files | `root:root` | `0644` |

The runtime-directory traverse permission does not grant access to the private executor state or database. Socket permissions and peer checks govern IPC. These are checks of installer-managed state, not a recursive repair recipe. [W2][W4]

### Both staging homes matter

The executor sets `HOME=/var/lib/aim-web-executor`, but delegated local Ansible tasks can expand `~svc_bf-ansible` through the account database. The installer provisions both applicable staging paths and only their narrow write exceptions while retaining `ProtectHome=read-only`, `ProtectSystem=strict`, `NoNewPrivileges=true` and empty capabilities. [C1, section 11][W4]

`ExecStartPre` runs `aim-web core-staging-check` inside the actual executor unit. Successful journal output followed by service startup is the relevant staging evidence. A plain interactive `sudo -u svc_bf-ansible aim-web core-staging-check` can fail to read the web-group configuration because it does not reproduce the unit's group context. A root invocation tests the wrong key/staging identity. Do not weaken file permissions to make those commands pass. [W4][W6]

## 9. Open the HTTPS site and initialize administration

Verify the approved proxy endpoint serves the correct installation with trusted HTTPS. For the shipped profile, the browser origin is:

```text
https://aim.desq-gaming.de
```

Do not use a raw LAN HTTP address as the browser origin or use disabled certificate validation as acceptance. No operational secret should be entered before transport has been accepted.

Read the generated bootstrap credential file **locally**:

```bash
sudo cat /var/lib/aim/webgui/.credentials
```

The fresh administrator username is `admin`. The JSON file contains the generated password and is protected as service-owned private state; it is not served over HTTP. Do not paste it into tickets, chat, deployment logs or this documentation. [W6][W7]

Sign in and perform the mandatory first password change. The application revokes the sessions for that account and invalidates/removes the bootstrap credential file after the password change. Sign in again with the new password as prompted. A later update or `init` command does not regenerate bootstrap credentials or reset an initialized administrator. [W7]

Use User administration to establish named accounts and reviewed execution grants. An eligible administrator or a matching customer/playbook grant is required in addition to the configured allowlist and Core authorization. Job/history/report visibility is owner-or-admin; supplying one-run credentials remains requester-only. [W7][W9]

Do not enable independent approval without providing a suitable separate approver. The shipped single-administrator site profile has `require_approval=false`; that does not bypass review of targets, mode, options or Core revisions.

## 10. Accept local readiness, then enable Core execution deliberately

Through the running add-on's executor:

```bash
sudo -u aim-web \
  /opt/aim-web/current/bin/aim-web \
  --config /etc/ansible/scripts/config/webgui.toml \
  core-check
```

Confirm Core product version, live capabilities and authorized customer discovery. An `execution.enabled=false` value at this point is the expected independent Core gate. [C3][W6]

For one existing approved Linux host with customer-key mode, replace the two literal identifiers before running:

```bash
sudo -u aim-web \
  /opt/aim-web/current/bin/aim-web \
  --config /etc/ansible/scripts/config/webgui.toml \
  core-check \
  --customer CUSTOMER \
  --playbook debug_test_connection \
  --host HOST \
  --key-mode customer
```

For a Windows/native-inventory test use `--key-mode none`. This is not forced password authentication. The CLI readiness request uses check mode and performs local preparation/runtime checks; it does not submit a job, unlock Vault, verify remote credentials or prove remote connectivity. [C3][W6]

Once Core terminal, transport, executor identity and local staging/readiness have been accepted, edit **Core's existing** configuration:

```bash
sudoedit /etc/ansible/scripts/aim.yml
```

Merge the approved opt-in, without creating a duplicate `addons` block:

```yaml
addons:
  execution_enabled: true
```

Leave Core's accepted `runtime.ansible_playbook`, collection path, key-owner and other controller settings intact. Re-run `aim-web core-check` as `aim-web` and confirm that effective Core execution is now enabled. [C1, section 13][C3]

The gates remain separate:

```text
Core addons.execution_enabled
    + WebGUI execution/credential/transport policy
    + eligible playbook and host limit
    + enabled account/grant and any required approval
    + fresh reviewed Core revision
    + worker readiness and required one-run credentials
    = an eligible execution request
```

No single `true` setting proves successful or safe managed-host execution.

## 11. Run the first controlled end-to-end test

Use a small approved scope. For a reporting acceptance test, **Detected host roles** is the Core guide's suggested starting operation. [C1, section 14][C4]

In the browser:

```text
New run
  -> Select the test customer and Detected host roles
  -> Select one approved host
  -> Review options, Apply/Check mode and key handling
  -> Review the normalized scope, warnings and report contract
  -> Confirm and Run once
  -> Unlock this run when the worker is ready
```

A saved plan is optional. For an encrypted Linux customer key, choose customer-key loading; the credential modal can use the Vault-stored passphrase where Core permits it. Windows/native-inventory runs do not need a Linux customer key. Only fields requested by Core are offered. A supplied connection password is a native default, not an override of inventory credentials. [C3][W9]

The modal's acceptance message means the handoff was accepted, not that a password was verified. The empty reservation lasts five minutes, followed by the existing bounded handoff/start interval. Each new attempt needs fresh credentials; never put passwords in CLI arguments, URLs, plans, config examples or a queued record. [W9]

Check all three result dimensions:

| Dimension | What to inspect |
|---|---|
| Core verdict | Status, stage, native exit and whether remote work may have started |
| Target outcomes | Each reviewed host's authoritative final outcome and counters |
| Operation report | Schema, availability, retained content and Apply/Check meaning |

The role report should expose `host_capabilities_v1` with the eight declared booleans. A missing report is not eight false values. A successful native exit can still produce `failed / result_validation` when required output is missing or invalid. Do not rerun automatically: remote changes may already be complete. [C3][C4][W9]

For a longer approved run, close/reopen the job, reload and check from another authorized device. Confirm recorded progress survives and new events continue without replaying execution or requesting the same credentials again. Validate Reports after finalization and verify the host activity link. No total-task percentage or ETA is promised.

After the first success, use Core's `SANITY.md` and WebGUI's `docs/CONTROLLER-PILOT.md` for separately approved Windows/Linux, mixed-result, cancellation, recovery and reporting cases. Updating packages, rebooting, starting services, exporting logs or deleting files needs its own test scope. One successful role query does not qualify every playbook. [C4][W10]

## 12. Review retention and storage capacity

Schema 5 stores accounts/workflows plus job-linked progress and operation reports. The shipped limits are: [W1][W9]

```toml
[journal]
max_events = 20000
max_bytes = 8388608

[reports]
max_bytes = 16777216
retain_configuration = false
```

The progress limit retains a bounded tail, not a complete raw terminal transcript. Omissions and capture gaps are disclosed. Report limits also remain subject to Core/catalog caps. Full parsed Checkmk configuration sections are not retained by default; availability/file/redaction metadata remain with a metadata-only label. Enabling full retention is a separate operator data decision and applies to future captured reports.

Deleting a job removes its journal and report rows and its contribution to the displayed host history. A compact audit deletion event remains. Existing backups have their own retention policy; deleting a job is not physical erasure of backups or storage remnants. Only WebGUI-owned retained runs contribute to Host Activity and Insights. [W9]

Allow space for the database, journals/reports, protected backups and multiple versioned virtual environments. The sources do not specify a universal minimum disk or memory size; capacity depends on retained jobs and report sizes. Monitor it rather than treating per-job limits as a total database cap.

The full `webgui.toml` is release-managed. Local policy edits are overwritten on a later managed update unless incorporated into the next reviewed release profile. [W1]

## 13. Know the managed layout and ownership boundary

```text
/usr/local/bin/aim-web                  -> /opt/aim-web/current/bin/aim-web
/opt/aim-web/current                    -> selected versioned virtual environment
/opt/aim-web/venvs/                     versioned add-on environments
/opt/aim-web/deployment.json            active deployment metadata
/opt/aim-web/pending.json               interrupted/in-progress deployment journal
/etc/ansible/scripts/addons/webgui/     managed add-on source
/etc/ansible/scripts/config/webgui.toml managed add-on configuration
/var/lib/aim/webgui/                    private accounts, workflows and evidence DB
/var/lib/aim-web-executor/              private executor process HOME/staging
/run/aim-web-executor/core.sock         private group-accessible Core bridge
/var/backups/aim-web/                   protected add-on deployment checkpoints
```

The passwd-home `.ansible` staging directories and the three systemd units are also release-managed as described above. An existing account-home `.ansible` parent is normalized to the managed owner-only policy; review shared/custom uses of that directory before installation. The whole home is not made writable inside the service. [W2][W4]

Core source, `aim.yml`, inventories, Vaults, canonical keys, collection installation, global SSH trust and certificates remain outside the add-on installer. Core and add-on recovery data are different directories with different deployer commands. [C1][W1]

### Patch-wave behavior after Core 3.3.0rc8

The public catalog adds the Windows-only Continue patching after reboot option (catalog hint false), plus reboot message/delay. Blank controls inherit native inventory/role policy; the add-on never forces continuation on. A successful Windows wave can require another reviewed run. See [Patch waves](docs/PATCH-WAVES.md) before testing update/reboot behavior. Reports show pending/unknown, reboot-deferred and fixed per-update failure metadata without parsing raw output or creating follow-up jobs. Historical patch reports retain their recorded schema.

## 14. Back up the add-on independently

The deployer creates checkpoints during install/update. Maintain an operator backup policy in addition to those checkpoints, especially now that the database contains retained reports. Keep Core runtime/customer data backups separate. [W1]

The supported database backup command uses SQLite's backup API. Run it as the database-owning web identity, with a **new, nonexisting** destination it can write:

```bash
BACKUP_FILE="/var/lib/aim/webgui/manual-backup-$(date -u +%Y%m%dT%H%M%SZ).sqlite3"

sudo -u aim-web \
  /opt/aim-web/current/bin/aim-web \
  --config /etc/ansible/scripts/config/webgui.toml \
  db backup "$BACKUP_FILE"
```

The backup is created as `0600`. Transfer it through the approved protected backup process, preserving confidentiality, and record the matching Core/WebGUI versions and configuration. Database contents include account hashes, session/workflow data and retained operational reports. A database backup alone is not the deployer's full source/configuration recovery checkpoint. [W6][W11]

Do not use an uncoordinated live copy of `webgui.sqlite3` as a replacement for the SQLite backup operation. Do not install pytest into a production environment to perform an installation check.

## 15. Updating an existing add-on

For an existing compatible managed installation, use the **new release's** fresh extraction:

```bash
cd /var/tmp/aim-web-2.1.0rc9
sudo python3 deploy/deploy.py update
```

Use this exact command only when moving from an earlier version to 2.1.0rc9; it is not a same-version refresh. Preserve the previously selected `--aim-scripts` and account options when they differ from defaults. [W2]

When moving to Core 3.3.0rc8 from the older supported Core, finish/cancel work deliberately, stop submissions and quiesce terminal writers. Stop the add-on services before independently updating Core:

```bash
sudo systemctl stop \
  aim-web-worker.service \
  aim-web.service \
  aim-web-executor.service
```

Follow Core's own preview/apply procedure, then deploy the compatible WebGUI release. Do not expect the older adapter to accept a newly installed Core automatically. [C1][W1]

From WebGUI 2.1.0rc3, this update keeps SQLite schema 5: no new database migration or historical report rewrite. From a schema-4 baseline, the existing schema-5 migration remains (pending/queued work blocked, running work interrupted, stale reviews removed). Source/schema changes still require fresh review; do not replay an old prepared request. [W1]

Ordinary 2.x updates do not need `--migrate-core`. That flag is reserved for the documented legacy 1.x integration/state migration and recognized key-export/capability retirement. Read `docs/MIGRATION-3.1.md` before using it. Never treat it as a general permission or version-check bypass.

## 16. Recover without changing Core accidentally

### Failed or interrupted installation/update

Record the complete deployment error and printed checkpoint identifier. The deployer attempts recovery where a checkpoint exists; an incomplete recovery leaves `pending.json` to prevent ambiguous repeated installation. Keep services stopped until the failure and recovery state are understood. [W1][W2]

For a recorded interrupted deployment, use the same release's extracted deployer:

```bash
cd /var/tmp/aim-web-2.1.0rc9
sudo python3 deploy/deploy.py recover
```

`recover` is not a generic repair command: it requires the pending deployment journal. Do not delete that journal or switch the `current` symlink manually to bypass recovery.

A failed first install preserves private state for deliberate recovery/retry; it does not automatically wipe accounts or generate another admin. A completed first-install checkpoint is not a prior installed version, and the rollback command rejects it. Inspect and preserve state instead of removing the database to force bootstrap. [W2][W7]

### Roll back to a previous managed add-on

After independently backing up current state and quiescing work, use the exact checkpoint **identifier**, not Core's `--from-backup` syntax:

```bash
sudo python3 deploy/deploy.py rollback \
  --backup CHECKPOINT_ID
```

If the selected previous release requires schema 4, crossing back from schema 5 requires an explicit historical database restore:

```bash
sudo python3 deploy/deploy.py rollback \
  --backup CHECKPOINT_ID \
  --restore-auth-db
```

That can restore old password hashes and discard newer accounts, plans, jobs, journals and reports. Restored sessions are revoked. The backup ID must be one actually printed by this installation. [W1][W2]

The restored adapter is checked against the installed Core. If incompatible, restored add-on services remain stopped and disabled. Core rollback is separate; after validating a matched Core/WebGUI/database combination, explicitly enable/start the intended services. Neither rollback reverses completed remote changes, dependency installation or independent account/trust decisions.

### Disable the add-on

After active work is finished or deliberately canceled:

```bash
sudo systemctl disable --now \
  aim-web-worker.service \
  aim-web.service \
  aim-web-executor.service
```

This disables the three add-on units, not AIM terminal. It is not a data purge. There is no `uninstall` command in this deployer; retain private state and backups until a separately approved removal decision. [W2]

## 17. Troubleshooting at the correct boundary

| Symptom | First check / action |
|---|---|
| `No managed add-on installation exists` | Use `install` for a genuinely new add-on, not `update`. |
| Missing manifest, coverage or hash error | Re-verify the published ZIP and use a fresh extraction. Do not disable the verifier. |
| Unmanaged unit drop-in blocks deployment | Inspect the exact file and preserve it for review; do not blindly delete or reinstate older capability overrides. |
| Root command reports unsafe database/staging ownership | Use the correct service identity or real executor startup context. Do not transfer service state to root. |
| Manual executor command gives `PermissionError` on `webgui.toml`, but startup passed | The plain shell may lack the unit-scoped `aim-web` group. Inspect service journal and active identity. |
| Web config says execution enabled, Core says disabled | Review Core's independent `addons.execution_enabled` opt-in. |
| Missing collections or wrong native version | Fix the separately approved Core runtime/discovery as executor; do not pip-install Ansible into WebGUI. |
| Customer key rejected | Inspect the selected canonical key's owner, `0600` mode and traversal as executor. `private_key_owner` does not migrate existing keys. |
| Delegated controller task becomes `UNREACHABLE` | Inspect actual task/delegation and both staging paths inside the sandbox before blaming WinRM. |
| Linux trust failure despite root terminal success | Review effective executor SSH trust and actual destination; retain the approved global trust file. |
| Browser origin/security-token rejection | Check the exact HTTPS origin, immediate proxy/header chain, current page/session and clock. Do not disable CSRF. |
| Queued job never asks for credentials | Check worker activity, Core execution policy, approval/schedule and current job state. Only its requester gets the modal. |
| Native exit 0 but failed at `result_validation` | Inspect report availability/schema. Remote work may have completed; no automatic replay. |
| Empty old timeline/report | Pre-capture history is not reconstructed. Check the job's version and capture/retention markers. |
| `pending.json` blocks another deployment | Investigate and use the same deployer's recorded recovery path. |

Start with version/status/configuration checks, the specific job's structured facts and the relevant service journal. Do not share passwords, full inventories/Vault dumps or unrestricted configuration/report bodies in diagnostics. Local Ansible/module logging may contain operational details even though the WebGUI journal excludes raw output. [C4][W1][W9]

## Fresh add-on completion checklist

Before broader operational use confirm:

- Core's prerequisite installation and controlled terminal acceptance are complete.
- The WebGUI ZIP checksum and release manifest pass, and the current profile fits the site.
- The existing executor is independently authorized and can discover the intended Core runtime/collections.
- HTTPS/proxy trust is accepted; no placeholder origin or unverified transport attestation was reused.
- Fresh installation used `install`, or a real managed upgrade used `update`.
- All three expected services start, and executor startup checks pass inside their actual sandbox.
- Account/group, socket, configuration, state and dual-home staging match the managed policy.
- The bootstrap admin password has been changed and appropriate named accounts/grants reviewed.
- Core add-on execution was enabled only through the separate operator opt-in.
- A small approved end-to-end run has accepted credentials, native outcomes and its declared report.
- Reload/second-device progress and finished-job evidence behave correctly without replaying work.
- Report retention, protected backups, storage growth and recovery checkpoints have been reviewed.
- Candidate acceptance is recorded by actual UID/sandbox/runtime/platform; unsupported or untested cases are not claimed as passed.
- Terminal AIM remains independently usable.

## Related documentation and source basis

This companion preserves the prerequisite boundary and terminology of the supplied Core guide. Add-on-specific commands and paths were checked against the published WebGUI archive. Core's installation guide has not been edited. This companion is updated with the add-on release; it is not a claim of a performed controller installation.

| Reference | Source | What it governs |
|---|---|---|
| C1 | Core `scripts/docs/INSTALLATION.md`, AIM 3.3.0rc8 | Prerequisite Core installation, independent environments, opt-in, terminal acceptance and Core recovery |
| C2 | Core `ADDON_AGENTS.md` | Same-UID public integration, ownership and hardening boundaries |
| C3 | Core `scripts/docs/ADDON_API.md` | Authorization, discovery, readiness, request/credential/result semantics |
| C4 | Core `scripts/docs/SANITY.md`, `VALIDATION.md`, `OPERATION_RESULTS.md` | Controller acceptance, candidate evidence and report rules |
| W1 | WebGUI `docs/DEPLOYMENT.md` | Coordinated migration, retention, backups and rollback |
| W2 | WebGUI `deploy/deploy.py`, `pyproject.toml`, `constraints.txt` | Actual supported installer flags, initialization, generated units, dependency handling and fixed paths |
| W3 | WebGUI `adapters/core_v1.py`, `core/executor.py` under `src/aim_webgui/` | Qualified public Core compatibility and executor boundary |
| W4 | WebGUI `docs/PERMISSIONS-ROLLOUT.md`, `AGENTS.md` | Resource ownership, split-home staging and known-hosts cautions |
| W5 | WebGUI `deploy/webgui.example.toml`, `deploy/nginx.example.conf`, `src/aim_webgui/config.py` | Actual shipped site profile, reference headers and configuration validation |
| W6 | WebGUI `src/aim_webgui/cli.py` | Version/configuration/readiness, backup and administration commands |
| W7 | WebGUI `src/aim_webgui/auth/service.py`, `workflows.py` | Bootstrap, first password change, account/grant and job authorization |
| W8 | WebGUI `deploy/fetch_assets.py` | Exact browser-asset pins and offline input names |
| W9 | WebGUI `docs/CREDENTIALS.md`, `RUN-COMFORT.md`, `EXECUTION.md`, `JOURNAL.md`, `REPORTS.md` | Credential modal, execution lifecycle and retained-evidence behavior |
| W10 | WebGUI `docs/CONTROLLER-PILOT.md`, `docs/VERIFICATION.md` | Operator tests and recorded qualification limits |
| W11 | WebGUI `src/aim_webgui/db/store.py` | Database ownership and consistent backup behavior |

Core topic files live in the independently installed Core tree. WebGUI topic/source paths above are relative to the verified `aim-web-2.1.0rc9` extraction unless stated otherwise. The current Core archive contains `scripts/docs/INSTALLATION.md`. Read that separately installed prerequisite; no Core source or runtime is copied into this add-on.

**Evidence limit:** this document was produced from the supplied guides and inspected release code. It does not add new installation, dependency-security, proxy, systemd or managed-host test evidence. Use the two products' current validation records and complete the controller acceptance steps for the actual deployment.