16.6 Server operations

16.6.1 adactl

adactl is the operations tool installed on the server (/usr/local/sbin/adactl, installation directory /opt/ada-classification by default). Run it with sudo. Every command accepts --help.

Design rules you can rely on:

• Measure first, then change. Destructive commands list what they will do and ask you to type the server’s host name; --dry-run is available on every destructive command.
• No secrets on the command line or screen. Passwords and passphrases are read from 0600 files (--password-file, --passphrase-file). Only two commands print a secret, on explicit request: admin initial-password and backup key create.
• Exit codes: 0 success · 1 error (work interrupted) · 2 work pending / findings · 3 precondition or usage error — nothing was changed.

16.6.2 Support bundles: console vs host

For a single agent, request diagnostics from the agent’s detail panel on Agents; the agent uploads a zip that the server keeps (last 3 per agent, 30 days).

16.6.3 Host watchdog

All in-product health checks and alerts run inside the API server. If the API itself dies or hangs, the host watchdog is what notices. It is a systemd timer on the host (ada-watchdog.timer, every minute), installed by the standard installation; it needs only curl and the Docker CLI.

adactl upgrade and adactl rollback pause the watchdog for the whole run and start it again on every exit path.

16.6.4 Upgrading the server

Upgrade the server before the agents. The supported path is adactl upgrade.

Before you start (about 5 minutes):

1. sudo adactl status — exit 0 means nothing pending; note the schema and rollback base lines and free disk.
2. sudo adactl verify.
3. Take an encrypted full backup: sudo adactl backup full --passphrase-file /root/backup.pass (copy it off the server).
4. Make sure a root-CA backup taken since installation or the last rotation exists off the server; if not, sudo adactl ca backup --passphrase-file /root/ca.pass.
5. Read the release notes.

Upgrade:

tar -xf ada-release-<version>.tar
sudo adactl upgrade ./ada-release-<version> --dry-run
sudo adactl upgrade ./ada-release-<version> --passphrase-file /root/backup.pass

Add --yes for unattended runs. The tool works in measured steps:

1. Preconditions: root, healthy stack, Docker Compose ≥ 2.24, free disk for the images and for the release archive.
2. The whole release package is copied to /var/lib/ada/releases/<version>/ (rollback reads it, so no external files are needed later) and /usr/local/sbin/adactl is re-pointed to the new tool.
3. Images are loaded and verified against the manifest; nothing is switched yet.
4. The database migrations the new image will apply are counted. If at least one exists, a full backup is mandatory (the --passphrase-file is required; there is no skip flag). If they cannot be counted, the upgrade stops.
5. The kit is laid over the installation; .env (except ADA_VERSION), secrets/ and agent packages are never touched.
6. Agent packages, ADA_VERSION, one up -d, health wait.

Exit 3 means nothing was changed. If anything fails, the exact rollback command for this server is printed.

Verify (about 3 minutes): sudo adactl status, sudo adactl verify, readlink -f /usr/local/sbin/adactl (must point into the new release directory), curl -sk https://127.0.0.1/healthz → {"status":"ok"}. In the console, check Status (version, DB schema, agents online) and the Audit Log chain badge.

Rollback: sudo adactl rollback [--dry-run] restores the previous version from /var/lib/ada/releases/. It first compares the database schema with the target image; if the database has migrations the older image does not know, the rollback is refused and the restore path (restore the pre-upgrade .adabk into the older version) is printed instead.

16.6.5 Verifying the audit chain from the host

The console verifies the chain on every Audit Log load. On a server without a licence (where the API returns HTTP 423), verify from inside the container:

sudo adactl compose -- exec ada-api node dist/cli-audit-verify.js --json

Exit codes: 0 chain intact · 2 chain broken (integrity finding, also appended to audit-verify-findings.log) · 1 could not be measured (fix the environment and retry — this is not an integrity finding) · 3 usage error.

16.6.6 Host hardening

sudo adactl harden measures the host against controls H-01 … H-25 (mapped to CIS Ubuntu and CIS Docker; modules such as container, apparmor, firewall, auditd, journald, sshd, apt, perms, time, docker, sysctl, db, tls, backup) and prints the fixing command for each finding. It changes nothing. Exit 2 means “there are findings”, not “the installation is broken”; on a stock Ubuntu host several warnings are host-policy decisions for your organisation.

Applying is a separate, deliberate step through deploy/harden.sh in the installation directory: