Tasks we can run

Read-only check of the GPU, driver, kernel Xid and ECC lines, temperatures, disk, memory, running containers, and the Vast daemon.

Restart vastai.service. Refuses while any container is running unless the ticket sets an explicit override, then checks the service is active.

Apt-only NVIDIA driver branch update. It follows the installed flavor: the standard nvidia-driver package, or the open kernel-module package (nvidia-driver-NNN-open). Records pre-flight state, tries ordered apt simulations against a status copy with NVIDIA holds released, and keeps the first plan that does not install nvidia-prime. Apply installs the post-unhold plan when those same safety checks pass, and records whether it differed from the pre-unhold plan. The report install and remove lists are that winner only. Removing standalone nvidia-persistenced is expected when nvidia-compute-utils for the new branch replaces it. Refuses a plan that would change the host CUDA toolkit (such as cuda-toolkit-13-2) or remove a package outside the driver allowlist. Before it stops vastai or releases holds, apply installs linux-headers for the running kernel when they are missing, downloads the solved plan, and checks those debs for file conflicts apt -s cannot see. Auto-installed owners outside the never-remove list (for example libnvidia-egl-xcb1 and libnvidia-egl-xlib1) are added to the removal set. Any other conflict aborts file-conflict with nothing removed. If the real install fails after packages are removed, it restores the previous branch instead of leaving the node half-changed. It checks DKMS, reboots on a clean apply, then checks the new branch, enables nvidia-persistenced, checks persistence mode and the container GPU, holds, and the Vast daemon. dry-run, rollback, and repair are ticket modes. Rollback and repair tolerate a broken nvidia-smi and use lspci plus /proc/driver/nvidia to see the GPUs. A non-empty dpkg audit is not an abort. Vast job containers still block the run. repair_strategy=forward (the default) finishes the new branch and reboots only when DKMS shows the new module installed for the running kernel and nvidia.ko is on disk. repair_strategy=restore, or a failed forward attempt, reinstalls the previous branch from the network, rebuilds DKMS, and does not reboot. Repair unholds NVIDIA driver holds before apt changes (nvidia-docker2 stays held) and puts them back on success and on failure. Apt failures include failed_step and the apt transcript in the JSON report. That report is also copied to /var/lib/fillrate-ops so the fillrate user can read abort_reason, failed_step, and the apt transcript. The state directory stays root-only.

Read-only report after an NVIDIA driver update. Prints the post-reboot unit journal for the current boot (last 80 lines) and systemctl status, lists /var/lib/fillrate-runner/nvidia-driver-update, and prints the last 80 lines of each text, JSON, and facts file there. Files named like ticket, key, nonce, signature, or secret are skipped, and lines with that material are left out. Reports whether pending-post exists. Then apt-mark showhold, dpkg --audit, dkms status, nvidia-smi, and systemctl is-active for nvidia-persistenced, vastai, and docker. The container check is timeout 120 docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi, after recording whether that image was already present. That run may pull the image. It does not otherwise change the machine.

Repair Docker --gpus when the NVIDIA container runtime mode is cdi. Refuses while any container is running, when nvidia-smi fails, or when the ticket expiry has passed. Prints the toolkit version, nvidia-ctk version, runtime config, daemon.json, docker runtimes, and the current --gpus failure. Backs up config.toml and daemon.json, sets the runtime mode to legacy, and registers the docker nvidia runtime only when it is missing. Does not make that runtime the default unless it already was. Restarts docker only when daemon.json changed, and restarts vastai.service only if it had been active and then stopped. Restores the backups if --gpus fails. When --gpus succeeds, disables and removes the failed driver-update post-reboot unit and deletes the pending-post marker. Does not install or upgrade packages and does not change holds.

One-time setup for read-only host diagnostics. Adds the fillrate user to the adm and systemd-journal groups (no sudo change), installs and enables rasdaemon, gives the adm group read access to the rasdaemon event DB (tmpfiles.d plus a rasdaemon drop-in, so it survives restarts and DB recreation), tries at most one EDAC driver and only one the kernel modalias tables match to this hardware (persisted in /etc/modules-load.d/fillrate-edac.conf only if it registers a memory controller; otherwise unloaded again), and unloads unbound EDAC modules it does not match (for example igen6_edac on a server CPU), and installs ipmitool only when an IPMI/BMC device is present. Before apt changes anything, apt-get -s must show only new installs plus revision-only upgrades of dependencies (same epoch and upstream version, only the Debian/Ubuntu revision changes, each logged as allow-upgrade), no removals, and nothing in nvidia*, libnvidia*, cuda*, docker*, containerd*, vast*, linux-image/modules/headers, libc6/libc-bin/locales, or systemd/udev. Refuses while apt or dpkg is busy or after the ticket expiry. needrestart runs in list-only mode. Never restarts docker, vastai, or driver services and never reboots; running containers are left alone. Idempotent. Ticket mode dry-run runs the checks and the simulation and changes nothing. Verifies that fillrate can read journalctl -k and ras-mc-ctl --summary. Every step is logged to stdout and to /var/lib/fillrate-runner/diag-access-setup.

Heavier diagnostics for an empty, host-approved maintenance window. Refuses while any container is running, while apt or dpkg is busy, or after the ticket expiry. Installs smartmontools, nvme-cli, lm-sensors, and kdump-tools, plus mcelog only on Intel CPUs when apt has a candidate. apt runs only when apt-get -s shows new installs plus logged revision-only dependency upgrades, no removals, and nothing NVIDIA, CUDA, Docker, containerd, Vast, kernel, core libc, or systemd. Sets USE_KDUMP=1, reserves crashkernel memory in /etc/default/grub.d/kdump-tools.cfg, runs update-grub, enables kdump-tools, installs a one-shot verify unit, re-checks that no container started, and reboots once. Ticket mode is required: dry-run runs the checks and the apt simulation and changes nothing; only apply reboots. No reboot when kdump is already armed. After boot the verify unit checks that kdump-config status reports ready to kdump and that the tools are present, writes the result to its journal and /var/lib/fillrate-runner/diag-tools-extended/kdump-status, then removes itself. No docker, vastai, or driver restarts. Every step is logged.

Clear stale per-container CDI specs that the Vast daemon wrote from an old GPU list (for example after a board swap moves GPUs to new PCI slots), the cause of "failed to inject CDI devices: unresolvable CDI devices". Refuses when nvidia-smi fails, after the ticket expiry, or while a long-running container (up 120 s or more, treated as a real rental) is running; younger containers are treated as the Vast daemon's retry loop. Records every existing container ID. Stops vastai.service (which stops the retries), waits for running containers to exit (up to 90 s; if one stays running or reaches 120 s it starts vastai again and refuses; it never stops a container itself), copies every /etc/cdi/D.*.yaml to /var/lib/fillrate/backup/cdi-spec-refresh-<stamp>, deletes each D.<id>.yaml whose id is not an existing container (a stopped container's spec is kept), starts vastai.service, and waits at least 60 seconds. Verifies that kaalia.log since the start reports the same GPU count as nvidia-smi, that any D spec written since the start lists every nvidia-smi UUID on the /dev/nvidiaN matching its minor number, and that nvidia-ctk cdi list shows nvidia.com/gpu=0..N-1 and all. Ends with read-only diagnostics: ls -la /var/lib/vastai_kaalia, any file there that looks like a GPU-list cache or lists GPU UUIDs (only its size, mtime, UUIDs and PCI bus ids are printed; .ssh, .docker, .cache, tls, data, docker_tmp and metrics_spool are skipped), the last kaalia.log lines naming GPU UUIDs, and the tail of purge_stale_cdi.log. Never stops, starts, or removes containers or volumes, never restarts docker, makes no driver or package changes, never reboots, and makes no Vast API calls. Ticket mode dry-run runs the checks and prints the plan and changes nothing. Exit 2 means a verify check failed.

Strictly read-only kernel and GPU log collection, safe while rentals run. Prints dmesg -T and journalctl -k --since "03:00 today" filtered for Xid, NVRM, nvidia, pcie, AER, MCE, and hung; nvidia-smi -q -d ECC,PAGE_RETIREMENT,ROW_REMAPPER,PERFORMANCE (ECC counts, retired pages, row remapping, throttle reasons); and the timestamps of kaalia.log lines for failed nvidia_smi polls (GPUs not found, bad nvidia_smi, on_read error). Each section keeps the newest 2000 lines, total output is capped at 400000 bytes, and every command runs under a 60 s timeout. Refuses after the ticket expiry. Writes no files (stdout only, captured by the runner log), restarts nothing, installs nothing, and never touches containers, docker, vastai, or GPU settings.

Rebuild the Vast daemon's GPU UUID-to-index map after a hardware change (board swap, GPUs moved slots, GPU replaced), when new containers fail with "unresolvable CDI devices" because /var/lib/vastai_kaalia/data/uuid-idx-mapping/HEAD still lists the old GPUs. Ticket mode required. Refuses when nvidia-smi fails, after the ticket expiry, or while a container up 120 s or more (a real rental) is running; younger containers are treated as the daemon's retry loop. Builds the map from nvidia-smi --query-gpu=index,uuid in kaalia's format (compact JSON {"GPU-<uuid>":<index>,...} in index order, no trailing newline) and shows the diff against the current HEAD. dry-run stops there. apply stops vastai.service, waits up to 90 s for running containers to exit (if one stays or reaches 120 s it starts vastai again and refuses; it never stops a container), backs up the uuid-idx-mapping directory and every /etc/cdi/D.*.yaml to /var/lib/fillrate/backup/gpu-map-rebuild-<stamp>, writes <unix-ns>-fillrate with the directory's owner and mode 644, swaps the HEAD symlink atomically (old map files are kept), and verifies HEAD index by index against nvidia-smi, putting the old HEAD back on a mismatch. Deletes D specs whose container no longer exists (specs of existing containers, including a stopped rental's, are kept), starts vastai.service, and verifies kaalia.log reports the nvidia-smi GPU count and nvidia_smi.json uuid_idx_mapping matches HEAD index by index. Exit 2 means a verify check failed. Never stops, starts, or removes containers or volumes, never restarts docker, makes no driver or package changes, never reboots, and makes no Vast API calls.

Turn off SSH password login, which Vast verification now fails on, with a password exception for an allowlist of source IPs. The allowlist is not in the script: it comes from the signed ticket line allow_ips (1 to 8 IPv4 addresses joined by "_"), which the runner passes as FILLRATE_ALLOW_IPS. Ticket mode required (apply or dry-run). Refuses after the ticket expiry, without a valid allowlist, or unless at least one admin login other than fillrate (sudo, admin or wheel member; root only when sshd allows root login) has a non-empty authorized_keys with a key line. apply backs up sshd_config and every sshd_config.d/*.conf with cp -n to .orig plus a stamped copy, and saves ufw status, iptables-save and nft list ruleset, under /var/lib/fillrate/backup/ssh-password-off-<stamp>. Sets PasswordAuthentication no in sshd_config and every drop-in, adds it before the first Match if the main file has none, and appends one marked Match Address <allowlist> block with PasswordAuthentication yes at the end of sshd_config (replacing an earlier one). Runs sshd -t (restores and exits 2 on failure), then systemctl reload ssh, never restart. Verifies sshd -T reports passwordauthentication no, sshd -T -C addr=<ip>,user=x,host=x reports yes for every allowlisted ip and no for an outside address; otherwise restores, reloads and exits 2. Firewall is add-only, tcp to the sshd port(s) from each allowlisted ip: ufw allow when ufw is active; ufw installed but inactive is reported and never enabled; otherwise iptables -I INPUT 1 with comment fillrate-ssh-allow (checked with -C first); nft insert into inet filter input only when no iptables exists and that chain does. Never changes a policy, never adds a deny, never deletes or saves rules, never touches FORWARD, Docker or Vast port rules. iptables rules last until reboot. Prints a rollback. Never touches containers, docker or vastai, never restarts sshd, installs nothing, never reboots, makes no Vast API calls. Hosts need the runner build that accepts allow_ips.

Not enabled. Needs an approved driver-version allowlist before this task is written.