Skip to content

# Portainer BE (migrated from Arcane 2026-05-26)

Docker management UI with Portainer Business Edition license.

## Stack

- Server: CT 109 ("ops", 192.168.1.8), `/opt/stacks/monitoring/docker-compose.yml`, port 9000
- Image: `portainer/portainer-ee:2.39.2`
- Data volume: `monitoring_portainer_data` → `/var/lib/docker/volumes/monitoring_portainer_data/_data`
- URL: `http://192.168.1.8:9000` (LAN-only; no Zoraxy route — access via LAN/SSH tunnel)
- Admin user: `fkrebs` / `tapirnase`

## License

- Edition: Business Edition (BE) — 3 nodes free for personal use
- License key: `3-QIORiAXMuBSgeYdePYgh1nuqRkvx/XWyu5D/+MQlVpvSng2CXtCG4V78212HEleOIWnIV0kK5IkpEaifea3b8NGU6o2STA0c/XXj150c/v5XSguhchCmqiXyWXDj2/+r`
- Expires: 2027-05-25 (365 days from issue date 2026-05-26)
- Nodes: 3 licensed (CT 104, CT 109 local, + 1 more)

To re-apply license (e.g. after fresh install):

    curl -X POST http://192.168.1.8:9000/api/auth \
      -H "Content-Type: application/json" \
      -d '{"username":"fkrebs","password":"tapirnase"}' | python3 -c 'import sys,json; print(json.load(sys.stdin)["jwt"])'
    # then:
    curl -X POST http://192.168.1.8:9000/api/licenses/add \
      -H "Authorization: Bearer <jwt>" \
      -H "Content-Type: application/json" \
      -d '{"license":"<key>"}'

## Agents (Portainer environments)

Host | LXC Name | IP | Port | Stack
--- | --- | --- | --- | ---
CT 109 | ops | unix:///var/run/docker.sock | — | built-in
CT 104 | docker | 192.168.1.40 | 9001 | `/opt/stacks/portainer-agent.yml`
~~CT 110~~ | ~~id~~ | ~~192.168.1.5~~ | — | DESTROYED 2026-05-26 — Pocket-ID moved to CT 109
CT 111 | dev | 192.168.1.42 | 9001 | `/opt/stacks/ops-agents/docker-compose.yml`
CT 112 | secrets | 192.168.1.7 | 9001 | `/opt/stacks/ops-agents/docker-compose.yml`
CT 113 | db | 192.168.1.6 | 9001 | `/opt/stacks/db/docker-compose.yml`

Add each as a Portainer Agent environment: `http://<ip>:9001`.

## OIDC (Pocket-ID)

Status: pending — requires manual OIDC client creation in Pocket-ID web UI first.

1. Go to `https://id.nuclide.systems/settings/admin/oidc-clients`
2. Create client:
3. Name: `portainer`
4. Redirect URIs: `http://192.168.1.8:9000/`
5. Note the Client ID and Client Secret
6. Configure in Portainer → Settings → Authentication → OAuth 2.0:
7. Authorization URL: `https://id.nuclide.systems/authorize`
8. Access Token URL: `https://id.nuclide.systems/api/oidc/token`
9. Resource URL: `https://id.nuclide.systems/api/oidc/userinfo`
10. Redirect URL: `http://192.168.1.8:9000/`
11. Client ID / Secret from step 2
12. User Identifier: `email`
13. Scopes: `openid profile email`
14. Logout URL: `https://id.nuclide.systems/logout`
15. Enable "Automatic user provisioning"

Or via API once client credentials are known:

    JWT=$(curl -s -X POST http://192.168.1.8:9000/api/auth \
      -H "Content-Type: application/json" \
      -d '{"username":"fkrebs","password":"tapirnase"}' | python3 -c 'import sys,json; print(json.load(sys.stdin)["jwt"])')

    curl -X PUT http://192.168.1.8:9000/api/settings \
      -H "Authorization: Bearer $JWT" \
      -H "Content-Type: application/json" \
      -d '{
        "AuthenticationMethod": 3,
        "OAuthSettings": {
          "ClientID": "<client-id>",
          "ClientSecret": "<client-secret>",
          "AuthorizationURI": "https://id.nuclide.systems/authorize",
          "AccessTokenURI": "https://id.nuclide.systems/api/oidc/token",
          "ResourceURI": "https://id.nuclide.systems/api/oidc/userinfo",
          "RedirectURI": "http://192.168.1.8:9000/",
          "LogoutURI": "https://id.nuclide.systems/logout",
          "UserIdentifier": "email",
          "Scopes": "openid profile email",
          "OAuthAutoCreateUsers": true,
          "SSO": true
        }
      }'

## Backup

- Local S3: daily at 01:15 UTC via `portainer-backup.timer` on CT 109
- Script: `/usr/local/sbin/portainer-backup.py`
- Bucket: `ct109-portainer-backup` on Garage (CT 104:10004)
- Key ID: `GKd4511c4a01155ebbc37aa7ff`
- Retention: 7 daily tarballs
- Offsite: CT 103 `walg-offsite-sync.sh` syncs `ct109-portainer-backup` → `jottacloud:WAL-G/ct109-portainer-backup/` daily at 02:30

Restore:

    # Download latest from Garage
    aws --endpoint-url http://192.168.1.40:10004 s3 ls s3://ct109-portainer-backup/
    aws --endpoint-url http://192.168.1.40:10004 s3 cp s3://ct109-portainer-backup/portainer-YYYY-MM-DD.tar.gz .
    tar xzf portainer-YYYY-MM-DD.tar.gz
    # Replace /var/lib/docker/volumes/monitoring_portainer_data/_data/ with extracted portainer_data/

## Ops

    # CT 109
    cd /opt/stacks/monitoring
    docker compose up -d --force-recreate portainer
    docker logs portainer -f

    # Trigger manual backup
    python3 /usr/local/sbin/portainer-backup.py

## Observability

Portainer metrics scraped by Prometheus on CT 109 at `/api/metrics`.

- Auth: `X-API-Key: ptr_tJhUVPuut6wreG6yTkuRZcfM0Rlu8Jehbx7+LHG4unc=` (prometheus-scrape API token, user fkrebs)
- Prometheus job: `portainer` in `/opt/stacks/monitoring/prometheus/prometheus.yml`
- Grafana dashboard: `http://192.168.1.8:3000/d/portainer-be/portainer-be`
- Environment count, health status table, CPU/memory per environment, auth attempt rate
- Alert rules: `/opt/stacks/monitoring/prometheus/rules/portainer.yml`
- `PortainerEnvironmentUnhealthy` — env status == 2 for >2m → warning
- `PortainerDown` — scrape target unreachable for >1m → critical

Key metrics: | Metric | Description | |---|---| | `portainer_environment_count` | Total environments by type (docker/k8s/swarm) | | `portainer_environment_status` | 1=healthy, 2=unhealthy per environment | | `portainer_environment_resource_usage` | CPU/memory % per environment | | `portainer_authentication_total_status` | Auth success/fail counters |

## Arcane decommission (2026-05-26)

Arcane was replaced by Portainer on 2026-05-26: - Arcane server (CT 109 `/opt/stacks/arcane/docker-compose.yml`) → renamed `.DECOMMISSIONED-2026-05-26` - Arcane agents removed from: CT 104 ops-agents, CT 105 ops-agents, CT 111 ops-agents, CT 112 ops-agents, CT 113 db stack - Arcane OIDC client `81cf4ed0-ea48-4df7-9c2d-cc1704b060f9` in Pocket-ID → to be deleted manually - Zoraxy route `arcane.nuclide.systems` → still exists, pending explicit confirmation to remove