docs/services/portainer.md

Portainer BE (migrated from Arcane 2026-05-26)

Docker management UI with Portainer Business Edition license.

Stack

• Server: CT 109 ("ops", 192.168.1.8), /opt/stacks/monitoring/docker-compose.yml, port 9000
• Image: portainer/portainer-ee:2.39.2
• Data volume: monitoring_portainer_data → /var/lib/docker/volumes/monitoring_portainer_data/_data
• URL: http://192.168.1.8:9000 (LAN-only; no Zoraxy route — access via LAN/SSH tunnel)
• Admin user: fkrebs / tapirnase

License

• Edition: Business Edition (BE) — 3 nodes free for personal use
• License key: 3-QIORiAXMuBSgeYdePYgh1nuqRkvx/XWyu5D/+MQlVpvSng2CXtCG4V78212HEleOIWnIV0kK5IkpEaifea3b8NGU6o2STA0c/XXj150c/v5XSguhchCmqiXyWXDj2/+r
• Expires: 2027-05-25 (365 days from issue date 2026-05-26)
• Nodes: 3 licensed (CT 104, CT 109 local, + 1 more)

To re-apply license (e.g. after fresh install):

curl -X POST http://192.168.1.8:9000/api/auth \
  -H "Content-Type: application/json" \
  -d '{"username":"fkrebs","password":"tapirnase"}' | python3 -c 'import sys,json; print(json.load(sys.stdin)["jwt"])'
# then:
curl -X POST http://192.168.1.8:9000/api/licenses/add \
  -H "Authorization: Bearer <jwt>" \
  -H "Content-Type: application/json" \
  -d '{"license":"<key>"}'

Agents (Portainer environments)

Host	LXC Name	IP	Port	Stack
CT 109	ops	unix:///var/run/docker.sock	—	built-in
CT 104	docker	192.168.1.40	9001	/opt/stacks/portainer-agent.yml
CT 110	id	192.168.1.5	—	DESTROYED 2026-05-26 — Pocket-ID moved to CT 109
CT 111	dev	192.168.1.42	9001	/opt/stacks/ops-agents/docker-compose.yml
CT 112	secrets	192.168.1.7	9001	/opt/stacks/ops-agents/docker-compose.yml
CT 113	db	192.168.1.6	9001	/opt/stacks/db/docker-compose.yml

Add each as a Portainer Agent environment: http://<ip>:9001.

OIDC (Pocket-ID)

Status: pending — requires manual OIDC client creation in Pocket-ID web UI first.

1. Go to https://id.nuclide.systems/settings/admin/oidc-clients
2. Create client:
   • Name: portainer
   • Redirect URIs: http://192.168.1.8:9000/
   • Note the Client ID and Client Secret
3. Configure in Portainer → Settings → Authentication → OAuth 2.0:
   • Authorization URL: https://id.nuclide.systems/authorize
   • Access Token URL: https://id.nuclide.systems/api/oidc/token
   • Resource URL: https://id.nuclide.systems/api/oidc/userinfo
   • Redirect URL: http://192.168.1.8:9000/
   • Client ID / Secret from step 2
   • User Identifier: email
   • Scopes: openid profile email
   • Logout URL: https://id.nuclide.systems/logout
   • Enable "Automatic user provisioning"

Or via API once client credentials are known:

JWT=$(curl -s -X POST http://192.168.1.8:9000/api/auth \
  -H "Content-Type: application/json" \
  -d '{"username":"fkrebs","password":"tapirnase"}' | python3 -c 'import sys,json; print(json.load(sys.stdin)["jwt"])')

curl -X PUT http://192.168.1.8:9000/api/settings \
  -H "Authorization: Bearer $JWT" \
  -H "Content-Type: application/json" \
  -d '{
    "AuthenticationMethod": 3,
    "OAuthSettings": {
      "ClientID": "<client-id>",
      "ClientSecret": "<client-secret>",
      "AuthorizationURI": "https://id.nuclide.systems/authorize",
      "AccessTokenURI": "https://id.nuclide.systems/api/oidc/token",
      "ResourceURI": "https://id.nuclide.systems/api/oidc/userinfo",
      "RedirectURI": "http://192.168.1.8:9000/",
      "LogoutURI": "https://id.nuclide.systems/logout",
      "UserIdentifier": "email",
      "Scopes": "openid profile email",
      "OAuthAutoCreateUsers": true,
      "SSO": true
    }
  }'

Backup

• Local S3: daily at 01:15 UTC via portainer-backup.timer on CT 109
  • Script: /usr/local/sbin/portainer-backup.py
  • Bucket: ct109-portainer-backup on Garage (CT 104:10004)
  • Key ID: GKd4511c4a01155ebbc37aa7ff
  • Retention: 7 daily tarballs
• Offsite: CT 103 walg-offsite-sync.sh syncs ct109-portainer-backup → jottacloud:WAL-G/ct109-portainer-backup/ daily at 02:30

Restore:

# Download latest from Garage
aws --endpoint-url http://192.168.1.40:10004 s3 ls s3://ct109-portainer-backup/
aws --endpoint-url http://192.168.1.40:10004 s3 cp s3://ct109-portainer-backup/portainer-YYYY-MM-DD.tar.gz .
tar xzf portainer-YYYY-MM-DD.tar.gz
# Replace /var/lib/docker/volumes/monitoring_portainer_data/_data/ with extracted portainer_data/

Ops

# CT 109
cd /opt/stacks/monitoring
docker compose up -d --force-recreate portainer
docker logs portainer -f

# Trigger manual backup
python3 /usr/local/sbin/portainer-backup.py

Observability

Portainer metrics scraped by Prometheus on CT 109 at /api/metrics.

• Auth: X-API-Key: ptr_tJhUVPuut6wreG6yTkuRZcfM0Rlu8Jehbx7+LHG4unc= (prometheus-scrape API token, user fkrebs)
• Prometheus job: portainer in /opt/stacks/monitoring/prometheus/prometheus.yml
• Grafana dashboard: http://192.168.1.8:3000/d/portainer-be/portainer-be
  • Environment count, health status table, CPU/memory per environment, auth attempt rate
• Alert rules: /opt/stacks/monitoring/prometheus/rules/portainer.yml
  • PortainerEnvironmentUnhealthy — env status == 2 for >2m → warning
  • PortainerDown — scrape target unreachable for >1m → critical

Key metrics:

Metric	Description
portainer_environment_count	Total environments by type (docker/k8s/swarm)
portainer_environment_status	1=healthy, 2=unhealthy per environment
portainer_environment_resource_usage	CPU/memory % per environment
portainer_authentication_total_status	Auth success/fail counters

Arcane decommission (2026-05-26)

Arcane was replaced by Portainer on 2026-05-26:

• Arcane server (CT 109 /opt/stacks/arcane/docker-compose.yml) → renamed .DECOMMISSIONED-2026-05-26
• Arcane agents removed from: CT 104 ops-agents, CT 105 ops-agents, CT 111 ops-agents, CT 112 ops-agents, CT 113 db stack
• Arcane OIDC client 81cf4ed0-ea48-4df7-9c2d-cc1704b060f9 in Pocket-ID → to be deleted manually
• Zoraxy route arcane.nuclide.systems → still exists, pending explicit confirmation to remove