docs/services/portainer.md

# Portainer BE (migrated from Arcane 2026-05-26)

Docker management UI with Portainer Business Edition license.

## Stack

- **Server**: CT 109 ("ops", 192.168.1.8), `/opt/stacks/monitoring/docker-compose.yml`, port **9000**
- **Image**: `portainer/portainer-ee:2.39.2`
- **Data volume**: `monitoring_portainer_data` → `/var/lib/docker/volumes/monitoring_portainer_data/_data`
- **URL**: `http://192.168.1.8:9000` (LAN-only; no Zoraxy route — access via LAN/SSH tunnel)
- **Admin user**: `fkrebs` / `tapirnase`

## License

- **Edition**: Business Edition (BE) — 3 nodes free for personal use
- **License key**: `3-QIORiAXMuBSgeYdePYgh1nuqRkvx/XWyu5D/+MQlVpvSng2CXtCG4V78212HEleOIWnIV0kK5IkpEaifea3b8NGU6o2STA0c/XXj150c/v5XSguhchCmqiXyWXDj2/+r`
- **Expires**: 2027-05-25 (365 days from issue date 2026-05-26)
- **Nodes**: 3 licensed (CT 104, CT 109 local, + 1 more)

To re-apply license (e.g. after fresh install):
```bash
curl -X POST http://192.168.1.8:9000/api/auth \
  -H "Content-Type: application/json" \
  -d '{"username":"fkrebs","password":"tapirnase"}' | python3 -c 'import sys,json; print(json.load(sys.stdin)["jwt"])'
# then:
curl -X POST http://192.168.1.8:9000/api/licenses/add \
  -H "Authorization: Bearer <jwt>" \
  -H "Content-Type: application/json" \
  -d '{"license":"<key>"}'
```

## Agents (Portainer environments)

| Host | LXC Name | IP | Port | Stack |
|---|---|---|---|---|
| CT 109 | ops | unix:///var/run/docker.sock | — | built-in |
| CT 104 | docker | 192.168.1.40 | 9001 | `/opt/stacks/portainer-agent.yml` |
| ~~CT 110~~ | ~~id~~ | ~~192.168.1.5~~ | — | **DESTROYED 2026-05-26** — Pocket-ID moved to CT 109 |
| CT 111 | dev | 192.168.1.42 | 9001 | `/opt/stacks/ops-agents/docker-compose.yml` |
| CT 112 | secrets | 192.168.1.7 | 9001 | `/opt/stacks/ops-agents/docker-compose.yml` |
| CT 113 | db | 192.168.1.6 | 9001 | `/opt/stacks/db/docker-compose.yml` |

Add each as a Portainer Agent environment: `http://<ip>:9001`.

## OIDC (Pocket-ID)

**Status: pending** — requires manual OIDC client creation in Pocket-ID web UI first.

1. Go to `https://id.nuclide.systems/settings/admin/oidc-clients`
2. Create client:
   - Name: `portainer`
   - Redirect URIs: `http://192.168.1.8:9000/`
   - Note the **Client ID** and **Client Secret**
3. Configure in Portainer → Settings → Authentication → OAuth 2.0:
   - Authorization URL: `https://id.nuclide.systems/authorize`
   - Access Token URL: `https://id.nuclide.systems/api/oidc/token`
   - Resource URL: `https://id.nuclide.systems/api/oidc/userinfo`
   - Redirect URL: `http://192.168.1.8:9000/`
   - Client ID / Secret from step 2
   - User Identifier: `email`
   - Scopes: `openid profile email`
   - Logout URL: `https://id.nuclide.systems/logout`
   - Enable "Automatic user provisioning"

Or via API once client credentials are known:
```bash
JWT=$(curl -s -X POST http://192.168.1.8:9000/api/auth \
  -H "Content-Type: application/json" \
  -d '{"username":"fkrebs","password":"tapirnase"}' | python3 -c 'import sys,json; print(json.load(sys.stdin)["jwt"])')

curl -X PUT http://192.168.1.8:9000/api/settings \
  -H "Authorization: Bearer $JWT" \
  -H "Content-Type: application/json" \
  -d '{
    "AuthenticationMethod": 3,
    "OAuthSettings": {
      "ClientID": "<client-id>",
      "ClientSecret": "<client-secret>",
      "AuthorizationURI": "https://id.nuclide.systems/authorize",
      "AccessTokenURI": "https://id.nuclide.systems/api/oidc/token",
      "ResourceURI": "https://id.nuclide.systems/api/oidc/userinfo",
      "RedirectURI": "http://192.168.1.8:9000/",
      "LogoutURI": "https://id.nuclide.systems/logout",
      "UserIdentifier": "email",
      "Scopes": "openid profile email",
      "OAuthAutoCreateUsers": true,
      "SSO": true
    }
  }'
```

## Backup

- **Local S3**: daily at 01:15 UTC via `portainer-backup.timer` on CT 109
  - Script: `/usr/local/sbin/portainer-backup.py`
  - Bucket: `ct109-portainer-backup` on Garage (CT 104:10004)
  - Key ID: `GKd4511c4a01155ebbc37aa7ff`
  - Retention: 7 daily tarballs
- **Offsite**: CT 103 `walg-offsite-sync.sh` syncs `ct109-portainer-backup` → `jottacloud:WAL-G/ct109-portainer-backup/` daily at 02:30

Restore:
```bash
# Download latest from Garage
aws --endpoint-url http://192.168.1.40:10004 s3 ls s3://ct109-portainer-backup/
aws --endpoint-url http://192.168.1.40:10004 s3 cp s3://ct109-portainer-backup/portainer-YYYY-MM-DD.tar.gz .
tar xzf portainer-YYYY-MM-DD.tar.gz
# Replace /var/lib/docker/volumes/monitoring_portainer_data/_data/ with extracted portainer_data/
```

## Ops

```bash
# CT 109
cd /opt/stacks/monitoring
docker compose up -d --force-recreate portainer
docker logs portainer -f

# Trigger manual backup
python3 /usr/local/sbin/portainer-backup.py
```

## Observability

Portainer metrics scraped by Prometheus on CT 109 at `/api/metrics`.

- **Auth**: `X-API-Key: ptr_tJhUVPuut6wreG6yTkuRZcfM0Rlu8Jehbx7+LHG4unc=` (prometheus-scrape API token, user fkrebs)
- **Prometheus job**: `portainer` in `/opt/stacks/monitoring/prometheus/prometheus.yml`
- **Grafana dashboard**: `http://192.168.1.8:3000/d/portainer-be/portainer-be`
  - Environment count, health status table, CPU/memory per environment, auth attempt rate
- **Alert rules**: `/opt/stacks/monitoring/prometheus/rules/portainer.yml`
  - `PortainerEnvironmentUnhealthy` — env status == 2 for >2m → warning
  - `PortainerDown` — scrape target unreachable for >1m → critical

Key metrics:
| Metric | Description |
|---|---|
| `portainer_environment_count` | Total environments by type (docker/k8s/swarm) |
| `portainer_environment_status` | 1=healthy, 2=unhealthy per environment |
| `portainer_environment_resource_usage` | CPU/memory % per environment |
| `portainer_authentication_total_status` | Auth success/fail counters |

## Arcane decommission (2026-05-26)

Arcane was replaced by Portainer on 2026-05-26:
- Arcane server (CT 109 `/opt/stacks/arcane/docker-compose.yml`) → renamed `.DECOMMISSIONED-2026-05-26`
- Arcane agents removed from: CT 104 ops-agents, CT 105 ops-agents, CT 111 ops-agents, CT 112 ops-agents, CT 113 db stack
- Arcane OIDC client `81cf4ed0-ea48-4df7-9c2d-cc1704b060f9` in Pocket-ID → to be deleted manually
- Zoraxy route `arcane.nuclide.systems` → still exists, pending explicit confirmation to remove