fkrebs/docs
1
0
Fork 0
Code Issues 14 Pull Requests Actions Packages Projects Releases Wiki Activity

Garage 0.9.4 → 1.x upgrade: reclaim 3 corrupted orphan buckets #19

New Issue
Open
opened 2026-07-21 16:18:50 +00:00 by fkrebs · 0 comments
fkrebs commented 2026-07-21 16:18:50 +00:00
Owner
Copy Link
Copy Source

Problem

Garage v0.9.4 on CT 104 has 3 orphan buckets that cannot be deleted due to a metadata corruption bug:

  • lobe-files (0 B, 0 objects) — LobeChat sunset 2026-05-26
  • lobe-pg-backup (150.8 MiB, 511 objects) — LobeChat WAL-G backups
  • memos (3.5 MiB, 4 objects) — Memos sunset 2026-07-10

All three return the same error on docker exec garage /garage bucket delete --yes <name>:

Error: Internal error: Remote error: Could not reach quorum of 1. 0 of 1 request succeeded, others returned errors: ["Unable to decode entry of object"]

What has been tried

  • garage repair --yes tables — no effect
  • Key deletion via garage key delete — keys deleted successfully, so no access possible
  • No aliases to unalias (bucket unalias fails because there are no other aliases to keep)

Impact

  • 154 MiB orphaned (not urgent, CT 104 disk at 80%)
  • Bucket names squat on the global namespace (blocks recreating with same name)

Fix

Garage 1.x has a full rewrite of the metadata subsystem which likely resolves this. Upgrade path:

  1. Stop garage on CT 104 (all clients: Immich, OWUI, Bifrost, Backrest, etc. — heavy blast radius)
  2. Backup /opt/stacks/shared-db/garage/ (96 GiB) to UNAS
  3. Bump image tag in /opt/stacks/shared-db/docker-compose.yml
  4. Follow Garage 0.9→1.x migration notes (schema change)
  5. Attempt bucket deletion after upgrade

Medium risk — Garage 1.x has been stable for a while but the on-disk migration is one-way.

Priority

Low — no active failure, just cosmetic. Bundle with next planned maintenance window.

## Problem Garage v0.9.4 on CT 104 has 3 orphan buckets that cannot be deleted due to a metadata corruption bug: - `lobe-files` (0 B, 0 objects) — LobeChat sunset 2026-05-26 - `lobe-pg-backup` (150.8 MiB, 511 objects) — LobeChat WAL-G backups - `memos` (3.5 MiB, 4 objects) — Memos sunset 2026-07-10 All three return the same error on `docker exec garage /garage bucket delete --yes <name>`: ``` Error: Internal error: Remote error: Could not reach quorum of 1. 0 of 1 request succeeded, others returned errors: ["Unable to decode entry of object"] ``` ## What has been tried - `garage repair --yes tables` — no effect - Key deletion via `garage key delete` — keys deleted successfully, so no access possible - No aliases to unalias (`bucket unalias` fails because there are no other aliases to keep) ## Impact - 154 MiB orphaned (not urgent, CT 104 disk at 80%) - Bucket names squat on the global namespace (blocks recreating with same name) ## Fix Garage 1.x has a full rewrite of the metadata subsystem which likely resolves this. Upgrade path: 1. Stop garage on CT 104 (all clients: Immich, OWUI, Bifrost, Backrest, etc. — heavy blast radius) 2. Backup `/opt/stacks/shared-db/garage/` (96 GiB) to UNAS 3. Bump image tag in `/opt/stacks/shared-db/docker-compose.yml` 4. Follow Garage 0.9→1.x migration notes (schema change) 5. Attempt bucket deletion after upgrade Medium risk — Garage 1.x has been stable for a while but the on-disk migration is one-way. ## Priority Low — no active failure, just cosmetic. Bundle with next planned maintenance window.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
fkrebs (Florian Krebs)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: fkrebs/docs#19
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?