From 0d19230ca13b5e71eba2636bd7ce72d41f43407e Mon Sep 17 00:00:00 2001 From: nathan Date: Sat, 11 Apr 2026 23:34:54 -0400 Subject: [PATCH] docs: Add KBA-001 for Komodo GitOps deployment troubleshooting --- ...Komodo-GitOps-Stack-Deployment-Failures.md | 245 ++++++++++++++++++ 1 file changed, 245 insertions(+) create mode 100644 documentation/KBA-001-Komodo-GitOps-Stack-Deployment-Failures.md diff --git a/documentation/KBA-001-Komodo-GitOps-Stack-Deployment-Failures.md b/documentation/KBA-001-Komodo-GitOps-Stack-Deployment-Failures.md new file mode 100644 index 0000000..5615003 --- /dev/null +++ b/documentation/KBA-001-Komodo-GitOps-Stack-Deployment-Failures.md @@ -0,0 +1,245 @@ +# KBA-001: Komodo GitOps Stack Deployment Failures + +**Status:** Resolved +**Created:** April 12, 2026 +**Last Updated:** April 12, 2026 +**Affected Systems:** Komodo Core v2.1.2, Komodo Periphery v2.1.2 +**Severity:** High + +--- + +## Summary + +Git-linked Stacks in Komodo fail to pull and deploy with "canonicalize" errors or "image not found" errors despite valid Git repository configuration. + +--- + +## Problem Description + +### Symptoms + +When attempting to deploy a Docker Compose stack from a Git repository in Komodo: + +1. **Pull Stack** operation fails with: + ``` + Failed to validate run directory on host after stack write (canonicalize error), + path=/etc/komodo/repos/homelab: No such file or directory (os error 2) + ``` + +2. **Deploy Stack** operation fails with: + ``` + failed to resolve reference "docker.io/app:v1.2.11": not found + ``` + +3. Stack deployment file (`/etc/komodo/stacks/{stack-name}/compose.yaml`) is not updated from Git repository source. + +### Environment + +- **Komodo Core:** v2.1.2 running on Heimdall (10.0.0.151) +- **Komodo Periphery:** v2.1.2 running on Waldorf (10.0.0.251) +- **Git Provider:** Gitea (self-hosted) +- **Repository:** `https://git.castaldifamily.com/nathan/homelab.git` +- **Stack Configuration:** + - Type: Git Repo + - Repo: `homelab` + - Branch: `main` + - Run Directory: `nodes/waldorf/tunarr` + - File Paths: (empty/blank) + +--- + +## Root Cause Analysis + +### Cause 1: Missing Repos Volume Mapping (Periphery) + +**Issue:** +Komodo Periphery containers require explicit volume mapping for the `/etc/komodo/repos` directory where Git repositories are cloned. + +**Impact:** +Without this mapping, the Periphery cannot access the Git repository files, causing "canonicalize" errors when attempting to validate the run directory path. + +**Affected Configuration:** +Any Komodo Periphery service deployed via Docker Compose that manages Git-linked stacks. + +**Detection:** +```bash +# Check current mounts +docker inspect komodo-periphery-{node-name} --format '{{range .Mounts}}{{.Source}} -> {{.Destination}}{{println}}{{end}}' | grep repos + +# Expected output (if missing): +(no output) + +# Correct output: +/mnt/appdata/komodo/{node-name}/repos -> /etc/komodo/repos +``` + +--- + +### Cause 2: Docker Image Tag Version Prefix + +**Issue:** +Docker image tags on Docker Hub typically **do not** include a `v` prefix, while Git tags commonly use the `v1.2.3` format. + +**Impact:** +Specifying an image as `app:v1.2.11` fails to pull because Docker Hub hosts the tag as `app:1.2.11` (without the `v`). + +**Example:** +```yaml +# ❌ Incorrect - Fails to pull +image: chrisbenincasa/tunarr:v1.2.11 + +# ✅ Correct - Pulls successfully +image: chrisbenincasa/tunarr:1.2.11 +``` + +**Verification:** +```bash +# Query Docker Hub API for available tags +curl -s "https://registry.hub.docker.com/v2/repositories/{user}/{image}/tags" | grep -oP '"name":"\K[^"]+' +``` + +--- + +## Resolution + +### Step 1: Add Repos Volume to Komodo Periphery + +**File:** `/mnt/appdata/docker-compose.yaml` (or wherever Core stack is deployed) + +**Add the following volume mapping to the Periphery service:** + +```yaml +periphery: + image: ghcr.io/moghtech/komodo-periphery:2 + volumes: + - /proc:/proc + - /mnt/appdata/komodo/{node-name}/keys:/config/keys + - /mnt/appdata/komodo/{node-name}/work:/etc/komodo + - /mnt/appdata/komodo/repos/{node-name}:/etc/komodo/repos # Add this line +``` + +**Create the repos directory on the host:** +```bash +mkdir -p /mnt/appdata/komodo/repos/{node-name} +chown -R 1000:1000 /mnt/appdata/komodo/repos/{node-name} +``` + +**Redeploy the Periphery container:** +```bash +cd /path/to/compose/file +docker compose down periphery +docker compose up -d periphery +``` + +--- + +### Step 2: Fix Docker Image Tag Prefix + +**File:** `nodes/{node-name}/{service-name}/compose.yaml` (in Git repo) + +**Remove the `v` prefix from image tags:** + +```yaml +services: + app: + image: user/app:1.2.11 # Changed from v1.2.11 +``` + +**Commit and push changes:** +```bash +git add . +git commit -m "Fix: Remove 'v' prefix from Docker image tags" +git push +``` + +--- + +### Step 3: Verify and Deploy + +**Pull the updated repository on the Periphery node:** +```bash +docker exec komodo-periphery-{node-name} sh -c 'cd /etc/komodo/repos/{repo-name} && git pull' +``` + +**In Komodo UI:** +1. Navigate to **Stacks** → Select your stack +2. Click **"Pull Stack"** button +3. Verify success notification +4. Click **"Deploy Stack"** button +5. Confirm container is recreated with correct image + +**Verification Commands:** +```bash +# Check if repos directory is mounted +docker inspect komodo-periphery-{node-name} | grep "/etc/komodo/repos" + +# Check running container image +docker ps --filter 'name={container-name}' --format '{{.Image}}' + +# Verify GPU access (if applicable) +docker exec {container-name} nvidia-smi +``` + +--- + +## Additional Notes + +### Known Issue: Komodo Pull Button Manual Workaround + +If the **Pull Stack** button does not update the deployment file automatically: + +**Temporary Workaround:** +```bash +# Manually copy file from Git repo to deployment location +docker exec komodo-periphery-{node-name} cp \ + /etc/komodo/repos/{repo-name}/{path}/compose.yaml \ + /etc/komodo/stacks/{stack-name}/compose.yaml +``` + +**This issue is under investigation.** + +--- + +### Best Practices + +1. **Always verify Docker image tags** before adding them to compose files: + - Check Docker Hub directly + - Use the API to list available tags + - Prefer numeric tags over `latest` for production + +2. **Standardize volume mappings** across all Komodo Periphery nodes: + ```yaml + - /mnt/appdata/komodo/{node-name}/keys:/config/keys + - /mnt/appdata/komodo/{node-name}/work:/etc/komodo + - /mnt/appdata/komodo/repos/{node-name}:/etc/komodo/repos + ``` + +3. **Document Git tag to Docker tag conversions** when they differ: + - Create a mapping reference file + - Use CI/CD to validate tags before merge + +--- + +## Related Documentation + +- [TECHNICAL_RUNBOOK.md](TECHNICAL_RUNBOOK.md) - Infrastructure overview and emergency procedures +- [LLM_HANDOVER.md](LLM_HANDOVER.md) - Quick-start context for AI sessions +- Repository Memory: `/memories/repo/critical-fixes.md` + +--- + +## Resolution Checklist + +- [x] Added repos volume mapping to Komodo Periphery +- [x] Fixed Docker image tag prefix (removed `v`) +- [x] Verified Git repository accessible inside Periphery container +- [x] Tested Pull Stack operation in Komodo UI +- [x] Tested Deploy Stack operation in Komodo UI +- [x] Confirmed container running with correct image version +- [x] Verified GPU passthrough (if applicable) +- [x] Updated repository memory with lessons learned + +--- + +**Document Version:** 1.0 +**Validation Status:** Tested and Verified ✅