homelab/documentation/KBAs/KBA-001-Komodo-GitOps-Stack-Deployment-Failures.md

KBA-001: Komodo GitOps Stack Deployment Failures

Status: Resolved
Created: April 12, 2026
Last Updated: April 12, 2026
Affected Systems: Komodo Core v2.1.2, Komodo Periphery v2.1.2
Severity: High

---

Summary

Git-linked Stacks in Komodo fail to pull and deploy with "canonicalize" errors or "image not found" errors despite valid Git repository configuration.

---

Problem Description

Symptoms

When attempting to deploy a Docker Compose stack from a Git repository in Komodo:

1. Pull Stack operation fails with:

   Failed to validate run directory on host after stack write (canonicalize error),
   path=/etc/komodo/repos/homelab: No such file or directory (os error 2)
   

2. Deploy Stack operation fails with:

   failed to resolve reference "docker.io/app:v1.2.11": not found
   

3. Stack deployment file (/etc/komodo/stacks/{stack-name}/compose.yaml) is not updated from Git repository source.

Environment

• Komodo Core: v2.1.2 running on Heimdall (10.0.0.151)
• Komodo Periphery: v2.1.2 running on Waldorf (10.0.0.251)
• Git Provider: Gitea (self-hosted)
• Repository: https://git.castaldifamily.com/nathan/homelab.git
• Stack Configuration:
  • Type: Git Repo
  • Repo: homelab
  • Branch: main
  • Run Directory: nodes/waldorf/tunarr
  • File Paths: (empty/blank)

---

Root Cause Analysis

Cause 1: Missing Repos Volume Mapping (Periphery)

Issue:
Komodo Periphery containers require explicit volume mapping for the /etc/komodo/repos directory where Git repositories are cloned.

Impact:
Without this mapping, the Periphery cannot access the Git repository files, causing "canonicalize" errors when attempting to validate the run directory path.

Affected Configuration:
Any Komodo Periphery service deployed via Docker Compose that manages Git-linked stacks.

Detection:

# Check current mounts
docker inspect komodo-periphery-{node-name} --format '{{range .Mounts}}{{.Source}} -> {{.Destination}}{{println}}{{end}}' | grep repos

# Expected output (if missing):
(no output)

# Correct output:
/mnt/appdata/komodo/{node-name}/repos -> /etc/komodo/repos

---

Cause 2: Docker Image Tag Version Prefix

Issue:
Docker image tags on Docker Hub typically do not include a v prefix, while Git tags commonly use the v1.2.3 format.

Impact:
Specifying an image as app:v1.2.11 fails to pull because Docker Hub hosts the tag as app:1.2.11 (without the v).

Example:

# ❌ Incorrect - Fails to pull
image: chrisbenincasa/tunarr:v1.2.11

# ✅ Correct - Pulls successfully
image: chrisbenincasa/tunarr:1.2.11

Verification:

# Query Docker Hub API for available tags
curl -s "https://registry.hub.docker.com/v2/repositories/{user}/{image}/tags" | grep -oP '"name":"\K[^"]+'

---

Resolution

Step 1: Add Repos Volume to Komodo Periphery

File: /mnt/appdata/docker-compose.yaml (or wherever Core stack is deployed)

Add the following volume mapping to the Periphery service:

periphery:
  image: ghcr.io/moghtech/komodo-periphery:2
  volumes:
    - /proc:/proc
    - /mnt/appdata/komodo/{node-name}/keys:/config/keys
    - /mnt/appdata/komodo/{node-name}/work:/etc/komodo
    - /mnt/appdata/komodo/repos/{node-name}:/etc/komodo/repos  # Add this line

Create the repos directory on the host:

mkdir -p /mnt/appdata/komodo/repos/{node-name}
chown -R 1000:1000 /mnt/appdata/komodo/repos/{node-name}

Redeploy the Periphery container:

cd /path/to/compose/file
docker compose down periphery
docker compose up -d periphery

---

Step 2: Fix Docker Image Tag Prefix

File: nodes/{node-name}/{service-name}/compose.yaml (in Git repo)

Remove the v prefix from image tags:

services:
  app:
    image: user/app:1.2.11  # Changed from v1.2.11

Commit and push changes:

git add .
git commit -m "Fix: Remove 'v' prefix from Docker image tags"
git push

---

Step 3: Verify and Deploy

Pull the updated repository on the Periphery node:

docker exec komodo-periphery-{node-name} sh -c 'cd /etc/komodo/repos/{repo-name} && git pull'

In Komodo UI:

1. Navigate to Stacks → Select your stack
2. Click "Pull Stack" button
3. Verify success notification
4. Click "Deploy Stack" button
5. Confirm container is recreated with correct image

Verification Commands:

# Check if repos directory is mounted
docker inspect komodo-periphery-{node-name} | grep "/etc/komodo/repos"

# Check running container image
docker ps --filter 'name={container-name}' --format '{{.Image}}'

# Verify GPU access (if applicable)
docker exec {container-name} nvidia-smi

---

Additional Notes

Known Issue: Komodo Pull Button Manual Workaround

If the Pull Stack button does not update the deployment file automatically:

Temporary Workaround:

# Manually copy file from Git repo to deployment location
docker exec komodo-periphery-{node-name} cp \
  /etc/komodo/repos/{repo-name}/{path}/compose.yaml \
  /etc/komodo/stacks/{stack-name}/compose.yaml

This issue is under investigation.

---

Best Practices

1. Always verify Docker image tags before adding them to compose files:

   • Check Docker Hub directly
   • Use the API to list available tags
   • Prefer numeric tags over latest for production

2. Standardize volume mappings across all Komodo Periphery nodes:

   - /mnt/appdata/komodo/{node-name}/keys:/config/keys
   - /mnt/appdata/komodo/{node-name}/work:/etc/komodo
   - /mnt/appdata/komodo/repos/{node-name}:/etc/komodo/repos
   

3. Document Git tag to Docker tag conversions when they differ:

   • Create a mapping reference file
   • Use CI/CD to validate tags before merge

---

Related Documentation

• TECHNICAL_RUNBOOK.md - Infrastructure overview and emergency procedures
• Repository Memory: /memories/repo/critical-fixes.md

---

Resolution Checklist

• Added repos volume mapping to Komodo Periphery
• Fixed Docker image tag prefix (removed v)
• Verified Git repository accessible inside Periphery container
• Tested Pull Stack operation in Komodo UI
• Tested Deploy Stack operation in Komodo UI
• Confirmed container running with correct image version
• Verified GPU passthrough (if applicable)
• Updated repository memory with lessons learned

---

Document Version: 1.0
Validation Status: Tested and Verified ✅