homelab/documentation/KBAs/KBA-001-Komodo-GitOps-Stack-Deployment-Failures.md

# KBA-001: Komodo GitOps Stack Deployment Failures

**Status:** Resolved  
**Created:** April 12, 2026  
**Last Updated:** April 12, 2026  
**Affected Systems:** Komodo Core v2.1.2, Komodo Periphery v2.1.2  
**Severity:** High

---

## Summary

Git-linked Stacks in Komodo fail to pull and deploy with "canonicalize" errors or "image not found" errors despite valid Git repository configuration.

---

## Problem Description

### Symptoms

When attempting to deploy a Docker Compose stack from a Git repository in Komodo:

1. **Pull Stack** operation fails with:
   ```
   Failed to validate run directory on host after stack write (canonicalize error),
   path=/etc/komodo/repos/homelab: No such file or directory (os error 2)
   ```

2. **Deploy Stack** operation fails with:
   ```
   failed to resolve reference "docker.io/app:v1.2.11": not found
   ```

3. Stack deployment file (`/etc/komodo/stacks/{stack-name}/compose.yaml`) is not updated from Git repository source.

### Environment

- **Komodo Core:** v2.1.2 running on Heimdall (10.0.0.151)
- **Komodo Periphery:** v2.1.2 running on Waldorf (10.0.0.251)
- **Git Provider:** Gitea (self-hosted)
- **Repository:** `https://git.castaldifamily.com/nathan/homelab.git`
- **Stack Configuration:**
  - Type: Git Repo
  - Repo: `homelab`
  - Branch: `main`
  - Run Directory: `nodes/waldorf/tunarr`
  - File Paths: (empty/blank)

---

## Root Cause Analysis

### Cause 1: Missing Repos Volume Mapping (Periphery)

**Issue:**  
Komodo Periphery containers require explicit volume mapping for the `/etc/komodo/repos` directory where Git repositories are cloned.

**Impact:**  
Without this mapping, the Periphery cannot access the Git repository files, causing "canonicalize" errors when attempting to validate the run directory path.

**Affected Configuration:**  
Any Komodo Periphery service deployed via Docker Compose that manages Git-linked stacks.

**Detection:**
```bash
# Check current mounts
docker inspect komodo-periphery-{node-name} --format '{{range .Mounts}}{{.Source}} -> {{.Destination}}{{println}}{{end}}' | grep repos

# Expected output (if missing):
(no output)

# Correct output:
/mnt/appdata/komodo/{node-name}/repos -> /etc/komodo/repos
```

---

### Cause 2: Docker Image Tag Version Prefix

**Issue:**  
Docker image tags on Docker Hub typically **do not** include a `v` prefix, while Git tags commonly use the `v1.2.3` format.

**Impact:**  
Specifying an image as `app:v1.2.11` fails to pull because Docker Hub hosts the tag as `app:1.2.11` (without the `v`).

**Example:**
```yaml
# ❌ Incorrect - Fails to pull
image: chrisbenincasa/tunarr:v1.2.11

# ✅ Correct - Pulls successfully
image: chrisbenincasa/tunarr:1.2.11
```

**Verification:**
```bash
# Query Docker Hub API for available tags
curl -s "https://registry.hub.docker.com/v2/repositories/{user}/{image}/tags" | grep -oP '"name":"\K[^"]+'
```

---

## Resolution

### Step 1: Add Repos Volume to Komodo Periphery

**File:** `/mnt/appdata/docker-compose.yaml` (or wherever Core stack is deployed)

**Add the following volume mapping to the Periphery service:**

```yaml
periphery:
  image: ghcr.io/moghtech/komodo-periphery:2
  volumes:
    - /proc:/proc
    - /mnt/appdata/komodo/{node-name}/keys:/config/keys
    - /mnt/appdata/komodo/{node-name}/work:/etc/komodo
    - /mnt/appdata/komodo/repos/{node-name}:/etc/komodo/repos  # Add this line
```

**Create the repos directory on the host:**
```bash
mkdir -p /mnt/appdata/komodo/repos/{node-name}
chown -R 1000:1000 /mnt/appdata/komodo/repos/{node-name}
```

**Redeploy the Periphery container:**
```bash
cd /path/to/compose/file
docker compose down periphery
docker compose up -d periphery
```

---

### Step 2: Fix Docker Image Tag Prefix

**File:** `nodes/{node-name}/{service-name}/compose.yaml` (in Git repo)

**Remove the `v` prefix from image tags:**

```yaml
services:
  app:
    image: user/app:1.2.11  # Changed from v1.2.11
```

**Commit and push changes:**
```bash
git add .
git commit -m "Fix: Remove 'v' prefix from Docker image tags"
git push
```

---

### Step 3: Verify and Deploy

**Pull the updated repository on the Periphery node:**
```bash
docker exec komodo-periphery-{node-name} sh -c 'cd /etc/komodo/repos/{repo-name} && git pull'
```

**In Komodo UI:**
1. Navigate to **Stacks** → Select your stack
2. Click **"Pull Stack"** button
3. Verify success notification
4. Click **"Deploy Stack"** button
5. Confirm container is recreated with correct image

**Verification Commands:**
```bash
# Check if repos directory is mounted
docker inspect komodo-periphery-{node-name} | grep "/etc/komodo/repos"

# Check running container image
docker ps --filter 'name={container-name}' --format '{{.Image}}'

# Verify GPU access (if applicable)
docker exec {container-name} nvidia-smi
```

---

## Additional Notes

### Known Issue: Komodo Pull Button Manual Workaround

If the **Pull Stack** button does not update the deployment file automatically:

**Temporary Workaround:**
```bash
# Manually copy file from Git repo to deployment location
docker exec komodo-periphery-{node-name} cp \
  /etc/komodo/repos/{repo-name}/{path}/compose.yaml \
  /etc/komodo/stacks/{stack-name}/compose.yaml
```

**This issue is under investigation.**

---

### Best Practices

1. **Always verify Docker image tags** before adding them to compose files:
   - Check Docker Hub directly
   - Use the API to list available tags
   - Prefer numeric tags over `latest` for production

2. **Standardize volume mappings** across all Komodo Periphery nodes:
   ```yaml
   - /mnt/appdata/komodo/{node-name}/keys:/config/keys
   - /mnt/appdata/komodo/{node-name}/work:/etc/komodo
   - /mnt/appdata/komodo/repos/{node-name}:/etc/komodo/repos
   ```

3. **Document Git tag to Docker tag conversions** when they differ:
   - Create a mapping reference file
   - Use CI/CD to validate tags before merge

---

## Related Documentation

- [TECHNICAL_RUNBOOK.md](../TECHNICAL_RUNBOOK.md) - Infrastructure overview and emergency procedures
- Repository Memory: `/memories/repo/critical-fixes.md`

---

## Resolution Checklist

- [x] Added repos volume mapping to Komodo Periphery
- [x] Fixed Docker image tag prefix (removed `v`)
- [x] Verified Git repository accessible inside Periphery container
- [x] Tested Pull Stack operation in Komodo UI
- [x] Tested Deploy Stack operation in Komodo UI
- [x] Confirmed container running with correct image version
- [x] Verified GPU passthrough (if applicable)
- [x] Updated repository memory with lessons learned

---

**Document Version:** 1.0  
**Validation Status:** Tested and Verified ✅