docs: Add KBA-001 for Komodo GitOps deployment troubleshooting
This commit is contained in:
parent
9eaceb5261
commit
0d19230ca1
245
documentation/KBA-001-Komodo-GitOps-Stack-Deployment-Failures.md
Normal file
245
documentation/KBA-001-Komodo-GitOps-Stack-Deployment-Failures.md
Normal file
@ -0,0 +1,245 @@
|
||||
# KBA-001: Komodo GitOps Stack Deployment Failures
|
||||
|
||||
**Status:** Resolved
|
||||
**Created:** April 12, 2026
|
||||
**Last Updated:** April 12, 2026
|
||||
**Affected Systems:** Komodo Core v2.1.2, Komodo Periphery v2.1.2
|
||||
**Severity:** High
|
||||
|
||||
---
|
||||
|
||||
## Summary
|
||||
|
||||
Git-linked Stacks in Komodo fail to pull and deploy with "canonicalize" errors or "image not found" errors despite valid Git repository configuration.
|
||||
|
||||
---
|
||||
|
||||
## Problem Description
|
||||
|
||||
### Symptoms
|
||||
|
||||
When attempting to deploy a Docker Compose stack from a Git repository in Komodo:
|
||||
|
||||
1. **Pull Stack** operation fails with:
|
||||
```
|
||||
Failed to validate run directory on host after stack write (canonicalize error),
|
||||
path=/etc/komodo/repos/homelab: No such file or directory (os error 2)
|
||||
```
|
||||
|
||||
2. **Deploy Stack** operation fails with:
|
||||
```
|
||||
failed to resolve reference "docker.io/app:v1.2.11": not found
|
||||
```
|
||||
|
||||
3. Stack deployment file (`/etc/komodo/stacks/{stack-name}/compose.yaml`) is not updated from Git repository source.
|
||||
|
||||
### Environment
|
||||
|
||||
- **Komodo Core:** v2.1.2 running on Heimdall (10.0.0.151)
|
||||
- **Komodo Periphery:** v2.1.2 running on Waldorf (10.0.0.251)
|
||||
- **Git Provider:** Gitea (self-hosted)
|
||||
- **Repository:** `https://git.castaldifamily.com/nathan/homelab.git`
|
||||
- **Stack Configuration:**
|
||||
- Type: Git Repo
|
||||
- Repo: `homelab`
|
||||
- Branch: `main`
|
||||
- Run Directory: `nodes/waldorf/tunarr`
|
||||
- File Paths: (empty/blank)
|
||||
|
||||
---
|
||||
|
||||
## Root Cause Analysis
|
||||
|
||||
### Cause 1: Missing Repos Volume Mapping (Periphery)
|
||||
|
||||
**Issue:**
|
||||
Komodo Periphery containers require explicit volume mapping for the `/etc/komodo/repos` directory where Git repositories are cloned.
|
||||
|
||||
**Impact:**
|
||||
Without this mapping, the Periphery cannot access the Git repository files, causing "canonicalize" errors when attempting to validate the run directory path.
|
||||
|
||||
**Affected Configuration:**
|
||||
Any Komodo Periphery service deployed via Docker Compose that manages Git-linked stacks.
|
||||
|
||||
**Detection:**
|
||||
```bash
|
||||
# Check current mounts
|
||||
docker inspect komodo-periphery-{node-name} --format '{{range .Mounts}}{{.Source}} -> {{.Destination}}{{println}}{{end}}' | grep repos
|
||||
|
||||
# Expected output (if missing):
|
||||
(no output)
|
||||
|
||||
# Correct output:
|
||||
/mnt/appdata/komodo/{node-name}/repos -> /etc/komodo/repos
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Cause 2: Docker Image Tag Version Prefix
|
||||
|
||||
**Issue:**
|
||||
Docker image tags on Docker Hub typically **do not** include a `v` prefix, while Git tags commonly use the `v1.2.3` format.
|
||||
|
||||
**Impact:**
|
||||
Specifying an image as `app:v1.2.11` fails to pull because Docker Hub hosts the tag as `app:1.2.11` (without the `v`).
|
||||
|
||||
**Example:**
|
||||
```yaml
|
||||
# ❌ Incorrect - Fails to pull
|
||||
image: chrisbenincasa/tunarr:v1.2.11
|
||||
|
||||
# ✅ Correct - Pulls successfully
|
||||
image: chrisbenincasa/tunarr:1.2.11
|
||||
```
|
||||
|
||||
**Verification:**
|
||||
```bash
|
||||
# Query Docker Hub API for available tags
|
||||
curl -s "https://registry.hub.docker.com/v2/repositories/{user}/{image}/tags" | grep -oP '"name":"\K[^"]+'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Resolution
|
||||
|
||||
### Step 1: Add Repos Volume to Komodo Periphery
|
||||
|
||||
**File:** `/mnt/appdata/docker-compose.yaml` (or wherever Core stack is deployed)
|
||||
|
||||
**Add the following volume mapping to the Periphery service:**
|
||||
|
||||
```yaml
|
||||
periphery:
|
||||
image: ghcr.io/moghtech/komodo-periphery:2
|
||||
volumes:
|
||||
- /proc:/proc
|
||||
- /mnt/appdata/komodo/{node-name}/keys:/config/keys
|
||||
- /mnt/appdata/komodo/{node-name}/work:/etc/komodo
|
||||
- /mnt/appdata/komodo/repos/{node-name}:/etc/komodo/repos # Add this line
|
||||
```
|
||||
|
||||
**Create the repos directory on the host:**
|
||||
```bash
|
||||
mkdir -p /mnt/appdata/komodo/repos/{node-name}
|
||||
chown -R 1000:1000 /mnt/appdata/komodo/repos/{node-name}
|
||||
```
|
||||
|
||||
**Redeploy the Periphery container:**
|
||||
```bash
|
||||
cd /path/to/compose/file
|
||||
docker compose down periphery
|
||||
docker compose up -d periphery
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Step 2: Fix Docker Image Tag Prefix
|
||||
|
||||
**File:** `nodes/{node-name}/{service-name}/compose.yaml` (in Git repo)
|
||||
|
||||
**Remove the `v` prefix from image tags:**
|
||||
|
||||
```yaml
|
||||
services:
|
||||
app:
|
||||
image: user/app:1.2.11 # Changed from v1.2.11
|
||||
```
|
||||
|
||||
**Commit and push changes:**
|
||||
```bash
|
||||
git add .
|
||||
git commit -m "Fix: Remove 'v' prefix from Docker image tags"
|
||||
git push
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Step 3: Verify and Deploy
|
||||
|
||||
**Pull the updated repository on the Periphery node:**
|
||||
```bash
|
||||
docker exec komodo-periphery-{node-name} sh -c 'cd /etc/komodo/repos/{repo-name} && git pull'
|
||||
```
|
||||
|
||||
**In Komodo UI:**
|
||||
1. Navigate to **Stacks** → Select your stack
|
||||
2. Click **"Pull Stack"** button
|
||||
3. Verify success notification
|
||||
4. Click **"Deploy Stack"** button
|
||||
5. Confirm container is recreated with correct image
|
||||
|
||||
**Verification Commands:**
|
||||
```bash
|
||||
# Check if repos directory is mounted
|
||||
docker inspect komodo-periphery-{node-name} | grep "/etc/komodo/repos"
|
||||
|
||||
# Check running container image
|
||||
docker ps --filter 'name={container-name}' --format '{{.Image}}'
|
||||
|
||||
# Verify GPU access (if applicable)
|
||||
docker exec {container-name} nvidia-smi
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Additional Notes
|
||||
|
||||
### Known Issue: Komodo Pull Button Manual Workaround
|
||||
|
||||
If the **Pull Stack** button does not update the deployment file automatically:
|
||||
|
||||
**Temporary Workaround:**
|
||||
```bash
|
||||
# Manually copy file from Git repo to deployment location
|
||||
docker exec komodo-periphery-{node-name} cp \
|
||||
/etc/komodo/repos/{repo-name}/{path}/compose.yaml \
|
||||
/etc/komodo/stacks/{stack-name}/compose.yaml
|
||||
```
|
||||
|
||||
**This issue is under investigation.**
|
||||
|
||||
---
|
||||
|
||||
### Best Practices
|
||||
|
||||
1. **Always verify Docker image tags** before adding them to compose files:
|
||||
- Check Docker Hub directly
|
||||
- Use the API to list available tags
|
||||
- Prefer numeric tags over `latest` for production
|
||||
|
||||
2. **Standardize volume mappings** across all Komodo Periphery nodes:
|
||||
```yaml
|
||||
- /mnt/appdata/komodo/{node-name}/keys:/config/keys
|
||||
- /mnt/appdata/komodo/{node-name}/work:/etc/komodo
|
||||
- /mnt/appdata/komodo/repos/{node-name}:/etc/komodo/repos
|
||||
```
|
||||
|
||||
3. **Document Git tag to Docker tag conversions** when they differ:
|
||||
- Create a mapping reference file
|
||||
- Use CI/CD to validate tags before merge
|
||||
|
||||
---
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [TECHNICAL_RUNBOOK.md](TECHNICAL_RUNBOOK.md) - Infrastructure overview and emergency procedures
|
||||
- [LLM_HANDOVER.md](LLM_HANDOVER.md) - Quick-start context for AI sessions
|
||||
- Repository Memory: `/memories/repo/critical-fixes.md`
|
||||
|
||||
---
|
||||
|
||||
## Resolution Checklist
|
||||
|
||||
- [x] Added repos volume mapping to Komodo Periphery
|
||||
- [x] Fixed Docker image tag prefix (removed `v`)
|
||||
- [x] Verified Git repository accessible inside Periphery container
|
||||
- [x] Tested Pull Stack operation in Komodo UI
|
||||
- [x] Tested Deploy Stack operation in Komodo UI
|
||||
- [x] Confirmed container running with correct image version
|
||||
- [x] Verified GPU passthrough (if applicable)
|
||||
- [x] Updated repository memory with lessons learned
|
||||
|
||||
---
|
||||
|
||||
**Document Version:** 1.0
|
||||
**Validation Status:** Tested and Verified ✅
|
||||
Loading…
x
Reference in New Issue
Block a user