docs: Add KBA-001 for Komodo GitOps deployment troubleshooting
This commit is contained in:
parent
9eaceb5261
commit
0d19230ca1
245
documentation/KBA-001-Komodo-GitOps-Stack-Deployment-Failures.md
Normal file
245
documentation/KBA-001-Komodo-GitOps-Stack-Deployment-Failures.md
Normal file
@ -0,0 +1,245 @@
|
|||||||
|
# KBA-001: Komodo GitOps Stack Deployment Failures
|
||||||
|
|
||||||
|
**Status:** Resolved
|
||||||
|
**Created:** April 12, 2026
|
||||||
|
**Last Updated:** April 12, 2026
|
||||||
|
**Affected Systems:** Komodo Core v2.1.2, Komodo Periphery v2.1.2
|
||||||
|
**Severity:** High
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Summary
|
||||||
|
|
||||||
|
Git-linked Stacks in Komodo fail to pull and deploy with "canonicalize" errors or "image not found" errors despite valid Git repository configuration.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Problem Description
|
||||||
|
|
||||||
|
### Symptoms
|
||||||
|
|
||||||
|
When attempting to deploy a Docker Compose stack from a Git repository in Komodo:
|
||||||
|
|
||||||
|
1. **Pull Stack** operation fails with:
|
||||||
|
```
|
||||||
|
Failed to validate run directory on host after stack write (canonicalize error),
|
||||||
|
path=/etc/komodo/repos/homelab: No such file or directory (os error 2)
|
||||||
|
```
|
||||||
|
|
||||||
|
2. **Deploy Stack** operation fails with:
|
||||||
|
```
|
||||||
|
failed to resolve reference "docker.io/app:v1.2.11": not found
|
||||||
|
```
|
||||||
|
|
||||||
|
3. Stack deployment file (`/etc/komodo/stacks/{stack-name}/compose.yaml`) is not updated from Git repository source.
|
||||||
|
|
||||||
|
### Environment
|
||||||
|
|
||||||
|
- **Komodo Core:** v2.1.2 running on Heimdall (10.0.0.151)
|
||||||
|
- **Komodo Periphery:** v2.1.2 running on Waldorf (10.0.0.251)
|
||||||
|
- **Git Provider:** Gitea (self-hosted)
|
||||||
|
- **Repository:** `https://git.castaldifamily.com/nathan/homelab.git`
|
||||||
|
- **Stack Configuration:**
|
||||||
|
- Type: Git Repo
|
||||||
|
- Repo: `homelab`
|
||||||
|
- Branch: `main`
|
||||||
|
- Run Directory: `nodes/waldorf/tunarr`
|
||||||
|
- File Paths: (empty/blank)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Root Cause Analysis
|
||||||
|
|
||||||
|
### Cause 1: Missing Repos Volume Mapping (Periphery)
|
||||||
|
|
||||||
|
**Issue:**
|
||||||
|
Komodo Periphery containers require explicit volume mapping for the `/etc/komodo/repos` directory where Git repositories are cloned.
|
||||||
|
|
||||||
|
**Impact:**
|
||||||
|
Without this mapping, the Periphery cannot access the Git repository files, causing "canonicalize" errors when attempting to validate the run directory path.
|
||||||
|
|
||||||
|
**Affected Configuration:**
|
||||||
|
Any Komodo Periphery service deployed via Docker Compose that manages Git-linked stacks.
|
||||||
|
|
||||||
|
**Detection:**
|
||||||
|
```bash
|
||||||
|
# Check current mounts
|
||||||
|
docker inspect komodo-periphery-{node-name} --format '{{range .Mounts}}{{.Source}} -> {{.Destination}}{{println}}{{end}}' | grep repos
|
||||||
|
|
||||||
|
# Expected output (if missing):
|
||||||
|
(no output)
|
||||||
|
|
||||||
|
# Correct output:
|
||||||
|
/mnt/appdata/komodo/{node-name}/repos -> /etc/komodo/repos
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Cause 2: Docker Image Tag Version Prefix
|
||||||
|
|
||||||
|
**Issue:**
|
||||||
|
Docker image tags on Docker Hub typically **do not** include a `v` prefix, while Git tags commonly use the `v1.2.3` format.
|
||||||
|
|
||||||
|
**Impact:**
|
||||||
|
Specifying an image as `app:v1.2.11` fails to pull because Docker Hub hosts the tag as `app:1.2.11` (without the `v`).
|
||||||
|
|
||||||
|
**Example:**
|
||||||
|
```yaml
|
||||||
|
# ❌ Incorrect - Fails to pull
|
||||||
|
image: chrisbenincasa/tunarr:v1.2.11
|
||||||
|
|
||||||
|
# ✅ Correct - Pulls successfully
|
||||||
|
image: chrisbenincasa/tunarr:1.2.11
|
||||||
|
```
|
||||||
|
|
||||||
|
**Verification:**
|
||||||
|
```bash
|
||||||
|
# Query Docker Hub API for available tags
|
||||||
|
curl -s "https://registry.hub.docker.com/v2/repositories/{user}/{image}/tags" | grep -oP '"name":"\K[^"]+'
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Resolution
|
||||||
|
|
||||||
|
### Step 1: Add Repos Volume to Komodo Periphery
|
||||||
|
|
||||||
|
**File:** `/mnt/appdata/docker-compose.yaml` (or wherever Core stack is deployed)
|
||||||
|
|
||||||
|
**Add the following volume mapping to the Periphery service:**
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
periphery:
|
||||||
|
image: ghcr.io/moghtech/komodo-periphery:2
|
||||||
|
volumes:
|
||||||
|
- /proc:/proc
|
||||||
|
- /mnt/appdata/komodo/{node-name}/keys:/config/keys
|
||||||
|
- /mnt/appdata/komodo/{node-name}/work:/etc/komodo
|
||||||
|
- /mnt/appdata/komodo/repos/{node-name}:/etc/komodo/repos # Add this line
|
||||||
|
```
|
||||||
|
|
||||||
|
**Create the repos directory on the host:**
|
||||||
|
```bash
|
||||||
|
mkdir -p /mnt/appdata/komodo/repos/{node-name}
|
||||||
|
chown -R 1000:1000 /mnt/appdata/komodo/repos/{node-name}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Redeploy the Periphery container:**
|
||||||
|
```bash
|
||||||
|
cd /path/to/compose/file
|
||||||
|
docker compose down periphery
|
||||||
|
docker compose up -d periphery
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Step 2: Fix Docker Image Tag Prefix
|
||||||
|
|
||||||
|
**File:** `nodes/{node-name}/{service-name}/compose.yaml` (in Git repo)
|
||||||
|
|
||||||
|
**Remove the `v` prefix from image tags:**
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
services:
|
||||||
|
app:
|
||||||
|
image: user/app:1.2.11 # Changed from v1.2.11
|
||||||
|
```
|
||||||
|
|
||||||
|
**Commit and push changes:**
|
||||||
|
```bash
|
||||||
|
git add .
|
||||||
|
git commit -m "Fix: Remove 'v' prefix from Docker image tags"
|
||||||
|
git push
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Step 3: Verify and Deploy
|
||||||
|
|
||||||
|
**Pull the updated repository on the Periphery node:**
|
||||||
|
```bash
|
||||||
|
docker exec komodo-periphery-{node-name} sh -c 'cd /etc/komodo/repos/{repo-name} && git pull'
|
||||||
|
```
|
||||||
|
|
||||||
|
**In Komodo UI:**
|
||||||
|
1. Navigate to **Stacks** → Select your stack
|
||||||
|
2. Click **"Pull Stack"** button
|
||||||
|
3. Verify success notification
|
||||||
|
4. Click **"Deploy Stack"** button
|
||||||
|
5. Confirm container is recreated with correct image
|
||||||
|
|
||||||
|
**Verification Commands:**
|
||||||
|
```bash
|
||||||
|
# Check if repos directory is mounted
|
||||||
|
docker inspect komodo-periphery-{node-name} | grep "/etc/komodo/repos"
|
||||||
|
|
||||||
|
# Check running container image
|
||||||
|
docker ps --filter 'name={container-name}' --format '{{.Image}}'
|
||||||
|
|
||||||
|
# Verify GPU access (if applicable)
|
||||||
|
docker exec {container-name} nvidia-smi
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Additional Notes
|
||||||
|
|
||||||
|
### Known Issue: Komodo Pull Button Manual Workaround
|
||||||
|
|
||||||
|
If the **Pull Stack** button does not update the deployment file automatically:
|
||||||
|
|
||||||
|
**Temporary Workaround:**
|
||||||
|
```bash
|
||||||
|
# Manually copy file from Git repo to deployment location
|
||||||
|
docker exec komodo-periphery-{node-name} cp \
|
||||||
|
/etc/komodo/repos/{repo-name}/{path}/compose.yaml \
|
||||||
|
/etc/komodo/stacks/{stack-name}/compose.yaml
|
||||||
|
```
|
||||||
|
|
||||||
|
**This issue is under investigation.**
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Best Practices
|
||||||
|
|
||||||
|
1. **Always verify Docker image tags** before adding them to compose files:
|
||||||
|
- Check Docker Hub directly
|
||||||
|
- Use the API to list available tags
|
||||||
|
- Prefer numeric tags over `latest` for production
|
||||||
|
|
||||||
|
2. **Standardize volume mappings** across all Komodo Periphery nodes:
|
||||||
|
```yaml
|
||||||
|
- /mnt/appdata/komodo/{node-name}/keys:/config/keys
|
||||||
|
- /mnt/appdata/komodo/{node-name}/work:/etc/komodo
|
||||||
|
- /mnt/appdata/komodo/repos/{node-name}:/etc/komodo/repos
|
||||||
|
```
|
||||||
|
|
||||||
|
3. **Document Git tag to Docker tag conversions** when they differ:
|
||||||
|
- Create a mapping reference file
|
||||||
|
- Use CI/CD to validate tags before merge
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Related Documentation
|
||||||
|
|
||||||
|
- [TECHNICAL_RUNBOOK.md](TECHNICAL_RUNBOOK.md) - Infrastructure overview and emergency procedures
|
||||||
|
- [LLM_HANDOVER.md](LLM_HANDOVER.md) - Quick-start context for AI sessions
|
||||||
|
- Repository Memory: `/memories/repo/critical-fixes.md`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Resolution Checklist
|
||||||
|
|
||||||
|
- [x] Added repos volume mapping to Komodo Periphery
|
||||||
|
- [x] Fixed Docker image tag prefix (removed `v`)
|
||||||
|
- [x] Verified Git repository accessible inside Periphery container
|
||||||
|
- [x] Tested Pull Stack operation in Komodo UI
|
||||||
|
- [x] Tested Deploy Stack operation in Komodo UI
|
||||||
|
- [x] Confirmed container running with correct image version
|
||||||
|
- [x] Verified GPU passthrough (if applicable)
|
||||||
|
- [x] Updated repository memory with lessons learned
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
**Document Version:** 1.0
|
||||||
|
**Validation Status:** Tested and Verified ✅
|
||||||
Loading…
x
Reference in New Issue
Block a user