From a9341175995a996edd61e607483c1cff1606f051 Mon Sep 17 00:00:00 2001
From: nathan <nathan@castaldifamily.com>
Date: Sun, 12 Apr 2026 01:00:56 -0400
Subject: [PATCH] feat(documentation): add migration guide for Git-crypt secret
 management

---
 documentation/plans/plan-gitcryptMigration.md | 663 ++++++++++++++++++
 1 file changed, 663 insertions(+)
 create mode 100644 documentation/plans/plan-gitcryptMigration.md

diff --git a/documentation/plans/plan-gitcryptMigration.md b/documentation/plans/plan-gitcryptMigration.md
new file mode 100644
index 0000000..a5e40e1
--- /dev/null
+++ b/documentation/plans/plan-gitcryptMigration.md
@@ -0,0 +1,663 @@
+# Migration Guide: Git-crypt for Secret Management
+
+## Overview
+
+Implement Git-crypt to encrypt sensitive `.env` files in the homelab repository, enabling safe commit of secrets while maintaining seamless integration with Komodo's GitOps workflow.
+
+**Goal:** Zero workflow changes for Komodo, encrypted secrets in Git, transparent decryption on pull.
+
+---
+
+## Prerequisites
+
+- [ ] SSH access to all Komodo nodes (Heimdall, Waldorf, Watchtower)
+- [ ] Git-crypt installed on local machine
+- [ ] Ability to push to Gitea repository
+- [ ] Current `.gitignore` already excludes `.env.secrets` (will be removed)
+
+---
+
+## Phase 1: Local Setup (Your Workstation)
+
+### Step 1: Install Git-crypt
+
+**Windows (via Git Bash):**
+```bash
+# Download latest release
+curl -L https://github.com/AGWA/git-crypt/releases/download/0.7.0/git-crypt-0.7.0-x86_64.exe -o git-crypt.exe
+sudo mv git-crypt.exe /usr/local/bin/git-crypt
+chmod +x /usr/local/bin/git-crypt
+```
+
+**Or via Homebrew (if using WSL/MacOS):**
+```bash
+brew install git-crypt
+```
+
+**Verify:**
+```bash
+git-crypt --version
+# Expected: git-crypt 0.7.0
+```
+
+---
+
+### Step 2: Initialize Git-crypt in Repository
+
+```bash
+cd ~/homelab
+
+# Initialize git-crypt
+git-crypt init
+
+# Export the symmetric key (CRITICAL - SAVE THIS SECURELY)
+git-crypt export-key ~/homelab-secrets.key
+
+# IMPORTANT: Back this key up in multiple secure locations:
+# - Password manager (1Password, Bitwarden)
+# - Encrypted USB drive
+# - Secure cloud storage (encrypted)
+```
+
+---
+
+### Step 3: Configure Encryption Rules
+
+Create `.gitattributes` in repository root:
+
+```bash
+cat > .gitattributes <<'EOF'
+# Git-crypt Encryption Rules
+# Encrypt all .env.secrets files across the repository
+**/.env.secrets filter=git-crypt diff=git-crypt
+*.env.secrets filter=git-crypt diff=git-crypt
+
+# Encrypt the key itself if accidentally added
+*.key filter=git-crypt diff=git-crypt
+
+# Encrypt specific config files (optional)
+# **/secrets.yml filter=git-crypt diff=git-crypt
+EOF
+
+git add .gitattributes
+git commit -m "chore(security): configure git-crypt encryption rules"
+```
+
+---
+
+### Step 4: Update .gitignore
+
+**Remove** `.env.secrets` from `.gitignore` since they'll now be encrypted:
+
+```bash
+# Edit .gitignore - remove these lines:
+# **/.env.secrets
+# **/*.env.secrets
+
+# But KEEP these:
+# **/.env.local
+# *.key  (prevent accidental key commit)
+```
+
+Update `.gitignore`:
+```bash
+# Environment variables and secrets
+# NOTE: .env.secrets are now ENCRYPTED via git-crypt, safe to commit
+**/.env.local
+.env.local
+
+# Git-crypt keys (NEVER commit these)
+*.key
+homelab-secrets.key
+
+# Temporary unencrypted files
+**/.env.secrets.decrypted
+```
+
+---
+
+### Step 5: Create Encrypted Secret Files
+
+**For Plex (Waldorf):**
+
+```bash
+# Create encrypted file
+cat > nodes/waldorf/plex/.env.secrets <<'EOF'
+# Plex Configuration Secrets
+PLEX_CLAIM=claim-sxFpsPTDzzF-9RZAxtUL
+EOF
+
+# Verify it will be encrypted
+git-crypt status nodes/waldorf/plex/.env.secrets
+# Expected output: encrypted: nodes/waldorf/plex/.env.secrets
+```
+
+**For Traefik (Heimdall):**
+
+```bash
+cat > nodes/heimdall/core/.env.secrets <<'EOF'
+# Cloudflare API Credentials
+CF_API_TOKEN=your_cloudflare_api_token_here
+CF_ZONE_TOKEN=your_cloudflare_zone_token_here
+
+# Komodo Database
+KOMODO_DATABASE_USERNAME=komodo_admin
+KOMODO_DATABASE_PASSWORD=your_database_password_here
+KOMODO_ONBOARDING_KEY_HEIMDALL=your_onboarding_key_here
+
+# Redis Password
+REDIS_PASSWORD=your_redis_password_here
+EOF
+
+git-crypt status nodes/heimdall/core/.env.secrets
+# Expected: encrypted: nodes/heimdall/core/.env.secrets
+```
+
+---
+
+### Step 6: Test Encryption Locally
+
+```bash
+# Check encryption status
+git-crypt status
+
+# Expected output:
+# encrypted: nodes/waldorf/plex/.env.secrets
+# encrypted: nodes/heimdall/core/.env.secrets
+
+# View file (should be readable on your machine)
+cat nodes/waldorf/plex/.env.secrets
+# You should see plaintext
+
+# Lock the repository to simulate what Git sees
+git-crypt lock
+
+# Try to read again
+cat nodes/waldorf/plex/.env.secrets
+# You should see binary garbage (encrypted)
+
+# Unlock to continue working
+git-crypt unlock
+
+# Or unlock with specific key
+git-crypt unlock ~/homelab-secrets.key
+```
+
+---
+
+### Step 7: Commit Encrypted Secrets
+
+```bash
+# Stage encrypted files
+git add nodes/waldorf/plex/.env.secrets
+git add nodes/heimdall/core/.env.secrets
+git add .gitattributes
+
+# Verify they're encrypted in staging
+git show :nodes/waldorf/plex/.env.secrets
+# Should show binary data, NOT plaintext
+
+# Commit
+git commit -m "chore(security): add encrypted secrets via git-crypt
+
+- nodes/waldorf/plex/.env.secrets: Plex claim token
+- nodes/heimdall/core/.env.secrets: Cloudflare, Komodo, Redis credentials
+- Safe to commit (encrypted with git-crypt)"
+
+# Push to Gitea
+git push origin main
+```
+
+---
+
+## Phase 2: Node Setup (Komodo Deployment Targets)
+
+### Step 8: Distribute Key to Komodo Nodes
+
+**SECURITY NOTE:** Use secure methods to transfer the key (not email, not Slack).
+
+**Option A: SCP (Secure Copy)**
+
+```bash
+# Copy key to Heimdall
+scp ~/homelab-secrets.key chester@10.0.0.151:~/
+
+# Copy key to Waldorf
+scp ~/homelab-secrets.key chester@10.0.0.251:~/
+
+# Copy key to Watchtower
+scp ~/homelab-secrets.key chester@10.0.0.200:~/
+```
+
+**Option B: Manual Transfer via USB**
+- Copy key to USB drive
+- SSH to each node
+- Transfer from USB to home directory
+
+---
+
+### Step 9: Install Git-crypt on Nodes
+
+**On each node (Heimdall, Waldorf, Watchtower):**
+
+```bash
+# SSH to node
+ssh chester@10.0.0.151  # Repeat for .251 and .200
+
+# Install git-crypt
+sudo apt update
+sudo apt install git-crypt -y
+
+# Verify installation
+git-crypt --version
+```
+
+---
+
+### Step 10: Unlock Repositories on Nodes
+
+**Critical:** This must be done in Komodo's repo directories, not just any clone.
+
+**On Heimdall:**
+
+```bash
+ssh chester@10.0.0.151
+
+# Navigate to Komodo's repo directory
+cd /etc/komodo/repos/homelab
+
+# Unlock with the key
+git-crypt unlock ~/homelab-secrets.key
+
+# Verify decryption worked
+cat nodes/heimdall/core/.env.secrets
+# Should show plaintext secrets
+
+# Pull to test
+git pull origin main
+# Secrets should auto-decrypt on pull
+
+# Secure the key
+chmod 600 ~/homelab-secrets.key
+```
+
+**On Waldorf:**
+
+```bash
+ssh chester@10.0.0.251
+
+# Find Komodo periphery repo path
+cd /etc/komodo/repos/homelab  # Or wherever Komodo clones to
+
+git-crypt unlock ~/homelab-secrets.key
+
+# Verify
+cat nodes/waldorf/plex/.env.secrets
+
+chmod 600 ~/homelab-secrets.key
+```
+
+**On Watchtower:**
+
+```bash
+ssh chester@10.0.0.200
+
+cd /etc/komodo/repos/homelab
+
+git-crypt unlock ~/homelab-secrets.key
+cat nodes/watchtower/*/..env.secrets  # If any exist
+
+chmod 600 ~/homelab-secrets.key
+```
+
+---
+
+## Phase 3: Update Compose Files
+
+### Step 11: Reference Encrypted Secret Files
+
+**Example: Plex (Waldorf)**
+
+Update `nodes/waldorf/plex/compose.yaml`:
+
+```yaml
+services:
+  plex:
+    image: lscr.io/linuxserver/plex:latest
+    container_name: plex
+    network_mode: host
+    restart: unless-stopped
+    env_file:
+      - .env.secrets  # Now encrypted in Git!
+    environment:
+      - PUID=1000
+      - PGID=1000
+      - TZ=America/New_York
+      - VERSION=docker
+      # PLEX_CLAIM loaded from .env.secrets
+    volumes:
+      - /mnt/appdata/plex:/config
+      - /mnt/media/tvshows:/tv
+      - /mnt/media/movies:/movies
+    deploy:
+      resources:
+        reservations:
+          devices:
+            - driver: nvidia
+              count: 1
+              capabilities: [gpu]
+```
+
+**Remove hardcoded secrets:**
+
+```diff
+- environment:
+-   - PLEX_CLAIM=claim-sxFpsPTDzzF-9RZAxtUL
+```
+
+**Example: Traefik (Heimdall)**
+
+Update `nodes/heimdall/core/compose.yaml`:
+
+```yaml
+services:
+  traefik:
+    image: traefik:v3.6.5
+    env_file:
+      - .env.secrets
+    environment:
+      - DOCKER_HOST=tcp://docker-socket-proxy:2375
+      # These are now loaded from .env.secrets:
+      # - CLOUDFLARE_DNS_API_TOKEN
+      # - CLOUDFLARE_ZONE_API_TOKEN
+```
+
+---
+
+### Step 12: Commit Compose Updates
+
+```bash
+git add nodes/waldorf/plex/compose.yaml
+git add nodes/heimdall/core/compose.yaml
+
+git commit -m "refactor(security): migrate secrets to encrypted .env files
+
+- Removed hardcoded PLEX_CLAIM from compose.yaml
+- Removed hardcoded Cloudflare tokens
+- Now loaded from git-crypt encrypted .env.secrets files"
+
+git push origin main
+```
+
+---
+
+## Phase 4: Testing & Validation
+
+### Step 13: Test Automated Deployment
+
+**Trigger a deployment via Komodo:**
+
+1. Make a minor change to a compose file (e.g., add a comment)
+2. Commit and push to Gitea
+3. Webhook triggers Komodo
+4. Komodo pulls repo (git-crypt auto-decrypts)
+5. Komodo deploys stack with decrypted secrets
+
+**Verify:**
+
+```bash
+# On Waldorf
+ssh chester@10.0.0.251
+docker exec plex env | grep PLEX_CLAIM
+# Should show the actual claim token (not placeholder)
+
+# Check container logs for errors
+docker logs plex --tail 50
+```
+
+---
+
+### Step 14: Test Secret Rotation
+
+**Scenario: Update Plex claim token**
+
+```bash
+# On local machine
+cd ~/homelab
+
+# Edit encrypted file (git-crypt auto-decrypts for you)
+nano nodes/waldorf/plex/.env.secrets
+# Change PLEX_CLAIM value
+
+# Commit
+git add nodes/waldorf/plex/.env.secrets
+git commit -m "chore(plex): rotate claim token"
+git push
+
+# Komodo auto-deploys with new secret
+```
+
+**Verify on Gitea:**
+- View the file in Gitea web UI
+- Should show binary/encrypted content (not plaintext)
+
+---
+
+## Phase 5: Security Hardening
+
+### Step 15: Secure the Keys
+
+**On each node:**
+
+```bash
+# Move key to more secure location
+sudo mkdir -p /etc/git-crypt
+sudo mv ~/homelab-secrets.key /etc/git-crypt/homelab.key
+sudo chmod 400 /etc/git-crypt/homelab.key
+sudo chown root:root /etc/git-crypt/homelab.key
+
+# Update unlock command for future use
+cd /etc/komodo/repos/homelab
+sudo git-crypt unlock /etc/git-crypt/homelab.key
+```
+
+**Backup Strategy:**
+
+```bash
+# Create encrypted backup of key
+gpg --symmetric --cipher-algo AES256 ~/homelab-secrets.key
+# Save homelab-secrets.key.gpg to password manager
+
+# Or use age
+age -p ~/homelab-secrets.key > homelab-secrets.key.age
+# Save .age file to secure storage
+```
+
+---
+
+### Step 16: Document Key Access
+
+Create `documentation/SECURITY_KEY_MANAGEMENT.md`:
+
+```markdown
+# Secret Key Management
+
+## Git-crypt Key Location
+
+**Production Nodes:**
+- Heimdall: `/etc/git-crypt/homelab.key`
+- Waldorf: `/etc/git-crypt/homelab.key`
+- Watchtower: `/etc/git-crypt/homelab.key`
+
+**Backup Locations:**
+- Primary: Password manager (encrypted)
+- Secondary: Encrypted USB drive (physical safe)
+- Tertiary: NAS encrypted backup
+
+## Key Recovery Procedure
+
+If a node loses git-crypt unlock state:
+
+1. SSH to node
+2. Navigate to `/etc/komodo/repos/homelab`
+3. Run: `sudo git-crypt unlock /etc/git-crypt/homelab.key`
+4. Verify: `cat nodes/{node}/{stack}/.env.secrets`
+
+## Key Rotation
+
+**Frequency:** Annually or after security incident
+
+**Process:**
+1. Generate new git-crypt key: `git-crypt init`
+2. Export new key: `git-crypt export-key ~/new-key`
+3. Re-encrypt all files: `git-crypt rotate-key ~/new-key`
+4. Distribute new key to all nodes
+5. Securely destroy old key
+```
+
+---
+
+## Troubleshooting
+
+### Issue: "File is not encrypted" after push
+
+**Cause:** `.gitattributes` not committed before files
+
+**Fix:**
+```bash
+git rm --cached nodes/waldorf/plex/.env.secrets
+git add .gitattributes
+git commit -m "Add encryption rules first"
+git add nodes/waldorf/plex/.env.secrets
+git commit -m "Add encrypted secrets"
+```
+
+---
+
+### Issue: Can't read secrets on node
+
+**Cause:** Repository not unlocked
+
+**Fix:**
+```bash
+cd /etc/komodo/repos/homelab
+git-crypt unlock ~/homelab-secrets.key
+
+# Verify
+git-crypt status
+```
+
+---
+
+### Issue: Secrets showing as plaintext in Gitea
+
+**Cause:** Git-crypt not configured server-side (this is EXPECTED)
+
+**Note:** Gitea displays raw Git objects. View the actual commit:
+```bash
+git show HEAD:nodes/waldorf/plex/.env.secrets
+# Should be binary garbage
+```
+
+---
+
+### Issue: Merge conflict in encrypted file
+
+**Fix:**
+```bash
+# Decrypt both versions
+git show HEAD:.env.secrets > .env.secrets.ours
+git show MERGE_HEAD:.env.secrets > .env.secrets.theirs
+
+# Manually merge
+nano .env.secrets
+
+# Re-encrypt
+git add .env.secrets
+git commit
+```
+
+---
+
+## Migration Checklist
+
+**Pre-Migration:**
+- [ ] Backup current secrets (screenshot Komodo UI environment variables)
+- [ ] Test git-crypt on dummy repo first
+- [ ] Verify all nodes have `git-crypt` installed
+
+**Migration Steps:**
+- [ ] Initialize git-crypt locally
+- [ ] Export and secure key
+- [ ] Configure `.gitattributes`
+- [ ] Create encrypted `.env.secrets` files
+- [ ] Test encryption locally (`git-crypt lock/unlock`)
+- [ ] Commit and push encrypted files
+- [ ] Distribute key to all nodes
+- [ ] Unlock repositories on each node
+- [ ] Update compose files to use `env_file`
+- [ ] Remove hardcoded secrets from compose files
+- [ ] Test deployment via Komodo webhook
+- [ ] Verify containers can read secrets
+- [ ] Document key locations
+- [ ] Delete unencrypted secret backups
+
+**Post-Migration:**
+- [ ] Update `.gitignore` to allow `.env.secrets`
+- [ ] Remove secrets from Komodo UI (optional - can keep as backup)
+- [ ] Update [SECURITY_AUDIT_REPORT.md](../documentation/SECURITY_AUDIT_REPORT.md)
+- [ ] Create SOP for secret rotation
+- [ ] Test disaster recovery (unlock after simulated node failure)
+
+---
+
+## Rollback Plan
+
+If git-crypt causes issues:
+
+```bash
+# 1. Remove encrypted files from Git
+git rm nodes/**/.env.secrets
+git commit -m "Rollback: Remove git-crypt secrets"
+
+# 2. Re-create .env.secrets files locally (gitignored)
+# 3. Manually copy to nodes via SCP
+# 4. Reset to Komodo UI environment variables
+
+# 5. Remove git-crypt config
+git-crypt deinit  # If this command exists
+# Or manually remove .git-crypt directory
+```
+
+---
+
+## Next Steps After Migration
+
+1. **Create SOP-002:** "Secret Rotation Procedure"
+2. **Automate key backup:** Add to NAS backup schedule
+3. **Monitor:** Set calendar reminder for annual key rotation
+4. **Scale:** Apply same pattern to other repositories
+5. **Enhance:** Consider adding GPG user keys for team access
+
+---
+
+## Related Documentation
+
+- [SECURITY_AUDIT_REPORT.md](../documentation/SECURITY_AUDIT_REPORT.md) - Initial security findings
+- [SOP-001](../documentation/SOPs/SOP-001-Migrate-Stack-from-UI-to-Git.md) - Secrets management section
+- Git-crypt Documentation: https://github.com/AGWA/git-crypt
+
+---
+
+## Success Criteria
+
+- ✅ All secrets encrypted in Git repository
+- ✅ Komodo auto-deploy works without changes
+- ✅ No plaintext secrets visible in Gitea web UI
+- ✅ Containers can read secrets from mounted files
+- ✅ Key securely backed up in multiple locations
+- ✅ Secret rotation tested and documented
+
+**Estimated Migration Time:** 2-3 hours (including testing)
+
+**Maintenance:** Near-zero (transparent after initial setup)