homelab/documentation/plans/plan-gitcryptMigration.md

Migration Guide: Git-crypt for Secret Management

Overview

Implement Git-crypt to encrypt sensitive .env files in the homelab repository, enabling safe commit of secrets while maintaining seamless integration with Komodo's GitOps workflow.

Goal: Zero workflow changes for Komodo, encrypted secrets in Git, transparent decryption on pull.

Estimated Time to Complete: 2-3 hours (first-time setup) | 1-1.5 hours (experienced operator)

---

Time Breakdown by Phase

Phase	Description	Time Estimate
Phase 1	Local Setup (Workstation)	30-40 minutes
Phase 2	Node Setup (Komodo Targets)	25-35 minutes
Phase 3	Update Compose Files	15-20 minutes
Phase 4	Testing & Validation	30-40 minutes
Phase 5	Security Hardening	20-30 minutes
Total	End-to-End Migration	2-3 hours

---

Prerequisites

• SSH access to all Komodo nodes (Heimdall, Waldorf, Watchtower)
• Git-crypt installed on local machine
• Ability to push to Gitea repository
• Current .gitignore already excludes .env.secrets (will be removed)

---

Phase 1: Local Setup (Your Workstation)

Estimated Time: 30-40 minutes

Step 1: Install Git-crypt

Time: 3-5 minutes

Windows (via Git Bash):

# Download latest release
curl -L https://github.com/AGWA/git-crypt/releases/download/0.7.0/git-crypt-0.7.0-x86_64.exe -o git-crypt.exe
sudo mv git-crypt.exe /usr/local/bin/git-crypt
chmod +x /usr/local/bin/git-crypt

Or via Homebrew (if using WSL/MacOS):

brew install git-crypt

Verify:

git-crypt --version
# Expected: git-crypt 0.7.0

---

Step 2: Initialize Git-crypt in Repository

Time: 3-5 minutes

cd ~/homelab

# Initialize git-crypt
git-crypt init

# Export the symmetric key (CRITICAL - SAVE THIS SECURELY)
git-crypt export-key ~/homelab-secrets.key

# IMPORTANT: Back this key up in multiple secure locations:
# - Password manager (1Password, Bitwarden)
# - Encrypted USB drive
# - Secure cloud storage (encrypted)

---

Step 3: Configure Encryption Rules

Time: 5-7 minutes

Create .gitattributes in repository root:

cat > .gitattributes <<'EOF'
# Git-crypt Encryption Rules
# Encrypt all .env.secrets files across the repository
**/.env.secrets filter=git-crypt diff=git-crypt
*.env.secrets filter=git-crypt diff=git-crypt

# Encrypt the key itself if accidentally added
*.key filter=git-crypt diff=git-crypt

# Encrypt specific config files (optional)
# **/secrets.yml filter=git-crypt diff=git-crypt
EOF

git add .gitattributes
git commit -m "chore(security): configure git-crypt encryption rules"

---

Step 4: Update .gitignore

Time: 3-5 minutes

Remove .env.secrets from .gitignore since they'll now be encrypted:

# Edit .gitignore - remove these lines:
# **/.env.secrets
# **/*.env.secrets

# But KEEP these:
# **/.env.local
# *.key  (prevent accidental key commit)

Update .gitignore:

# Environment variables and secrets
# NOTE: .env.secrets are now ENCRYPTED via git-crypt, safe to commit
**/.env.local
.env.local

# Git-crypt keys (NEVER commit these)
*.key
homelab-secrets.key

# Temporary unencrypted files
**/.env.secrets.decrypted

---

Step 5: Create Encrypted Secret Files

Time: 8-10 minutes

For Plex (Waldorf):

# Create encrypted file
cat > nodes/waldorf/plex/.env.secrets <<'EOF'
# Plex Configuration Secrets
PLEX_CLAIM=claim-sxFpsPTDzzF-9RZAxtUL
EOF

# Verify it will be encrypted
git-crypt status nodes/waldorf/plex/.env.secrets
# Expected output: encrypted: nodes/waldorf/plex/.env.secrets

For Traefik (Heimdall):

cat > nodes/heimdall/core/.env.secrets <<'EOF'
# Cloudflare API Credentials
CF_API_TOKEN=your_cloudflare_api_token_here
CF_ZONE_TOKEN=your_cloudflare_zone_token_here

# Komodo Database
KOMODO_DATABASE_USERNAME=komodo_admin
KOMODO_DATABASE_PASSWORD=your_database_password_here
KOMODO_ONBOARDING_KEY_HEIMDALL=your_onboarding_key_here

# Redis Password
REDIS_PASSWORD=your_redis_password_here
EOF

git-crypt status nodes/heimdall/core/.env.secrets
# Expected: encrypted: nodes/heimdall/core/.env.secrets

---

Step 6: Test Encryption Locally

Time: 5-7 minutes

# Check encryption status
git-crypt status

# Expected output:
# encrypted: nodes/waldorf/plex/.env.secrets
# encrypted: nodes/heimdall/core/.env.secrets

# View file (should be readable on your machine)
cat nodes/waldorf/plex/.env.secrets
# You should see plaintext

# Lock the repository to simulate what Git sees
git-crypt lock

# Try to read again
cat nodes/waldorf/plex/.env.secrets
# You should see binary garbage (encrypted)

# Unlock to continue working
git-crypt unlock

# Or unlock with specific key
git-crypt unlock ~/homelab-secrets.key

---

Step 7: Commit Encrypted Secrets

Time: 3-5 minutes

# Stage encrypted files
git add nodes/waldorf/plex/.env.secrets
git add nodes/heimdall/core/.env.secrets
git add .gitattributes

# Verify they're encrypted in staging
git show :nodes/waldorf/plex/.env.secrets
# Should show binary data, NOT plaintext

# Commit
git commit -m "chore(security): add encrypted secrets via git-crypt

- nodes/waldorf/plex/.env.secrets: Plex claim token
- nodes/heimdall/core/.env.secrets: Cloudflare, Komodo, Redis credentials
- Safe to commit (encrypted with git-crypt)"

# Push to Gitea
git push origin main

---

Phase 2: Node Setup (Komodo Deployment Targets)

Estimated Time: 25-35 minutes

Step 8: Distribute Key to Komodo Nodes

Time: 5-8 minutes

SECURITY NOTE: Use secure methods to transfer the key (not email, not Slack).

Option A: SCP (Secure Copy)

# Copy key to Heimdall
scp ~/homelab-secrets.key chester@10.0.0.151:~/

# Copy key to Waldorf
scp ~/homelab-secrets.key chester@10.0.0.251:~/

# Copy key to Watchtower
scp ~/homelab-secrets.key chester@10.0.0.200:~/

Option B: Manual Transfer via USB

• Copy key to USB drive
• SSH to each node
• Transfer from USB to home directory

---

Step 9: Install Git-crypt on Nodes

Time: 10-15 minutes (across all 3 nodes)

On each node (Heimdall, Waldorf, Watchtower):

# SSH to node
ssh chester@10.0.0.151  # Repeat for .251 and .200

# Install git-crypt
sudo apt update
sudo apt install git-crypt -y

# Verify installation
git-crypt --version

---

Step 10: Unlock Repositories on Nodes

Time: 10-12 minutes (across all 3 nodes)

Critical: This must be done in Komodo's repo directories, not just any clone.

On Heimdall:

ssh chester@10.0.0.151

# Navigate to Komodo's repo directory
cd /etc/komodo/repos/homelab

# Unlock with the key
git-crypt unlock ~/homelab-secrets.key

# Verify decryption worked
cat nodes/heimdall/core/.env.secrets
# Should show plaintext secrets

# Pull to test
git pull origin main
# Secrets should auto-decrypt on pull

# Secure the key
chmod 600 ~/homelab-secrets.key

On Waldorf:

ssh chester@10.0.0.251

# Find Komodo periphery repo path
cd /etc/komodo/repos/homelab  # Or wherever Komodo clones to

git-crypt unlock ~/homelab-secrets.key

# Verify
cat nodes/waldorf/plex/.env.secrets

chmod 600 ~/homelab-secrets.key

On Watchtower:

ssh chester@10.0.0.200

cd /etc/komodo/repos/homelab

git-crypt unlock ~/homelab-secrets.key
cat nodes/watchtower/*/..env.secrets  # If any exist

chmod 600 ~/homelab-secrets.key

---

Phase 3: Update Compose Files

Estimated Time: 15-20 minutes

Step 11: Reference Encrypted Secret Files

Time: 10-12 minutes

Example: Plex (Waldorf)

Update nodes/waldorf/plex/compose.yaml:

services:
  plex:
    image: lscr.io/linuxserver/plex:latest
    container_name: plex
    network_mode: host
    restart: unless-stopped
    env_file:
      - .env.secrets  # Now encrypted in Git!
    environment:
      - PUID=1000
      - PGID=1000
      - TZ=America/New_York
      - VERSION=docker
      # PLEX_CLAIM loaded from .env.secrets
    volumes:
      - /mnt/appdata/plex:/config
      - /mnt/media/tvshows:/tv
      - /mnt/media/movies:/movies
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

Remove hardcoded secrets:

- environment:
-   - PLEX_CLAIM=claim-sxFpsPTDzzF-9RZAxtUL

Example: Traefik (Heimdall)

Update nodes/heimdall/core/compose.yaml:

services:
  traefik:
    image: traefik:v3.6.5
    env_file:
      - .env.secrets
    environment:
      - DOCKER_HOST=tcp://docker-socket-proxy:2375
      # These are now loaded from .env.secrets:
      # - CLOUDFLARE_DNS_API_TOKEN
      # - CLOUDFLARE_ZONE_API_TOKEN

---

Step 12: Commit Compose Updates

Time: 5-8 minutes

git add nodes/waldorf/plex/compose.yaml
git add nodes/heimdall/core/compose.yaml

git commit -m "refactor(security): migrate secrets to encrypted .env files

- Removed hardcoded PLEX_CLAIM from compose.yaml
- Removed hardcoded Cloudflare tokens
- Now loaded from git-crypt encrypted .env.secrets files"

git push origin main

---

Phase 4: Testing & Validation

Estimated Time: 30-40 minutes

Step 13: Test Automated Deployment

Time: 20-25 minutes (includes waiting for deployment)

Trigger a deployment via Komodo:

1. Make a minor change to a compose file (e.g., add a comment)
2. Commit and push to Gitea
3. Webhook triggers Komodo
4. Komodo pulls repo (git-crypt auto-decrypts)
5. Komodo deploys stack with decrypted secrets

Verify:

# On Waldorf
ssh chester@10.0.0.251
docker exec plex env | grep PLEX_CLAIM
# Should show the actual claim token (not placeholder)

# Check container logs for errors
docker logs plex --tail 50

---

Step 14: Test Secret Rotation

Time: 10-15 minutes

Scenario: Update Plex claim token

# On local machine
cd ~/homelab

# Edit encrypted file (git-crypt auto-decrypts for you)
nano nodes/waldorf/plex/.env.secrets
# Change PLEX_CLAIM value

# Commit
git add nodes/waldorf/plex/.env.secrets
git commit -m "chore(plex): rotate claim token"
git push

# Komodo auto-deploys with new secret

Verify on Gitea:

• View the file in Gitea web UI
• Should show binary/encrypted content (not plaintext)

---

Phase 5: Security Hardening

Estimated Time: 20-30 minutes

Step 15: Secure the Keys

Time: 12-15 minutes (across all nodes)

On each node:

# Move key to more secure location
sudo mkdir -p /etc/git-crypt
sudo mv ~/homelab-secrets.key /etc/git-crypt/homelab.key
sudo chmod 400 /etc/git-crypt/homelab.key
sudo chown root:root /etc/git-crypt/homelab.key

# Update unlock command for future use
cd /etc/komodo/repos/homelab
sudo git-crypt unlock /etc/git-crypt/homelab.key

Backup Strategy:

# Create encrypted backup of key
gpg --symmetric --cipher-algo AES256 ~/homelab-secrets.key
# Save homelab-secrets.key.gpg to password manager

# Or use age
age -p ~/homelab-secrets.key > homelab-secrets.key.age
# Save .age file to secure storage

---

Step 16: Document Key Access

Time: 8-15 minutes

Create documentation/SECURITY_KEY_MANAGEMENT.md:

# Secret Key Management

## Git-crypt Key Location

**Production Nodes:**
- Heimdall: `/etc/git-crypt/homelab.key`
- Waldorf: `/etc/git-crypt/homelab.key`
- Watchtower: `/etc/git-crypt/homelab.key`

**Backup Locations:**
- Primary: Password manager (encrypted)
- Secondary: Encrypted USB drive (physical safe)
- Tertiary: NAS encrypted backup

## Key Recovery Procedure

If a node loses git-crypt unlock state:

1. SSH to node
2. Navigate to `/etc/komodo/repos/homelab`
3. Run: `sudo git-crypt unlock /etc/git-crypt/homelab.key`
4. Verify: `cat nodes/{node}/{stack}/.env.secrets`

## Key Rotation

**Frequency:** Annually or after security incident

**Process:**
1. Generate new git-crypt key: `git-crypt init`
2. Export new key: `git-crypt export-key ~/new-key`
3. Re-encrypt all files: `git-crypt rotate-key ~/new-key`
4. Distribute new key to all nodes
5. Securely destroy old key

---

Troubleshooting

Issue: "File is not encrypted" after push

Cause: .gitattributes not committed before files

Fix:

git rm --cached nodes/waldorf/plex/.env.secrets
git add .gitattributes
git commit -m "Add encryption rules first"
git add nodes/waldorf/plex/.env.secrets
git commit -m "Add encrypted secrets"

---

Issue: Can't read secrets on node

Cause: Repository not unlocked

Fix:

cd /etc/komodo/repos/homelab
git-crypt unlock ~/homelab-secrets.key

# Verify
git-crypt status

---

Issue: Secrets showing as plaintext in Gitea

Cause: Git-crypt not configured server-side (this is EXPECTED)

Note: Gitea displays raw Git objects. View the actual commit:

git show HEAD:nodes/waldorf/plex/.env.secrets
# Should be binary garbage

---

Issue: Merge conflict in encrypted file

Fix:

# Decrypt both versions
git show HEAD:.env.secrets > .env.secrets.ours
git show MERGE_HEAD:.env.secrets > .env.secrets.theirs

# Manually merge
nano .env.secrets

# Re-encrypt
git add .env.secrets
git commit

---

Migration Checklist

Pre-Migration:

• Backup current secrets (screenshot Komodo UI environment variables)
• Test git-crypt on dummy repo first
• Verify all nodes have git-crypt installed

Migration Steps:

• Initialize git-crypt locally
• Export and secure key
• Configure .gitattributes
• Create encrypted .env.secrets files
• Test encryption locally (git-crypt lock/unlock)
• Commit and push encrypted files
• Distribute key to all nodes
• Unlock repositories on each node
• Update compose files to use env_file
• Remove hardcoded secrets from compose files
• Test deployment via Komodo webhook
• Verify containers can read secrets
• Document key locations
• Delete unencrypted secret backups

Post-Migration:

• Update .gitignore to allow .env.secrets
• Remove secrets from Komodo UI (optional - can keep as backup)
• Update SECURITY_AUDIT_REPORT.md
• Create SOP for secret rotation
• Test disaster recovery (unlock after simulated node failure)

---

Rollback Plan

If git-crypt causes issues:

# 1. Remove encrypted files from Git
git rm nodes/**/.env.secrets
git commit -m "Rollback: Remove git-crypt secrets"

# 2. Re-create .env.secrets files locally (gitignored)
# 3. Manually copy to nodes via SCP
# 4. Reset to Komodo UI environment variables

# 5. Remove git-crypt config
git-crypt deinit  # If this command exists
# Or manually remove .git-crypt directory

---

Next Steps After Migration

1. Create SOP-002: "Secret Rotation Procedure"
2. Automate key backup: Add to NAS backup schedule
3. Monitor: Set calendar reminder for annual key rotation
4. Scale: Apply same pattern to other repositories
5. Enhance: Consider adding GPG user keys for team access

---

Related Documentation

• SECURITY_AUDIT_REPORT.md - Initial security findings
• SOP-001 - Secrets management section
• Git-crypt Documentation: https://github.com/AGWA/git-crypt

---

Success Criteria

• ✅ All secrets encrypted in Git repository
• ✅ Komodo auto-deploy works without changes
• ✅ No plaintext secrets visible in Gitea web UI
• ✅ Containers can read secrets from mounted files
• ✅ Key securely backed up in multiple locations
• ✅ Secret rotation tested and documented

Estimated Migration Time: 2-3 hours (including testing)

Maintenance: Near-zero (transparent after initial setup)