# Migration Guide: Git-crypt for Secret Management

## Overview

Implement Git-crypt to encrypt sensitive `.env` files in the homelab repository, enabling safe commit of secrets while maintaining seamless integration with Komodo's GitOps workflow.

**Goal:** Zero workflow changes for Komodo, encrypted secrets in Git, transparent decryption on pull.

**Estimated Time to Complete:** 2-3 hours (first-time setup) | 1-1.5 hours (experienced operator)

---

## Time Breakdown by Phase

| Phase | Description | Time Estimate |
|-------|-------------|---------------|
| **Phase 1** | Local Setup (Workstation) | 30-40 minutes |
| **Phase 2** | Node Setup (Komodo Targets) | 25-35 minutes |
| **Phase 3** | Update Compose Files | 15-20 minutes |
| **Phase 4** | Testing & Validation | 30-40 minutes |
| **Phase 5** | Security Hardening | 20-30 minutes |
| **Total** | End-to-End Migration | **2-3 hours** |

---

## Prerequisites

- [ ] SSH access to all Komodo nodes (Heimdall, Waldorf, Watchtower)
- [ ] Git-crypt installed on local machine
- [ ] Ability to push to Gitea repository
- [ ] Current `.gitignore` already excludes `.env.secrets` (will be removed)

---

## Phase 1: Local Setup (Your Workstation)
**Estimated Time:** 30-40 minutes

### Step 1: Install Git-crypt
**Time:** 3-5 minutes

**Windows (via Git Bash):**
```bash
# Download latest release
curl -L https://github.com/AGWA/git-crypt/releases/download/0.7.0/git-crypt-0.7.0-x86_64.exe -o git-crypt.exe
sudo mv git-crypt.exe /usr/local/bin/git-crypt
chmod +x /usr/local/bin/git-crypt
```

**Or via Homebrew (if using WSL/MacOS):**
```bash
brew install git-crypt
```

**Verify:**
```bash
git-crypt --version
# Expected: git-crypt 0.7.0
```

---

### Step 2: Initialize Git-crypt in Repository
**Time:** 3-5 minutes

```bash
cd ~/homelab

# Initialize git-crypt
git-crypt init

# Export the symmetric key (CRITICAL - SAVE THIS SECURELY)
git-crypt export-key ~/homelab-secrets.key

# IMPORTANT: Back this key up in multiple secure locations:
# - Password manager (1Password, Bitwarden)
# - Encrypted USB drive
# - Secure cloud storage (encrypted)
```

---

### Step 3: Configure Encryption Rules
**Time:** 5-7 minutes

Create `.gitattributes` in repository root:

```bash
cat > .gitattributes <<'EOF'
# Git-crypt Encryption Rules
# Encrypt all .env.secrets files across the repository
**/.env.secrets filter=git-crypt diff=git-crypt
*.env.secrets filter=git-crypt diff=git-crypt

# Encrypt the key itself if accidentally added
*.key filter=git-crypt diff=git-crypt

# Encrypt specific config files (optional)
# **/secrets.yml filter=git-crypt diff=git-crypt
EOF

git add .gitattributes
git commit -m "chore(security): configure git-crypt encryption rules"
```

---

### Step 4: Update .gitignore
**Time:** 3-5 minutes

**Remove** `.env.secrets` from `.gitignore` since they'll now be encrypted:

```bash
# Edit .gitignore - remove these lines:
# **/.env.secrets
# **/*.env.secrets

# But KEEP these:
# **/.env.local
# *.key  (prevent accidental key commit)
```

Update `.gitignore`:
```bash
# Environment variables and secrets
# NOTE: .env.secrets are now ENCRYPTED via git-crypt, safe to commit
**/.env.local
.env.local

# Git-crypt keys (NEVER commit these)
*.key
homelab-secrets.key

# Temporary unencrypted files
**/.env.secrets.decrypted
```

---

### Step 5: Create Encrypted Secret Files
**Time:** 8-10 minutes

**For Plex (Waldorf):**

```bash
# Create encrypted file
cat > nodes/waldorf/plex/.env.secrets <<'EOF'
# Plex Configuration Secrets
PLEX_CLAIM=claim-sxFpsPTDzzF-9RZAxtUL
EOF

# Verify it will be encrypted
git-crypt status nodes/waldorf/plex/.env.secrets
# Expected output: encrypted: nodes/waldorf/plex/.env.secrets
```

**For Traefik (Heimdall):**

```bash
cat > nodes/heimdall/core/.env.secrets <<'EOF'
# Cloudflare API Credentials
CF_API_TOKEN=your_cloudflare_api_token_here
CF_ZONE_TOKEN=your_cloudflare_zone_token_here

# Komodo Database
KOMODO_DATABASE_USERNAME=komodo_admin
KOMODO_DATABASE_PASSWORD=your_database_password_here
KOMODO_ONBOARDING_KEY_HEIMDALL=your_onboarding_key_here

# Redis Password
REDIS_PASSWORD=your_redis_password_here
EOF

git-crypt status nodes/heimdall/core/.env.secrets
# Expected: encrypted: nodes/heimdall/core/.env.secrets
```

---

### Step 6: Test Encryption Locally
**Time:** 5-7 minutes

```bash
# Check encryption status
git-crypt status

# Expected output:
# encrypted: nodes/waldorf/plex/.env.secrets
# encrypted: nodes/heimdall/core/.env.secrets

# View file (should be readable on your machine)
cat nodes/waldorf/plex/.env.secrets
# You should see plaintext

# Lock the repository to simulate what Git sees
git-crypt lock

# Try to read again
cat nodes/waldorf/plex/.env.secrets
# You should see binary garbage (encrypted)

# Unlock to continue working
git-crypt unlock

# Or unlock with specific key
git-crypt unlock ~/homelab-secrets.key
```

---

### Step 7: Commit Encrypted Secrets
**Time:** 3-5 minutes

```bash
# Stage encrypted files
git add nodes/waldorf/plex/.env.secrets
git add nodes/heimdall/core/.env.secrets
git add .gitattributes

# Verify they're encrypted in staging
git show :nodes/waldorf/plex/.env.secrets
# Should show binary data, NOT plaintext

# Commit
git commit -m "chore(security): add encrypted secrets via git-crypt

- nodes/waldorf/plex/.env.secrets: Plex claim token
- nodes/heimdall/core/.env.secrets: Cloudflare, Komodo, Redis credentials
- Safe to commit (encrypted with git-crypt)"

# Push to Gitea
git push origin main
```

---

## Phase 2: Node Setup (Komodo Deployment Targets)
**Estimated Time:** 25-35 minutes

### Step 8: Distribute Key to Komodo Nodes
**Time:** 5-8 minutes

**SECURITY NOTE:** Use secure methods to transfer the key (not email, not Slack).

**Option A: SCP (Secure Copy)**

```bash
# Copy key to Heimdall
scp ~/homelab-secrets.key chester@10.0.0.151:~/

# Copy key to Waldorf
scp ~/homelab-secrets.key chester@10.0.0.251:~/

# Copy key to Watchtower
scp ~/homelab-secrets.key chester@10.0.0.200:~/
```

**Option B: Manual Transfer via USB**
- Copy key to USB drive
- SSH to each node
- Transfer from USB to home directory

---

### Step 9: Install Git-crypt on Nodes
**Time:** 10-15 minutes (across all 3 nodes)

**On each node (Heimdall, Waldorf, Watchtower):**

```bash
# SSH to node
ssh chester@10.0.0.151  # Repeat for .251 and .200

# Install git-crypt
sudo apt update
sudo apt install git-crypt -y

# Verify installation
git-crypt --version
```

---

### Step 10: Unlock Repositories on Nodes
**Time:** 10-12 minutes (across all 3 nodes)

**Critical:** This must be done in Komodo's repo directories, not just any clone.

**On Heimdall:**

```bash
ssh chester@10.0.0.151

# Navigate to Komodo's repo directory
cd /etc/komodo/repos/homelab

# Unlock with the key
git-crypt unlock ~/homelab-secrets.key

# Verify decryption worked
cat nodes/heimdall/core/.env.secrets
# Should show plaintext secrets

# Pull to test
git pull origin main
# Secrets should auto-decrypt on pull

# Secure the key
chmod 600 ~/homelab-secrets.key
```

**On Waldorf:**

```bash
ssh chester@10.0.0.251

# Find Komodo periphery repo path
cd /etc/komodo/repos/homelab  # Or wherever Komodo clones to

git-crypt unlock ~/homelab-secrets.key

# Verify
cat nodes/waldorf/plex/.env.secrets

chmod 600 ~/homelab-secrets.key
```

**On Watchtower:**

```bash
ssh chester@10.0.0.200

cd /etc/komodo/repos/homelab

git-crypt unlock ~/homelab-secrets.key
cat nodes/watchtower/*/..env.secrets  # If any exist

chmod 600 ~/homelab-secrets.key
```

---

## Phase 3: Update Compose Files
**Estimated Time:** 15-20 minutes

### Step 11: Reference Encrypted Secret Files
**Time:** 10-12 minutes

**Example: Plex (Waldorf)**

Update `nodes/waldorf/plex/compose.yaml`:

```yaml
services:
  plex:
    image: lscr.io/linuxserver/plex:latest
    container_name: plex
    network_mode: host
    restart: unless-stopped
    env_file:
      - .env.secrets  # Now encrypted in Git!
    environment:
      - PUID=1000
      - PGID=1000
      - TZ=America/New_York
      - VERSION=docker
      # PLEX_CLAIM loaded from .env.secrets
    volumes:
      - /mnt/appdata/plex:/config
      - /mnt/media/tvshows:/tv
      - /mnt/media/movies:/movies
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]
```

**Remove hardcoded secrets:**

```diff
- environment:
-   - PLEX_CLAIM=claim-sxFpsPTDzzF-9RZAxtUL
```

**Example: Traefik (Heimdall)**

Update `nodes/heimdall/core/compose.yaml`:

```yaml
services:
  traefik:
    image: traefik:v3.6.5
    env_file:
      - .env.secrets
    environment:
      - DOCKER_HOST=tcp://docker-socket-proxy:2375
      # These are now loaded from .env.secrets:
      # - CLOUDFLARE_DNS_API_TOKEN
      # - CLOUDFLARE_ZONE_API_TOKEN
```

---

### Step 12: Commit Compose Updates
**Time:** 5-8 minutes

```bash
git add nodes/waldorf/plex/compose.yaml
git add nodes/heimdall/core/compose.yaml

git commit -m "refactor(security): migrate secrets to encrypted .env files

- Removed hardcoded PLEX_CLAIM from compose.yaml
- Removed hardcoded Cloudflare tokens
- Now loaded from git-crypt encrypted .env.secrets files"

git push origin main
```

---

## Phase 4: Testing & Validation
**Estimated Time:** 30-40 minutes

### Step 13: Test Automated Deployment
**Time:** 20-25 minutes (includes waiting for deployment)

**Trigger a deployment via Komodo:**

1. Make a minor change to a compose file (e.g., add a comment)
2. Commit and push to Gitea
3. Webhook triggers Komodo
4. Komodo pulls repo (git-crypt auto-decrypts)
5. Komodo deploys stack with decrypted secrets

**Verify:**

```bash
# On Waldorf
ssh chester@10.0.0.251
docker exec plex env | grep PLEX_CLAIM
# Should show the actual claim token (not placeholder)

# Check container logs for errors
docker logs plex --tail 50
```

---

### Step 14: Test Secret Rotation
**Time:** 10-15 minutes

**Scenario: Update Plex claim token**

```bash
# On local machine
cd ~/homelab

# Edit encrypted file (git-crypt auto-decrypts for you)
nano nodes/waldorf/plex/.env.secrets
# Change PLEX_CLAIM value

# Commit
git add nodes/waldorf/plex/.env.secrets
git commit -m "chore(plex): rotate claim token"
git push

# Komodo auto-deploys with new secret
```

**Verify on Gitea:**
- View the file in Gitea web UI
- Should show binary/encrypted content (not plaintext)

---

## Phase 5: Security Hardening
**Estimated Time:** 20-30 minutes

### Step 15: Secure the Keys
**Time:** 12-15 minutes (across all nodes)

**On each node:**

```bash
# Move key to more secure location
sudo mkdir -p /etc/git-crypt
sudo mv ~/homelab-secrets.key /etc/git-crypt/homelab.key
sudo chmod 400 /etc/git-crypt/homelab.key
sudo chown root:root /etc/git-crypt/homelab.key

# Update unlock command for future use
cd /etc/komodo/repos/homelab
sudo git-crypt unlock /etc/git-crypt/homelab.key
```

**Backup Strategy:**

```bash
# Create encrypted backup of key
gpg --symmetric --cipher-algo AES256 ~/homelab-secrets.key
# Save homelab-secrets.key.gpg to password manager

# Or use age
age -p ~/homelab-secrets.key > homelab-secrets.key.age
# Save .age file to secure storage
```

---

### Step 16: Document Key Access
**Time:** 8-15 minutes

Create `documentation/SECURITY_KEY_MANAGEMENT.md`:

```markdown
# Secret Key Management

## Git-crypt Key Location

**Production Nodes:**
- Heimdall: `/etc/git-crypt/homelab.key`
- Waldorf: `/etc/git-crypt/homelab.key`
- Watchtower: `/etc/git-crypt/homelab.key`

**Backup Locations:**
- Primary: Password manager (encrypted)
- Secondary: Encrypted USB drive (physical safe)
- Tertiary: NAS encrypted backup

## Key Recovery Procedure

If a node loses git-crypt unlock state:

1. SSH to node
2. Navigate to `/etc/komodo/repos/homelab`
3. Run: `sudo git-crypt unlock /etc/git-crypt/homelab.key`
4. Verify: `cat nodes/{node}/{stack}/.env.secrets`

## Key Rotation

**Frequency:** Annually or after security incident

**Process:**
1. Generate new git-crypt key: `git-crypt init`
2. Export new key: `git-crypt export-key ~/new-key`
3. Re-encrypt all files: `git-crypt rotate-key ~/new-key`
4. Distribute new key to all nodes
5. Securely destroy old key
```

---

## Troubleshooting

### Issue: "File is not encrypted" after push

**Cause:** `.gitattributes` not committed before files

**Fix:**
```bash
git rm --cached nodes/waldorf/plex/.env.secrets
git add .gitattributes
git commit -m "Add encryption rules first"
git add nodes/waldorf/plex/.env.secrets
git commit -m "Add encrypted secrets"
```

---

### Issue: Can't read secrets on node

**Cause:** Repository not unlocked

**Fix:**
```bash
cd /etc/komodo/repos/homelab
git-crypt unlock ~/homelab-secrets.key

# Verify
git-crypt status
```

---

### Issue: Secrets showing as plaintext in Gitea

**Cause:** Git-crypt not configured server-side (this is EXPECTED)

**Note:** Gitea displays raw Git objects. View the actual commit:
```bash
git show HEAD:nodes/waldorf/plex/.env.secrets
# Should be binary garbage
```

---

### Issue: Merge conflict in encrypted file

**Fix:**
```bash
# Decrypt both versions
git show HEAD:.env.secrets > .env.secrets.ours
git show MERGE_HEAD:.env.secrets > .env.secrets.theirs

# Manually merge
nano .env.secrets

# Re-encrypt
git add .env.secrets
git commit
```

---

## Migration Checklist

**Pre-Migration:**
- [ ] Backup current secrets (screenshot Komodo UI environment variables)
- [ ] Test git-crypt on dummy repo first
- [ ] Verify all nodes have `git-crypt` installed

**Migration Steps:**
- [ ] Initialize git-crypt locally
- [ ] Export and secure key
- [ ] Configure `.gitattributes`
- [ ] Create encrypted `.env.secrets` files
- [ ] Test encryption locally (`git-crypt lock/unlock`)
- [ ] Commit and push encrypted files
- [ ] Distribute key to all nodes
- [ ] Unlock repositories on each node
- [ ] Update compose files to use `env_file`
- [ ] Remove hardcoded secrets from compose files
- [ ] Test deployment via Komodo webhook
- [ ] Verify containers can read secrets
- [ ] Document key locations
- [ ] Delete unencrypted secret backups

**Post-Migration:**
- [ ] Update `.gitignore` to allow `.env.secrets`
- [ ] Remove secrets from Komodo UI (optional - can keep as backup)
- [ ] Update [SECURITY_AUDIT_REPORT.md](../documentation/SECURITY_AUDIT_REPORT.md)
- [ ] Create SOP for secret rotation
- [ ] Test disaster recovery (unlock after simulated node failure)

---

## Rollback Plan

If git-crypt causes issues:

```bash
# 1. Remove encrypted files from Git
git rm nodes/**/.env.secrets
git commit -m "Rollback: Remove git-crypt secrets"

# 2. Re-create .env.secrets files locally (gitignored)
# 3. Manually copy to nodes via SCP
# 4. Reset to Komodo UI environment variables

# 5. Remove git-crypt config
git-crypt deinit  # If this command exists
# Or manually remove .git-crypt directory
```

---

## Next Steps After Migration

1. **Create SOP-002:** "Secret Rotation Procedure"
2. **Automate key backup:** Add to NAS backup schedule
3. **Monitor:** Set calendar reminder for annual key rotation
4. **Scale:** Apply same pattern to other repositories
5. **Enhance:** Consider adding GPG user keys for team access

---

## Related Documentation

- [SECURITY_AUDIT_REPORT.md](../documentation/SECURITY_AUDIT_REPORT.md) - Initial security findings
- [SOP-001](../documentation/SOPs/SOP-001-Migrate-Stack-from-UI-to-Git.md) - Secrets management section
- Git-crypt Documentation: https://github.com/AGWA/git-crypt

---

## Success Criteria

- ✅ All secrets encrypted in Git repository
- ✅ Komodo auto-deploy works without changes
- ✅ No plaintext secrets visible in Gitea web UI
- ✅ Containers can read secrets from mounted files
- ✅ Key securely backed up in multiple locations
- ✅ Secret rotation tested and documented

**Estimated Migration Time:** 2-3 hours (including testing)

**Maintenance:** Near-zero (transparent after initial setup)