nathan/homelab
1
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Releases Wiki Activity

feat(documentation): add migration guide for Git-crypt secret management

Browse Source
This commit is contained in:
nathan 2026-04-12 01:00:56 -04:00
parent eded49e711
commit a934117599
1 changed files with 663 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

663
documentation/plans/plan-gitcryptMigration.md Normal file
View File

@ -0,0 +1,663 @@
# Migration Guide: Git-crypt for Secret Management
## Overview
Implement Git-crypt to encrypt sensitive `.env` files in the homelab repository, enabling safe commit of secrets while maintaining seamless integration with Komodo's GitOps workflow.
**Goal:** Zero workflow changes for Komodo, encrypted secrets in Git, transparent decryption on pull.
---
## Prerequisites
- [ ] SSH access to all Komodo nodes (Heimdall, Waldorf, Watchtower)
- [ ] Git-crypt installed on local machine
- [ ] Ability to push to Gitea repository
- [ ] Current `.gitignore` already excludes `.env.secrets` (will be removed)
---
## Phase 1: Local Setup (Your Workstation)
### Step 1: Install Git-crypt
**Windows (via Git Bash):**
```bash
# Download latest release
curl -L https://github.com/AGWA/git-crypt/releases/download/0.7.0/git-crypt-0.7.0-x86_64.exe -o git-crypt.exe
sudo mv git-crypt.exe /usr/local/bin/git-crypt
chmod +x /usr/local/bin/git-crypt
```
**Or via Homebrew (if using WSL/MacOS):**
```bash
brew install git-crypt
```
**Verify:**
```bash
git-crypt --version
# Expected: git-crypt 0.7.0
```
---
### Step 2: Initialize Git-crypt in Repository
```bash
cd ~/homelab
# Initialize git-crypt
git-crypt init
# Export the symmetric key (CRITICAL - SAVE THIS SECURELY)
git-crypt export-key ~/homelab-secrets.key
# IMPORTANT: Back this key up in multiple secure locations:
# - Password manager (1Password, Bitwarden)
# - Encrypted USB drive
# - Secure cloud storage (encrypted)
```
---
### Step 3: Configure Encryption Rules
Create `.gitattributes` in repository root:
```bash
cat > .gitattributes <<'EOF'
# Git-crypt Encryption Rules
# Encrypt all .env.secrets files across the repository
**/.env.secrets filter=git-crypt diff=git-crypt
*.env.secrets filter=git-crypt diff=git-crypt
# Encrypt the key itself if accidentally added
*.key filter=git-crypt diff=git-crypt
# Encrypt specific config files (optional)
# **/secrets.yml filter=git-crypt diff=git-crypt
EOF
git add .gitattributes
git commit -m "chore(security): configure git-crypt encryption rules"
```
---
### Step 4: Update .gitignore
**Remove** `.env.secrets` from `.gitignore` since they'll now be encrypted:
```bash
# Edit .gitignore - remove these lines:
# **/.env.secrets
# **/*.env.secrets
# But KEEP these:
# **/.env.local
# *.key (prevent accidental key commit)
```
Update `.gitignore`:
```bash
# Environment variables and secrets
# NOTE: .env.secrets are now ENCRYPTED via git-crypt, safe to commit
**/.env.local
.env.local
# Git-crypt keys (NEVER commit these)
*.key
homelab-secrets.key
# Temporary unencrypted files
**/.env.secrets.decrypted
```
---
### Step 5: Create Encrypted Secret Files
**For Plex (Waldorf):**
```bash
# Create encrypted file
cat > nodes/waldorf/plex/.env.secrets <<'EOF'
# Plex Configuration Secrets
PLEX_CLAIM=claim-sxFpsPTDzzF-9RZAxtUL
EOF
# Verify it will be encrypted
git-crypt status nodes/waldorf/plex/.env.secrets
# Expected output: encrypted: nodes/waldorf/plex/.env.secrets
```
**For Traefik (Heimdall):**
```bash
cat > nodes/heimdall/core/.env.secrets <<'EOF'
# Cloudflare API Credentials
CF_API_TOKEN=your_cloudflare_api_token_here
CF_ZONE_TOKEN=your_cloudflare_zone_token_here
# Komodo Database
KOMODO_DATABASE_USERNAME=komodo_admin
KOMODO_DATABASE_PASSWORD=your_database_password_here
KOMODO_ONBOARDING_KEY_HEIMDALL=your_onboarding_key_here
# Redis Password
REDIS_PASSWORD=your_redis_password_here
EOF
git-crypt status nodes/heimdall/core/.env.secrets
# Expected: encrypted: nodes/heimdall/core/.env.secrets
```
---
### Step 6: Test Encryption Locally
```bash
# Check encryption status
git-crypt status
# Expected output:
# encrypted: nodes/waldorf/plex/.env.secrets
# encrypted: nodes/heimdall/core/.env.secrets
# View file (should be readable on your machine)
cat nodes/waldorf/plex/.env.secrets
# You should see plaintext
# Lock the repository to simulate what Git sees
git-crypt lock
# Try to read again
cat nodes/waldorf/plex/.env.secrets
# You should see binary garbage (encrypted)
# Unlock to continue working
git-crypt unlock
# Or unlock with specific key
git-crypt unlock ~/homelab-secrets.key
```
---
### Step 7: Commit Encrypted Secrets
```bash
# Stage encrypted files
git add nodes/waldorf/plex/.env.secrets
git add nodes/heimdall/core/.env.secrets
git add .gitattributes
# Verify they're encrypted in staging
git show :nodes/waldorf/plex/.env.secrets
# Should show binary data, NOT plaintext
# Commit
git commit -m "chore(security): add encrypted secrets via git-crypt
- nodes/waldorf/plex/.env.secrets: Plex claim token
- nodes/heimdall/core/.env.secrets: Cloudflare, Komodo, Redis credentials
- Safe to commit (encrypted with git-crypt)"
# Push to Gitea
git push origin main
```
---
## Phase 2: Node Setup (Komodo Deployment Targets)
### Step 8: Distribute Key to Komodo Nodes
**SECURITY NOTE:** Use secure methods to transfer the key (not email, not Slack).
**Option A: SCP (Secure Copy)**
```bash
# Copy key to Heimdall
scp ~/homelab-secrets.key chester@10.0.0.151:~/
# Copy key to Waldorf
scp ~/homelab-secrets.key chester@10.0.0.251:~/
# Copy key to Watchtower
scp ~/homelab-secrets.key chester@10.0.0.200:~/
```
**Option B: Manual Transfer via USB**
- Copy key to USB drive
- SSH to each node
- Transfer from USB to home directory
---
### Step 9: Install Git-crypt on Nodes
**On each node (Heimdall, Waldorf, Watchtower):**
```bash
# SSH to node
ssh chester@10.0.0.151 # Repeat for .251 and .200
# Install git-crypt
sudo apt update
sudo apt install git-crypt -y
# Verify installation
git-crypt --version
```
---
### Step 10: Unlock Repositories on Nodes
**Critical:** This must be done in Komodo's repo directories, not just any clone.
**On Heimdall:**
```bash
ssh chester@10.0.0.151
# Navigate to Komodo's repo directory
cd /etc/komodo/repos/homelab
# Unlock with the key
git-crypt unlock ~/homelab-secrets.key
# Verify decryption worked
cat nodes/heimdall/core/.env.secrets
# Should show plaintext secrets
# Pull to test
git pull origin main
# Secrets should auto-decrypt on pull
# Secure the key
chmod 600 ~/homelab-secrets.key
```
**On Waldorf:**
```bash
ssh chester@10.0.0.251
# Find Komodo periphery repo path
cd /etc/komodo/repos/homelab # Or wherever Komodo clones to
git-crypt unlock ~/homelab-secrets.key
# Verify
cat nodes/waldorf/plex/.env.secrets
chmod 600 ~/homelab-secrets.key
```
**On Watchtower:**
```bash
ssh chester@10.0.0.200
cd /etc/komodo/repos/homelab
git-crypt unlock ~/homelab-secrets.key
cat nodes/watchtower/*/..env.secrets # If any exist
chmod 600 ~/homelab-secrets.key
```
---
## Phase 3: Update Compose Files
### Step 11: Reference Encrypted Secret Files
**Example: Plex (Waldorf)**
Update `nodes/waldorf/plex/compose.yaml`:
```yaml
services:
plex:
image: lscr.io/linuxserver/plex:latest
container_name: plex
network_mode: host
restart: unless-stopped
env_file:
- .env.secrets # Now encrypted in Git!
environment:
- PUID=1000
- PGID=1000
- TZ=America/New_York
- VERSION=docker
# PLEX_CLAIM loaded from .env.secrets
volumes:
- /mnt/appdata/plex:/config
- /mnt/media/tvshows:/tv
- /mnt/media/movies:/movies
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: 1
capabilities: [gpu]
```
**Remove hardcoded secrets:**
```diff
- environment:
- - PLEX_CLAIM=claim-sxFpsPTDzzF-9RZAxtUL
```
**Example: Traefik (Heimdall)**
Update `nodes/heimdall/core/compose.yaml`:
```yaml
services:
traefik:
image: traefik:v3.6.5
env_file:
- .env.secrets
environment:
- DOCKER_HOST=tcp://docker-socket-proxy:2375
# These are now loaded from .env.secrets:
# - CLOUDFLARE_DNS_API_TOKEN
# - CLOUDFLARE_ZONE_API_TOKEN
```
---
### Step 12: Commit Compose Updates
```bash
git add nodes/waldorf/plex/compose.yaml
git add nodes/heimdall/core/compose.yaml
git commit -m "refactor(security): migrate secrets to encrypted .env files
- Removed hardcoded PLEX_CLAIM from compose.yaml
- Removed hardcoded Cloudflare tokens
- Now loaded from git-crypt encrypted .env.secrets files"
git push origin main
```
---
## Phase 4: Testing & Validation
### Step 13: Test Automated Deployment
**Trigger a deployment via Komodo:**
1. Make a minor change to a compose file (e.g., add a comment)
2. Commit and push to Gitea
3. Webhook triggers Komodo
4. Komodo pulls repo (git-crypt auto-decrypts)
5. Komodo deploys stack with decrypted secrets
**Verify:**
```bash
# On Waldorf
ssh chester@10.0.0.251
docker exec plex env | grep PLEX_CLAIM
# Should show the actual claim token (not placeholder)
# Check container logs for errors
docker logs plex --tail 50
```
---
### Step 14: Test Secret Rotation
**Scenario: Update Plex claim token**
```bash
# On local machine
cd ~/homelab
# Edit encrypted file (git-crypt auto-decrypts for you)
nano nodes/waldorf/plex/.env.secrets
# Change PLEX_CLAIM value
# Commit
git add nodes/waldorf/plex/.env.secrets
git commit -m "chore(plex): rotate claim token"
git push
# Komodo auto-deploys with new secret
```
**Verify on Gitea:**
- View the file in Gitea web UI
- Should show binary/encrypted content (not plaintext)
---
## Phase 5: Security Hardening
### Step 15: Secure the Keys
**On each node:**
```bash
# Move key to more secure location
sudo mkdir -p /etc/git-crypt
sudo mv ~/homelab-secrets.key /etc/git-crypt/homelab.key
sudo chmod 400 /etc/git-crypt/homelab.key
sudo chown root:root /etc/git-crypt/homelab.key
# Update unlock command for future use
cd /etc/komodo/repos/homelab
sudo git-crypt unlock /etc/git-crypt/homelab.key
```
**Backup Strategy:**
```bash
# Create encrypted backup of key
gpg --symmetric --cipher-algo AES256 ~/homelab-secrets.key
# Save homelab-secrets.key.gpg to password manager
# Or use age
age -p ~/homelab-secrets.key > homelab-secrets.key.age
# Save .age file to secure storage
```
---
### Step 16: Document Key Access
Create `documentation/SECURITY_KEY_MANAGEMENT.md`:
```markdown
# Secret Key Management
## Git-crypt Key Location
**Production Nodes:**
- Heimdall: `/etc/git-crypt/homelab.key`
- Waldorf: `/etc/git-crypt/homelab.key`
- Watchtower: `/etc/git-crypt/homelab.key`
**Backup Locations:**
- Primary: Password manager (encrypted)
- Secondary: Encrypted USB drive (physical safe)
- Tertiary: NAS encrypted backup
## Key Recovery Procedure
If a node loses git-crypt unlock state:
1. SSH to node
2. Navigate to `/etc/komodo/repos/homelab`
3. Run: `sudo git-crypt unlock /etc/git-crypt/homelab.key`
4. Verify: `cat nodes/{node}/{stack}/.env.secrets`
## Key Rotation
**Frequency:** Annually or after security incident
**Process:**
1. Generate new git-crypt key: `git-crypt init`
2. Export new key: `git-crypt export-key ~/new-key`
3. Re-encrypt all files: `git-crypt rotate-key ~/new-key`
4. Distribute new key to all nodes
5. Securely destroy old key
```
---
## Troubleshooting
### Issue: "File is not encrypted" after push
**Cause:** `.gitattributes` not committed before files
**Fix:**
```bash
git rm --cached nodes/waldorf/plex/.env.secrets
git add .gitattributes
git commit -m "Add encryption rules first"
git add nodes/waldorf/plex/.env.secrets
git commit -m "Add encrypted secrets"
```
---
### Issue: Can't read secrets on node
**Cause:** Repository not unlocked
**Fix:**
```bash
cd /etc/komodo/repos/homelab
git-crypt unlock ~/homelab-secrets.key
# Verify
git-crypt status
```
---
### Issue: Secrets showing as plaintext in Gitea
**Cause:** Git-crypt not configured server-side (this is EXPECTED)
**Note:** Gitea displays raw Git objects. View the actual commit:
```bash
git show HEAD:nodes/waldorf/plex/.env.secrets
# Should be binary garbage
```
---
### Issue: Merge conflict in encrypted file
**Fix:**
```bash
# Decrypt both versions
git show HEAD:.env.secrets > .env.secrets.ours
git show MERGE_HEAD:.env.secrets > .env.secrets.theirs
# Manually merge
nano .env.secrets
# Re-encrypt
git add .env.secrets
git commit
```
---
## Migration Checklist
**Pre-Migration:**
- [ ] Backup current secrets (screenshot Komodo UI environment variables)
- [ ] Test git-crypt on dummy repo first
- [ ] Verify all nodes have `git-crypt` installed
**Migration Steps:**
- [ ] Initialize git-crypt locally
- [ ] Export and secure key
- [ ] Configure `.gitattributes`
- [ ] Create encrypted `.env.secrets` files
- [ ] Test encryption locally (`git-crypt lock/unlock`)
- [ ] Commit and push encrypted files
- [ ] Distribute key to all nodes
- [ ] Unlock repositories on each node
- [ ] Update compose files to use `env_file`
- [ ] Remove hardcoded secrets from compose files
- [ ] Test deployment via Komodo webhook
- [ ] Verify containers can read secrets
- [ ] Document key locations
- [ ] Delete unencrypted secret backups
**Post-Migration:**
- [ ] Update `.gitignore` to allow `.env.secrets`
- [ ] Remove secrets from Komodo UI (optional - can keep as backup)
- [ ] Update [SECURITY_AUDIT_REPORT.md](../documentation/SECURITY_AUDIT_REPORT.md)
- [ ] Create SOP for secret rotation
- [ ] Test disaster recovery (unlock after simulated node failure)
---
## Rollback Plan
If git-crypt causes issues:
```bash
# 1. Remove encrypted files from Git
git rm nodes/**/.env.secrets
git commit -m "Rollback: Remove git-crypt secrets"
# 2. Re-create .env.secrets files locally (gitignored)
# 3. Manually copy to nodes via SCP
# 4. Reset to Komodo UI environment variables
# 5. Remove git-crypt config
git-crypt deinit # If this command exists
# Or manually remove .git-crypt directory
```
---
## Next Steps After Migration
1. **Create SOP-002:** "Secret Rotation Procedure"
2. **Automate key backup:** Add to NAS backup schedule
3. **Monitor:** Set calendar reminder for annual key rotation
4. **Scale:** Apply same pattern to other repositories
5. **Enhance:** Consider adding GPG user keys for team access
---
## Related Documentation
- [SECURITY_AUDIT_REPORT.md](../documentation/SECURITY_AUDIT_REPORT.md) - Initial security findings
- [SOP-001](../documentation/SOPs/SOP-001-Migrate-Stack-from-UI-to-Git.md) - Secrets management section
- Git-crypt Documentation: https://github.com/AGWA/git-crypt
---
## Success Criteria
- ✅ All secrets encrypted in Git repository
- ✅ Komodo auto-deploy works without changes
- ✅ No plaintext secrets visible in Gitea web UI
- ✅ Containers can read secrets from mounted files
- ✅ Key securely backed up in multiple locations
- ✅ Secret rotation tested and documented
**Estimated Migration Time:** 2-3 hours (including testing)
**Maintenance:** Near-zero (transparent after initial setup)