feat(documentation): add centralized prompt repository plan with automated distribution
This commit is contained in:
parent
63fd1eb034
commit
eded49e711
406
documentation/plans/plan-promptDistribution.md
Normal file
406
documentation/plans/plan-promptDistribution.md
Normal file
@ -0,0 +1,406 @@
|
||||
# Centralized Prompt Repository with Automated Distribution
|
||||
|
||||
## Goal
|
||||
|
||||
Create a centralized "llm-prompts" repository that automatically distributes prompt files, instructions, and knowledge bases to multiple consumer repositories whenever changes are pushed.
|
||||
|
||||
## Architecture Overview
|
||||
|
||||
```mermaid
|
||||
graph LR
|
||||
A[llm-prompts repo<br/>Central Source] -->|Push| B[Gitea Webhook]
|
||||
B -->|Trigger| C[Sync Script]
|
||||
C -->|Git Pull & Copy| D[homelab repo]
|
||||
C -->|Git Pull & Copy| E[repo-2]
|
||||
C -->|Git Pull & Copy| F[repo-N]
|
||||
|
||||
style A fill:#ff6b6b,color:#fff
|
||||
style C fill:#326ce5,color:#fff
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Option 1: Gitea Webhook + Sync Script (Recommended)
|
||||
|
||||
**Best for:** Your current stack (Gitea, automation-friendly)
|
||||
|
||||
### Step 1: Create Central Repository
|
||||
|
||||
```bash
|
||||
# On Gitea: Create new repo "llm-prompts"
|
||||
# URL: https://git.castaldifamily.com/nathan/llm-prompts
|
||||
|
||||
# Initialize locally
|
||||
mkdir ~/llm-prompts
|
||||
cd ~/llm-prompts
|
||||
|
||||
# Move your existing prompts
|
||||
cp -r ~/homelab/.github/prompts ./
|
||||
cp -r ~/homelab/.github/instructions ./
|
||||
cp -r ~/homelab/.github/knowledge ./
|
||||
|
||||
git init
|
||||
git add .
|
||||
git commit -m "feat: initialize centralized prompt library"
|
||||
git remote add origin https://git.castaldifamily.com/nathan/llm-prompts.git
|
||||
git push -u origin main
|
||||
```
|
||||
|
||||
### Step 2: Create Sync Script
|
||||
|
||||
Create `/mnt/appdata/scripts/sync-prompts.sh` on Heimdall:
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
# Sync Script: Distribute prompts from central repo to consumers
|
||||
set -euo pipefail
|
||||
|
||||
CENTRAL_REPO="/mnt/appdata/git-repos/llm-prompts"
|
||||
TARGET_REPOS=(
|
||||
"/mnt/appdata/git-repos/homelab"
|
||||
"/mnt/appdata/git-repos/another-project"
|
||||
# Add more repos here
|
||||
)
|
||||
|
||||
# Pull latest from central repo
|
||||
cd "$CENTRAL_REPO"
|
||||
git pull origin main
|
||||
|
||||
# Sync to each target repo
|
||||
for REPO in "${TARGET_REPOS[@]}"; do
|
||||
echo "Syncing to $REPO..."
|
||||
|
||||
# Create .github directory if it doesn't exist
|
||||
mkdir -p "$REPO/.github"
|
||||
|
||||
# Copy directories (overwrite)
|
||||
rsync -av --delete "$CENTRAL_REPO/prompts/" "$REPO/.github/prompts/"
|
||||
rsync -av --delete "$CENTRAL_REPO/instructions/" "$REPO/.github/instructions/"
|
||||
rsync -av --delete "$CENTRAL_REPO/knowledge/" "$REPO/.github/knowledge/"
|
||||
|
||||
# Auto-commit in target repo
|
||||
cd "$REPO"
|
||||
if [[ -n $(git status --porcelain) ]]; then
|
||||
git add .github/
|
||||
git commit -m "chore: sync prompts from llm-prompts@$(cd $CENTRAL_REPO && git rev-parse --short HEAD)"
|
||||
git push origin main
|
||||
fi
|
||||
done
|
||||
|
||||
echo "✅ Sync complete"
|
||||
```
|
||||
|
||||
### Step 3: Configure Gitea Webhook
|
||||
|
||||
1. **In Gitea UI** → llm-prompts repo → Settings → Webhooks
|
||||
2. **Add Webhook:**
|
||||
- Target URL: `http://script-runner:8080/sync-prompts` (see Option A or B below)
|
||||
- Content Type: `application/json`
|
||||
- Trigger: `Push events` only
|
||||
- Active: ✅
|
||||
|
||||
### Step 4: Choose Webhook Receiver
|
||||
|
||||
**Option A: Simple HTTP Server (Quick)**
|
||||
|
||||
```bash
|
||||
# On Heimdall, install webhook listener
|
||||
mkdir -p /mnt/appdata/webhook-listener
|
||||
cat > /mnt/appdata/webhook-listener/docker-compose.yaml <<EOF
|
||||
services:
|
||||
webhook:
|
||||
image: adnanh/webhook:latest
|
||||
container_name: webhook-listener
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "9000:9000"
|
||||
volumes:
|
||||
- ./hooks.json:/etc/webhook/hooks.json:ro
|
||||
- /mnt/appdata/scripts:/scripts:ro
|
||||
- /mnt/appdata/git-repos:/repos:rw
|
||||
command: -verbose -hooks=/etc/webhook/hooks.json
|
||||
EOF
|
||||
|
||||
# Create hooks configuration
|
||||
cat > /mnt/appdata/webhook-listener/hooks.json <<EOF
|
||||
[
|
||||
{
|
||||
"id": "sync-prompts",
|
||||
"execute-command": "/scripts/sync-prompts.sh",
|
||||
"command-working-directory": "/repos",
|
||||
"response-message": "Syncing prompts...",
|
||||
"trigger-rule": {
|
||||
"match": {
|
||||
"type": "payload-hash-sha256",
|
||||
"secret": "YOUR_WEBHOOK_SECRET_HERE",
|
||||
"parameter": {
|
||||
"source": "header",
|
||||
"name": "X-Gitea-Signature"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
EOF
|
||||
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
**Option B: Komodo Stack (Integrated)**
|
||||
|
||||
Add to your Komodo deployment if you prefer centralized management.
|
||||
|
||||
---
|
||||
|
||||
## Option 2: Git Submodules (Traditional)
|
||||
|
||||
**Best for:** Simple, standard Git workflow
|
||||
|
||||
```bash
|
||||
# In each target repo
|
||||
cd ~/homelab
|
||||
git submodule add https://git.castaldifamily.com/nathan/llm-prompts.git .github/prompts-lib
|
||||
|
||||
# Create symlinks
|
||||
ln -s .github/prompts-lib/prompts .github/prompts
|
||||
ln -s .github/prompts-lib/instructions .github/instructions
|
||||
|
||||
# Update submodules in all repos
|
||||
git submodule update --remote
|
||||
```
|
||||
|
||||
**Pros:**
|
||||
- ✅ Native Git feature
|
||||
- ✅ Version pinning per repo
|
||||
- ✅ No automation needed
|
||||
|
||||
**Cons:**
|
||||
- ⚠️ Manual update required (`git submodule update`)
|
||||
- ⚠️ Complexity with symlinks
|
||||
|
||||
---
|
||||
|
||||
## Option 3: NFS Shared Directory (Simplest)
|
||||
|
||||
**Best for:** Instant sync across all repos
|
||||
|
||||
```bash
|
||||
# Move prompts to NFS share
|
||||
mkdir /mnt/appdata/shared-prompts
|
||||
mv ~/homelab/.github/prompts/* /mnt/appdata/shared-prompts/
|
||||
|
||||
# In each repo, create symlink
|
||||
cd ~/homelab
|
||||
rm -rf .github/prompts
|
||||
ln -s /mnt/appdata/shared-prompts .github/prompts
|
||||
```
|
||||
|
||||
**Pros:**
|
||||
- ✅ Zero latency (instant sync)
|
||||
- ✅ No automation needed
|
||||
- ✅ Single source of truth
|
||||
|
||||
**Cons:**
|
||||
- ⚠️ Not versioned per-repo
|
||||
- ⚠️ All repos must access same NFS
|
||||
|
||||
---
|
||||
|
||||
## Recommended Approach for Your Setup
|
||||
|
||||
**Use Option 1 (Webhook + Script)** because:
|
||||
|
||||
1. ✅ You already have Gitea webhooks working
|
||||
2. ✅ Fits your GitOps workflow (everything versioned)
|
||||
3. ✅ Gives you audit trail (commit per sync)
|
||||
4. ✅ Can add validation/testing before sync
|
||||
5. ✅ Scales to non-.github files (configs, etc.)
|
||||
|
||||
---
|
||||
|
||||
## Enhanced Version: With Validation
|
||||
|
||||
Improve the sync script to validate prompts before pushing:
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
# sync-prompts.sh (Enhanced with validation)
|
||||
set -euo pipefail
|
||||
|
||||
CENTRAL_REPO="/mnt/appdata/git-repos/llm-prompts"
|
||||
|
||||
# Pull latest
|
||||
cd "$CENTRAL_REPO"
|
||||
git pull origin main
|
||||
|
||||
# Validate structure
|
||||
echo "🔍 Validating prompt structure..."
|
||||
if [[ ! -d "prompts" ]] || [[ ! -d "instructions" ]]; then
|
||||
echo "❌ ERROR: Missing required directories"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Check for required files
|
||||
REQUIRED_FILES=("instructions/core.instructions.md")
|
||||
for FILE in "${REQUIRED_FILES[@]}"; do
|
||||
if [[ ! -f "$FILE" ]]; then
|
||||
echo "❌ ERROR: Missing $FILE"
|
||||
exit 1
|
||||
fi
|
||||
done
|
||||
|
||||
echo "✅ Validation passed"
|
||||
|
||||
# Sync to targets (same as before)
|
||||
# ... rest of script ...
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Migration Strategy
|
||||
|
||||
### Week 1: Setup Central Repository
|
||||
- Create `llm-prompts` repo on Gitea
|
||||
- Copy existing prompts/instructions/knowledge from homelab
|
||||
- Test structure and validate files
|
||||
- Document repository layout
|
||||
|
||||
### Week 2: Test Automation (Single Target)
|
||||
- Deploy webhook listener container
|
||||
- Create sync script in `/mnt/appdata/scripts/`
|
||||
- Configure Gitea webhook
|
||||
- Test with homelab repo only
|
||||
- Monitor logs and verify commits
|
||||
|
||||
### Week 3: Scale to Multiple Repos
|
||||
- Add additional target repositories to sync script
|
||||
- Test with 2-3 repos
|
||||
- Verify no conflicts or issues
|
||||
- Document any edge cases
|
||||
|
||||
### Week 4: Production Rollout
|
||||
- Archive old `.github` directories in target repos
|
||||
- Add monitoring/alerting for sync failures
|
||||
- Create troubleshooting guide
|
||||
- Update documentation with new workflow
|
||||
|
||||
---
|
||||
|
||||
## Security Considerations
|
||||
|
||||
### Webhook Authentication
|
||||
- Generate strong webhook secret: `openssl rand -hex 32`
|
||||
- Store in Gitea webhook configuration
|
||||
- Match in `hooks.json` file
|
||||
- Never commit secrets to Git
|
||||
|
||||
### Script Permissions
|
||||
- Sync script should run as dedicated user (not root)
|
||||
- Git repos should have appropriate ownership
|
||||
- Use SSH keys for Git authentication (not passwords)
|
||||
- Validate all inputs before executing commands
|
||||
|
||||
### Access Control
|
||||
- Central repo: Restricted write access (you only)
|
||||
- Target repos: Can be read-only or read-write depending on workflow
|
||||
- Webhook listener: Internal network only (not exposed to internet)
|
||||
|
||||
---
|
||||
|
||||
## Monitoring & Troubleshooting
|
||||
|
||||
### Log Files
|
||||
```bash
|
||||
# Webhook listener logs
|
||||
docker logs webhook-listener
|
||||
|
||||
# Sync script execution
|
||||
# Add to script: exec > >(tee -a /var/log/prompt-sync.log)
|
||||
|
||||
# Check last sync status
|
||||
cd /mnt/appdata/git-repos/homelab
|
||||
git log --oneline | grep "sync prompts"
|
||||
```
|
||||
|
||||
### Common Issues
|
||||
|
||||
**Issue: Sync script fails with "permission denied"**
|
||||
- Check file ownership: `ls -la /mnt/appdata/scripts/sync-prompts.sh`
|
||||
- Verify execute permission: `chmod +x /mnt/appdata/scripts/sync-prompts.sh`
|
||||
- Check Git authentication: `ssh -T git@git.castaldifamily.com`
|
||||
|
||||
**Issue: Webhook not triggering**
|
||||
- Verify webhook URL is accessible from Gitea
|
||||
- Check Gitea webhook delivery logs
|
||||
- Test webhook manually: `curl -X POST http://webhook-listener:9000/hooks/sync-prompts`
|
||||
|
||||
**Issue: Merge conflicts in target repos**
|
||||
- Never manually edit `.github/prompts/` in target repos
|
||||
- If conflict occurs, revert local changes: `git checkout HEAD -- .github/prompts/`
|
||||
- Re-run sync manually
|
||||
|
||||
---
|
||||
|
||||
## Alternative: GitHub Actions Style Workflow
|
||||
|
||||
If you want more advanced features (testing, validation, notifications):
|
||||
|
||||
```yaml
|
||||
# In llm-prompts repo: .gitea/workflows/sync.yaml
|
||||
name: Sync Prompts to Consumer Repos
|
||||
|
||||
on:
|
||||
push:
|
||||
branches:
|
||||
- main
|
||||
|
||||
jobs:
|
||||
validate:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v3
|
||||
- name: Validate prompt structure
|
||||
run: |
|
||||
# Add validation logic
|
||||
|
||||
sync:
|
||||
needs: validate
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Sync to homelab
|
||||
run: |
|
||||
# Clone target repo
|
||||
# Copy files
|
||||
# Push changes
|
||||
```
|
||||
|
||||
**Note:** Gitea Actions support varies by version. Verify your Gitea version supports Actions before implementing.
|
||||
|
||||
---
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. **Decision Point:** Choose Option 1, 2, or 3 based on your priorities
|
||||
2. **Create Infrastructure:** Set up webhook listener or NFS share
|
||||
3. **Test in Isolation:** Start with single target repo
|
||||
4. **Document Process:** Add to SOPs when stable
|
||||
5. **Scale Gradually:** Add repos incrementally
|
||||
|
||||
---
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- Gitea Webhooks: https://docs.gitea.io/en-us/webhooks/
|
||||
- Webhook Tool: https://github.com/adnanh/webhook
|
||||
- Git Submodules: https://git-scm.com/book/en/v2/Git-Tools-Submodules
|
||||
|
||||
---
|
||||
|
||||
## Future Enhancements
|
||||
|
||||
- **Selective Sync:** Only sync specific directories based on target repo config
|
||||
- **Version Pinning:** Allow target repos to specify prompt version
|
||||
- **Rollback Mechanism:** Automatic rollback if validation fails in target
|
||||
- **Notification System:** Slack/Discord notifications on sync completion
|
||||
- **Dry-Run Mode:** Preview changes before applying
|
||||
- **Conflict Resolution:** Automatic strategies for handling local modifications
|
||||
Loading…
x
Reference in New Issue
Block a user