nathan/homelab
1
0
Fork 0
Code Issues Pull Requests Actions Packages Releases Wiki Activity
homelab/documentation/RCA-001-Authentik-Outpost-Configuration-Failure.md
Nathan 16f7be6439
All checks were successful
Auto-Deploy Changed Stacks / deploy (push) Successful in 12s
Details
Add RCA-001: Authentik proxy outpost configuration failure
2026-06-14 09:26:40 -04:00

504 lines
23 KiB
Markdown
Raw Blame History

# Root Cause Analysis: Authentik Proxy Outpost Configuration Failure
**Incident ID:** RCA-001
**Date:** 2026-06-14
**Severity:** High (Service Outage)
**Status:** Resolved
**Author:** AI Assistant (Frank v6)
---
## Executive Summary
Four Authentik proxy outpost containers (prowlarr_outpost, radarr_outpost, sonarr_outpost, sabnzbd_outpost) failed to connect to the Authentik server, resulting in complete inaccessibility of protected services. The outposts continuously returned `404 Not Found` errors when attempting to fetch configuration from the Authentik API.
**Root Cause:** Authentik outpost configuration stored in the database overrides container environment variables. The `authentik_host` field in the Authentik admin panel contained values that prevented the outposts from successfully connecting to the Authentik server.
**Resolution:** Updated the `authentik_host` field in Authentik admin panel to `http://authentik_server:9000` for all four outposts, matching the internal Docker network configuration.
**Impact Duration:** ~2 hours
**Services Affected:** Prowlarr, Radarr, Sonarr, Sabnzbd
---
## Timeline
### Initial State (Before Incident)
- Services running but not externally accessible
- Hairpin NAT issue preventing internal clients from using external DNS
- Services reachable internally via `10.0.0.151`
- Traefik reverse proxy functioning correctly
### 13:00 UTC - Problem Discovery
- User reported services accessible internally but returning 404 externally
- Initial hypothesis: Traefik routing or DNS issue
- Verified Traefik listening on `0.0.0.0:80` and `0.0.0.0:443`
- Confirmed DNS resolving correctly to external IP `73.106.212.82`
### 13:05 UTC - Authentik Outpost Investigation Begins
- Examined outpost container logs
- Found continuous `404 Not Found` errors in all four outpost containers:
```
{"error":"404 Not Found","event":"Failed to fetch outpost configuration, retrying in 3 seconds","level":"error","logger":"authentik.outpost.ak-api-controller"}
```
- Observed exponential backoff pattern (3s → 6s → 12s → 24s → 48s)
### 13:15 UTC - Configuration Analysis
- Checked container environment variables via `docker exec`
- Initial configuration:
- `AUTHENTIK_HOST=https://authentik_server:9443`
- `AUTHENTIK_HOST_BROWSER=https://sso.castaldifamily.com`
- `AUTHENTIK_INSECURE=false`
### 13:20 UTC - First Remediation Attempt
- Updated compose files to use HTTPS with insecure flag:
- Changed `AUTHENTIK_HOST` to `https://authentik_server:9443`
- Added `AUTHENTIK_HOST_BROWSER=https://sso.castaldifamily.com`
- Set `AUTHENTIK_INSECURE=true`
- Deployed via Git push to Komodo
- **Result:** Failed - 404 errors persisted
### 13:30 UTC - Authentik Admin Panel Configuration
- Changed Integration type from "Local Docker connection" to "No integration active"
- Set `authentik_host_insecure=true` in admin panel
- Verified via API that changes were reflected
- **Result:** Failed - 404 errors persisted
### 13:40 UTC - Discovery of Configuration Precedence Issue
- Queried Authentik API: `/api/v3/outposts/instances/`
- Discovered `authentik_host` in API response: `"https://sso.castaldifamily.com/"`
- Identified that database configuration was overriding environment variables
- User attempted to clear `authentik_host` field (set to null/blank)
- **Result:** Failed - Authentik validation error: "wrong value type for field 'authentik_host' - should be 'str' instead of value 'None' of type 'NoneType'"
### 13:50 UTC - Protocol and Port Testing
- Switched from HTTPS port 9443 to HTTP port 9000
- Updated compose files: `AUTHENTIK_HOST=http://authentik_server:9000`
- Updated Authentik admin: `authentik_host=http://authentik_server:9000`
- **Result:** Failed - 404 errors persisted
### 13:55 UTC - Network Debugging
- Tested API accessibility from inside outpost container
- Using hostname: `wget http://authentik_server:9000/api/v3/outposts/instances/` → 404
- Using IP address: `wget http://172.18.0.31:9000/api/v3/outposts/instances/` → 200 OK (Success)
- **Discovery:** Host header validation issue suspected
### 14:00 UTC - IP Address Configuration Attempt
- Updated compose files to use IP: `AUTHENTIK_HOST=http://172.18.0.31:9000`
- Prepared to update Authentik admin panel
- User reported success using hostname `http://authentik_server:9000` in admin panel
- **Result:** Success - Outposts connected successfully
### 14:05 UTC - Resolution Confirmed
- Logs showed `"Loaded application"` messages
- Successful proxy requests with HTTP 200 status codes
- Services accessible externally through Authentik authentication
- All four outposts (prowlarr, radarr, sonarr, sabnzbd) operational
---
## Root Cause Analysis
### Primary Root Cause
**Configuration Precedence:** Authentik proxy outposts prioritize configuration stored in the Authentik database over container environment variables. When an outpost starts, it queries the Authentik API at `/api/v3/outposts/instances/` to fetch its configuration. The response includes a `config` object with an `authentik_host` field. If this field has a value in the database, it completely overrides the `AUTHENTIK_HOST` environment variable.
### Contributing Factors
1. **Incomplete Documentation Understanding**
- The interaction between environment variables and database configuration was not well understood
- Authentik documentation does not clearly state that database values override environment variables
- The admin panel allows setting `authentik_host` but does not explain when to use it vs. environment variables
2. **External URL in Database Configuration**
- Initial Authentik setup stored `https://sso.castaldifamily.com/` as the `authentik_host`
- This external URL was inaccessible from within the Docker network due to hairpin NAT limitations
- The outposts attempting to connect to the external URL from inside Docker resulted in routing failures
3. **TLS Certificate Validation**
- Initial attempts used HTTPS with the internal hostname `authentik_server`
- The TLS certificate was valid for `*.castaldifamily.com`, not for the Docker internal hostname
- This contributed to connection failures even when using the correct internal endpoint
4. **Docker Network DNS Resolution**
- While Docker DNS correctly resolved `authentik_server` to `172.18.0.31`, there appeared to be Host header validation differences
- Direct `wget`/`curl` requests from containers to `http://authentik_server:9000` returned 404
- The same requests using the IP address `http://172.18.0.31:9000` succeeded
- However, the Authentik outpost application code successfully connected using the hostname
5. **GitOps Deployment Timing**
- Changes to compose files were deployed via Git commits triggering Komodo
- This created a delay between file changes and container updates
- Multiple iterations were required before all configuration aligned
---
## What Worked vs. What Didn't
### What Didn't Work
| Approach | Configuration | Result | Reason |
|----------|---------------|--------|--------|
| External HTTPS URL | `https://sso.castaldifamily.com` | 404 errors | Hairpin NAT prevented internal Docker containers from reaching external IP |
| Internal HTTPS with hostname | `https://authentik_server:9443` | 404 errors | TLS certificate mismatch + database config override |
| Blank/null `authentik_host` | Attempted to clear field | Validation error | Authentik requires a string value, not null |
| IP address in env var only | `AUTHENTIK_HOST=http://172.18.0.31:9000` (env only) | 404 errors | Database config still had old value, which took precedence |
### What Worked
| Approach | Configuration | Result | Key Success Factor |
|----------|---------------|--------|-------------------|
| Internal HTTP with hostname | **Admin Panel:** `authentik_host=http://authentik_server:9000`<br>**Env Var:** `AUTHENTIK_HOST=http://authentik_server:9000` | ✅ Success | Both configurations aligned; outpost application code handles hostname correctly |
| Insecure flag | `AUTHENTIK_INSECURE=true` and `authentik_host_insecure=true` | ✅ Success | Disabled TLS verification for internal Docker network |
| Integration type | Changed to "No integration active" | ✅ Success | Removed automatic configuration interference |
---
## Technical Deep Dive
### Authentik Outpost Configuration Flow
```
┌─────────────────────────────────────────────────────────────┐
│ 1. Outpost Container Starts │
│ - Loads environment variables (AUTHENTIK_HOST, etc.) │
│ - Initializes with fallback values │
└──────────────────────┬──────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────┐
│ 2. Query Authentik API │
│ GET /api/v3/outposts/instances/ │
│ Authorization: Bearer {AUTHENTIK_TOKEN} │
└──────────────────────┬──────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────┐
│ 3. Receive Configuration from Database │
│ { │
│ "config": { │
│ "authentik_host": "http://authentik_server:9000", │
│ "authentik_host_browser": "", │
│ "authentik_host_insecure": true │
│ } │
│ } │
└──────────────────────┬──────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────┐
│ 4. Apply Configuration (Database Overrides Environment) │
│ IF config.authentik_host EXISTS AND NOT EMPTY: │
│ USE config.authentik_host │
│ ELSE: │
│ USE AUTHENTIK_HOST environment variable │
└──────────────────────┬──────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────┐
│ 5. Connect to Authentik Server │
│ - Fetch application configuration │
│ - Load providers, flows, policies │
│ - Initialize proxy routes │
└─────────────────────────────────────────────────────────────┘
```
### Network Topology
```
External Network (Internet)
│
│ Port Forwarding: 80→10.0.0.151:80, 443→10.0.0.151:443
│
▼
┌────────────────────────────────────────────────────────────┐
│ Heimdall Server (10.0.0.151) │
│ │
│ Docker Network: proxy-net (172.18.0.0/16) │
│ ┌─────────────────────────────────────────────────────┐ │
│ │ Traefik (172.18.0.x) │ │
│ │ - Listens: 0.0.0.0:80, 0.0.0.0:443 │ │
│ │ - Routes based on Host header │ │
│ └──────────┬──────────────────────────────────────────┘ │
│ │ │
│ ┌──────────┴──────────────────────────────────────────┐ │
│ │ Authentik Server (172.18.0.31) │ │
│ │ - Ports: 9000 (HTTP), 9443 (HTTPS) │ │
│ │ - Hostname: authentik_server │ │
│ └──────────┬──────────────────────────────────────────┘ │
│ │ │
│ ┌──────────┴──────────────────────────────────────────┐ │
│ │ Proxy Outposts │ │
│ │ - prowlarr_outpost (172.18.0.x) │ │
│ │ - radarr_outpost (172.18.0.x) │ │
│ │ - sonarr_outpost (172.18.0.x) │ │
│ │ - sabnzbd_outpost (172.18.0.x) │ │
│ │ │ │
│ │ Each outpost connects to: │ │
│ │ http://authentik_server:9000/api/v3/... │ │
│ └─────────────────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────┘
DNS: *.castaldifamily.com → 73.106.212.82 (External IP)
Hairpin NAT Issue: Internal clients can't reach external IP
```
### Key Configuration Files
**Working Configuration:**
```yaml
# nodes/heimdall/prowlarr/compose.yaml
services:
prowlarr_outpost:
image: ghcr.io/goauthentik/proxy:2026.2.2
container_name: prowlarr_outpost
environment:
AUTHENTIK_HOST: http://authentik_server:9000 # ✅ Matches admin panel
AUTHENTIK_HOST_BROWSER: https://sso.castaldifamily.com
AUTHENTIK_TOKEN: 42FCcV9gmTfixaak77xW4eAZIMUUJ0u5vGsxvumfo1Lav5DIyLViDz4xqinE
AUTHENTIK_INSECURE: true # ✅ Disables TLS verification
networks:
- proxy-net
```
**Authentik Admin Panel:**
- Navigate to: Applications → Outposts → prowlarr_outpost
- Configuration section:
- `authentik_host`: `http://authentik_server:9000` ✅
- `authentik_host_insecure`: `true` ✅
- `authentik_host_browser`: (empty/blank) ✅
- Integration: "No integration active"
---
## Lessons Learned
### What We Learned
1. **Database Configuration Takes Precedence**
- Authentik outposts fetch configuration from the database via API
- This configuration overrides container environment variables
- Both sources must be aligned for successful operation
2. **Field Validation Requirements**
- Authentik requires `authentik_host` to be a valid string
- Cannot be set to `null`, `None`, or blank in the admin panel
- Must provide a valid URL even if using environment variables
3. **Internal vs. External URLs**
- External URLs (e.g., `https://sso.castaldifamily.com`) don't work for internal Docker communication
- Hairpin NAT prevents containers from reaching the host's external IP
- Internal Docker hostnames (e.g., `authentik_server`) or IP addresses work correctly
4. **Protocol and Port Selection**
- HTTPS (port 9443) requires valid TLS certificates matching the hostname
- HTTP (port 9000) works for internal Docker network communication
- Using `AUTHENTIK_INSECURE=true` bypasses TLS validation when necessary
5. **GitOps Deployment Considerations**
- Environment variable changes require container recreation
- Komodo auto-deployment has a slight delay after git push
- Must verify actual container environment after deployment
- Database configuration changes take effect immediately on outpost restart
6. **Debugging Methodology**
- Check container logs first for immediate error messages
- Verify environment variables inside running containers (`docker exec`)
- Query the Authentik API to see database configuration
- Test network connectivity from inside containers
- Distinguish between configuration issues and network issues
### What Went Well
1. **Systematic Troubleshooting**
- Methodical progression from symptoms to root cause
- Each hypothesis was tested with verification steps
- Logs and API responses provided clear evidence
2. **Infrastructure Observability**
- Container logs accessible and informative
- API endpoints available for configuration introspection
- Docker network inspection tools worked reliably
3. **GitOps Workflow**
- Changes tracked in version control
- Automated deployment reduced manual errors
- Rollback capability available if needed
### What Could Be Improved
1. **Documentation Gaps**
- Need better documentation of Authentik configuration precedence
- Should document the relationship between environment variables and database config
- Create runbook for Authentik outpost troubleshooting
2. **Initial Configuration**
- Should have used internal URLs from the start
- Better planning for Docker network topology
- More thorough testing during initial setup
3. **Monitoring and Alerting**
- No automated alerts when outposts fail to connect
- No health checks monitoring outpost status
- Could benefit from Grafana dashboard showing outpost health
4. **Testing Procedures**
- Should have tested outpost connectivity before production use
- Need integration tests for Authentik configuration changes
- Should validate both internal and external access paths
---
## Prevention Measures
### Immediate Actions Taken
1. ✅ **Documented Working Configuration**
- This RCA serves as reference for future troubleshooting
- Configuration examples included in this document
2. ✅ **Verified All Outposts**
- Confirmed prowlarr_outpost, radarr_outpost, sonarr_outpost, sabnzbd_outpost all functioning
- Validated logs show successful authentication and proxying
3. ✅ **Updated Compose Files**
- All outpost compose files now use correct internal configuration
- Changes committed to version control
### Recommended Future Actions
1. **Create KBA Document**
- Knowledge Base Article for Authentik outpost troubleshooting
- Include configuration precedence explanation
- Step-by-step diagnostic procedures
2. **Create SOP for Authentik Outpost Deployment**
- Standard Operating Procedure for adding new outposts
- Configuration checklist:
- [ ] Set `authentik_host` in admin panel to internal URL
- [ ] Set `authentik_host_insecure=true` for HTTP
- [ ] Leave `authentik_host_browser` empty or set to external URL
- [ ] Set Integration to "No integration active"
- [ ] Match environment variables to admin panel config
- [ ] Verify logs show "Loaded application" after deployment
- [ ] Test external access through Traefik
3. **Implement Health Monitoring**
- Add Prometheus metrics export from Authentik
- Create Grafana dashboard for outpost status
- Set up alerting for outpost connection failures
- Monitor API endpoint `/api/v3/outposts/instances/` health
4. **Enhance Network Configuration**
- Consider implementing internal DNS (Pi-hole or similar)
- Resolve `*.castaldifamily.com` to `10.0.0.151` internally
- Eliminates hairpin NAT issues for internal clients
5. **Automate Configuration Validation**
- Script to verify compose file and admin panel config match
- Pre-deployment checks for Authentik outposts
- Automated testing of outpost connectivity
6. **Documentation Improvements**
- Update homelab documentation with Authentik architecture
- Document Docker network topology
- Create troubleshooting flowchart for authentication issues
---
## References
### Related Documentation
- [Traefik Configuration](../nodes/heimdall/core/compose.yaml)
- [Authentik Server Configuration](../nodes/heimdall/authentik/compose.yaml)
- [Prowlarr Outpost Configuration](../nodes/heimdall/prowlarr/compose.yaml)
### External Resources
- [Authentik Outposts Documentation](https://docs.goauthentik.io/docs/outposts/)
- [Authentik Proxy Provider](https://docs.goauthentik.io/docs/providers/proxy/)
- [Docker Networking](https://docs.docker.com/network/)
### Git Commits Related to This Incident
- `f770331` - Initial outpost environment variable updates (HTTPS, insecure flag)
- `f10155a` - Changed AUTHENTIK_HOST to HTTP port 9000
- `7b250b9` - Attempted IP address configuration (reverted)
---
## Appendix A: Error Messages
### Typical 404 Error Log Entry
```json
{
"error": "404 Not Found",
"event": "Failed to fetch outpost configuration, retrying in 3 seconds",
"level": "error",
"logger": "authentik.outpost.ak-api-controller",
"timestamp": "2026-06-14T13:05:57Z"
}
```
### Successful Connection Log Entry
```json
{
"event": "Loaded application",
"host": "prowlarr.castaldifamily.com",
"level": "info",
"logger": "authentik.outpost.proxyv2",
"name": "Provider for Prowlarr",
"timestamp": "2026-06-14T13:22:05Z"
}
```
---
## Appendix B: Diagnostic Commands
### Check Outpost Container Logs
```bash
ssh chester@10.0.0.151 'docker logs prowlarr_outpost 2>&1 | tail -25'
```
### Verify Container Environment Variables
```bash
ssh chester@10.0.0.151 'docker exec prowlarr_outpost env | grep AUTHENTIK'
```
### Query Authentik API Configuration
```bash
curl -k -s \
-H "Authorization: Bearer {AUTHENTIK_TOKEN}" \
https://10.0.0.151:9443/api/v3/outposts/instances/ | jq
```
### Test Connectivity from Inside Container
```bash
ssh chester@10.0.0.151 'docker exec prowlarr_outpost wget -O- \
--header="Authorization: Bearer {AUTHENTIK_TOKEN}" \
http://authentik_server:9000/api/v3/outposts/instances/ 2>&1'
```
### Check Docker Network Details
```bash
ssh chester@10.0.0.151 'docker inspect authentik_server \
--format="{{json .NetworkSettings.Networks}}" | jq'
```
---
## Approval and Sign-off
**Prepared By:** AI Assistant (Frank v6) - Homelab Operations
**Date:** 2026-06-14
**Status:** Draft
**Technical Review:** Pending
**Management Approval:** Pending
---
*Document Version: 1.0*
*Last Updated: 2026-06-14*
*Next Review Date: 2026-07-14*
Reference in New Issue View Git Blame Copy Permalink