# Root Cause Analysis: Authentik Proxy Outpost Configuration Failure **Incident ID:** RCA-001 **Date:** 2026-06-14 **Severity:** High (Service Outage) **Status:** Resolved **Author:** AI Assistant (Frank v6) --- ## Executive Summary Four Authentik proxy outpost containers (prowlarr_outpost, radarr_outpost, sonarr_outpost, sabnzbd_outpost) failed to connect to the Authentik server, resulting in complete inaccessibility of protected services. The outposts continuously returned `404 Not Found` errors when attempting to fetch configuration from the Authentik API. **Root Cause:** Authentik outpost configuration stored in the database overrides container environment variables. The `authentik_host` field in the Authentik admin panel contained values that prevented the outposts from successfully connecting to the Authentik server. **Resolution:** Updated the `authentik_host` field in Authentik admin panel to `http://authentik_server:9000` for all four outposts, matching the internal Docker network configuration. **Impact Duration:** ~2 hours **Services Affected:** Prowlarr, Radarr, Sonarr, Sabnzbd --- ## Timeline ### Initial State (Before Incident) - Services running but not externally accessible - Hairpin NAT issue preventing internal clients from using external DNS - Services reachable internally via `10.0.0.151` - Traefik reverse proxy functioning correctly ### 13:00 UTC - Problem Discovery - User reported services accessible internally but returning 404 externally - Initial hypothesis: Traefik routing or DNS issue - Verified Traefik listening on `0.0.0.0:80` and `0.0.0.0:443` - Confirmed DNS resolving correctly to external IP `73.106.212.82` ### 13:05 UTC - Authentik Outpost Investigation Begins - Examined outpost container logs - Found continuous `404 Not Found` errors in all four outpost containers: ``` {"error":"404 Not Found","event":"Failed to fetch outpost configuration, retrying in 3 seconds","level":"error","logger":"authentik.outpost.ak-api-controller"} ``` - Observed exponential backoff pattern (3s → 6s → 12s → 24s → 48s) ### 13:15 UTC - Configuration Analysis - Checked container environment variables via `docker exec` - Initial configuration: - `AUTHENTIK_HOST=https://authentik_server:9443` - `AUTHENTIK_HOST_BROWSER=https://sso.castaldifamily.com` - `AUTHENTIK_INSECURE=false` ### 13:20 UTC - First Remediation Attempt - Updated compose files to use HTTPS with insecure flag: - Changed `AUTHENTIK_HOST` to `https://authentik_server:9443` - Added `AUTHENTIK_HOST_BROWSER=https://sso.castaldifamily.com` - Set `AUTHENTIK_INSECURE=true` - Deployed via Git push to Komodo - **Result:** Failed - 404 errors persisted ### 13:30 UTC - Authentik Admin Panel Configuration - Changed Integration type from "Local Docker connection" to "No integration active" - Set `authentik_host_insecure=true` in admin panel - Verified via API that changes were reflected - **Result:** Failed - 404 errors persisted ### 13:40 UTC - Discovery of Configuration Precedence Issue - Queried Authentik API: `/api/v3/outposts/instances/` - Discovered `authentik_host` in API response: `"https://sso.castaldifamily.com/"` - Identified that database configuration was overriding environment variables - User attempted to clear `authentik_host` field (set to null/blank) - **Result:** Failed - Authentik validation error: "wrong value type for field 'authentik_host' - should be 'str' instead of value 'None' of type 'NoneType'" ### 13:50 UTC - Protocol and Port Testing - Switched from HTTPS port 9443 to HTTP port 9000 - Updated compose files: `AUTHENTIK_HOST=http://authentik_server:9000` - Updated Authentik admin: `authentik_host=http://authentik_server:9000` - **Result:** Failed - 404 errors persisted ### 13:55 UTC - Network Debugging - Tested API accessibility from inside outpost container - Using hostname: `wget http://authentik_server:9000/api/v3/outposts/instances/` → 404 - Using IP address: `wget http://172.18.0.31:9000/api/v3/outposts/instances/` → 200 OK (Success) - **Discovery:** Host header validation issue suspected ### 14:00 UTC - IP Address Configuration Attempt - Updated compose files to use IP: `AUTHENTIK_HOST=http://172.18.0.31:9000` - Prepared to update Authentik admin panel - User reported success using hostname `http://authentik_server:9000` in admin panel - **Result:** Success - Outposts connected successfully ### 14:05 UTC - Resolution Confirmed - Logs showed `"Loaded application"` messages - Successful proxy requests with HTTP 200 status codes - Services accessible externally through Authentik authentication - All four outposts (prowlarr, radarr, sonarr, sabnzbd) operational --- ## Root Cause Analysis ### Primary Root Cause **Configuration Precedence:** Authentik proxy outposts prioritize configuration stored in the Authentik database over container environment variables. When an outpost starts, it queries the Authentik API at `/api/v3/outposts/instances/` to fetch its configuration. The response includes a `config` object with an `authentik_host` field. If this field has a value in the database, it completely overrides the `AUTHENTIK_HOST` environment variable. ### Contributing Factors 1. **Incomplete Documentation Understanding** - The interaction between environment variables and database configuration was not well understood - Authentik documentation does not clearly state that database values override environment variables - The admin panel allows setting `authentik_host` but does not explain when to use it vs. environment variables 2. **External URL in Database Configuration** - Initial Authentik setup stored `https://sso.castaldifamily.com/` as the `authentik_host` - This external URL was inaccessible from within the Docker network due to hairpin NAT limitations - The outposts attempting to connect to the external URL from inside Docker resulted in routing failures 3. **TLS Certificate Validation** - Initial attempts used HTTPS with the internal hostname `authentik_server` - The TLS certificate was valid for `*.castaldifamily.com`, not for the Docker internal hostname - This contributed to connection failures even when using the correct internal endpoint 4. **Docker Network DNS Resolution** - While Docker DNS correctly resolved `authentik_server` to `172.18.0.31`, there appeared to be Host header validation differences - Direct `wget`/`curl` requests from containers to `http://authentik_server:9000` returned 404 - The same requests using the IP address `http://172.18.0.31:9000` succeeded - However, the Authentik outpost application code successfully connected using the hostname 5. **GitOps Deployment Timing** - Changes to compose files were deployed via Git commits triggering Komodo - This created a delay between file changes and container updates - Multiple iterations were required before all configuration aligned --- ## What Worked vs. What Didn't ### What Didn't Work | Approach | Configuration | Result | Reason | |----------|---------------|--------|--------| | External HTTPS URL | `https://sso.castaldifamily.com` | 404 errors | Hairpin NAT prevented internal Docker containers from reaching external IP | | Internal HTTPS with hostname | `https://authentik_server:9443` | 404 errors | TLS certificate mismatch + database config override | | Blank/null `authentik_host` | Attempted to clear field | Validation error | Authentik requires a string value, not null | | IP address in env var only | `AUTHENTIK_HOST=http://172.18.0.31:9000` (env only) | 404 errors | Database config still had old value, which took precedence | ### What Worked | Approach | Configuration | Result | Key Success Factor | |----------|---------------|--------|-------------------| | Internal HTTP with hostname | **Admin Panel:** `authentik_host=http://authentik_server:9000`
**Env Var:** `AUTHENTIK_HOST=http://authentik_server:9000` | ✅ Success | Both configurations aligned; outpost application code handles hostname correctly | | Insecure flag | `AUTHENTIK_INSECURE=true` and `authentik_host_insecure=true` | ✅ Success | Disabled TLS verification for internal Docker network | | Integration type | Changed to "No integration active" | ✅ Success | Removed automatic configuration interference | --- ## Technical Deep Dive ### Authentik Outpost Configuration Flow ``` ┌─────────────────────────────────────────────────────────────┐ │ 1. Outpost Container Starts │ │ - Loads environment variables (AUTHENTIK_HOST, etc.) │ │ - Initializes with fallback values │ └──────────────────────┬──────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────┐ │ 2. Query Authentik API │ │ GET /api/v3/outposts/instances/ │ │ Authorization: Bearer {AUTHENTIK_TOKEN} │ └──────────────────────┬──────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────┐ │ 3. Receive Configuration from Database │ │ { │ │ "config": { │ │ "authentik_host": "http://authentik_server:9000", │ │ "authentik_host_browser": "", │ │ "authentik_host_insecure": true │ │ } │ │ } │ └──────────────────────┬──────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────┐ │ 4. Apply Configuration (Database Overrides Environment) │ │ IF config.authentik_host EXISTS AND NOT EMPTY: │ │ USE config.authentik_host │ │ ELSE: │ │ USE AUTHENTIK_HOST environment variable │ └──────────────────────┬──────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────┐ │ 5. Connect to Authentik Server │ │ - Fetch application configuration │ │ - Load providers, flows, policies │ │ - Initialize proxy routes │ └─────────────────────────────────────────────────────────────┘ ``` ### Network Topology ``` External Network (Internet) │ │ Port Forwarding: 80→10.0.0.151:80, 443→10.0.0.151:443 │ ▼ ┌────────────────────────────────────────────────────────────┐ │ Heimdall Server (10.0.0.151) │ │ │ │ Docker Network: proxy-net (172.18.0.0/16) │ │ ┌─────────────────────────────────────────────────────┐ │ │ │ Traefik (172.18.0.x) │ │ │ │ - Listens: 0.0.0.0:80, 0.0.0.0:443 │ │ │ │ - Routes based on Host header │ │ │ └──────────┬──────────────────────────────────────────┘ │ │ │ │ │ ┌──────────┴──────────────────────────────────────────┐ │ │ │ Authentik Server (172.18.0.31) │ │ │ │ - Ports: 9000 (HTTP), 9443 (HTTPS) │ │ │ │ - Hostname: authentik_server │ │ │ └──────────┬──────────────────────────────────────────┘ │ │ │ │ │ ┌──────────┴──────────────────────────────────────────┐ │ │ │ Proxy Outposts │ │ │ │ - prowlarr_outpost (172.18.0.x) │ │ │ │ - radarr_outpost (172.18.0.x) │ │ │ │ - sonarr_outpost (172.18.0.x) │ │ │ │ - sabnzbd_outpost (172.18.0.x) │ │ │ │ │ │ │ │ Each outpost connects to: │ │ │ │ http://authentik_server:9000/api/v3/... │ │ │ └─────────────────────────────────────────────────────┘ │ │ │ └─────────────────────────────────────────────────────────────┘ DNS: *.castaldifamily.com → 73.106.212.82 (External IP) Hairpin NAT Issue: Internal clients can't reach external IP ``` ### Key Configuration Files **Working Configuration:** ```yaml # nodes/heimdall/prowlarr/compose.yaml services: prowlarr_outpost: image: ghcr.io/goauthentik/proxy:2026.2.2 container_name: prowlarr_outpost environment: AUTHENTIK_HOST: http://authentik_server:9000 # ✅ Matches admin panel AUTHENTIK_HOST_BROWSER: https://sso.castaldifamily.com AUTHENTIK_TOKEN: 42FCcV9gmTfixaak77xW4eAZIMUUJ0u5vGsxvumfo1Lav5DIyLViDz4xqinE AUTHENTIK_INSECURE: true # ✅ Disables TLS verification networks: - proxy-net ``` **Authentik Admin Panel:** - Navigate to: Applications → Outposts → prowlarr_outpost - Configuration section: - `authentik_host`: `http://authentik_server:9000` ✅ - `authentik_host_insecure`: `true` ✅ - `authentik_host_browser`: (empty/blank) ✅ - Integration: "No integration active" --- ## Lessons Learned ### What We Learned 1. **Database Configuration Takes Precedence** - Authentik outposts fetch configuration from the database via API - This configuration overrides container environment variables - Both sources must be aligned for successful operation 2. **Field Validation Requirements** - Authentik requires `authentik_host` to be a valid string - Cannot be set to `null`, `None`, or blank in the admin panel - Must provide a valid URL even if using environment variables 3. **Internal vs. External URLs** - External URLs (e.g., `https://sso.castaldifamily.com`) don't work for internal Docker communication - Hairpin NAT prevents containers from reaching the host's external IP - Internal Docker hostnames (e.g., `authentik_server`) or IP addresses work correctly 4. **Protocol and Port Selection** - HTTPS (port 9443) requires valid TLS certificates matching the hostname - HTTP (port 9000) works for internal Docker network communication - Using `AUTHENTIK_INSECURE=true` bypasses TLS validation when necessary 5. **GitOps Deployment Considerations** - Environment variable changes require container recreation - Komodo auto-deployment has a slight delay after git push - Must verify actual container environment after deployment - Database configuration changes take effect immediately on outpost restart 6. **Debugging Methodology** - Check container logs first for immediate error messages - Verify environment variables inside running containers (`docker exec`) - Query the Authentik API to see database configuration - Test network connectivity from inside containers - Distinguish between configuration issues and network issues ### What Went Well 1. **Systematic Troubleshooting** - Methodical progression from symptoms to root cause - Each hypothesis was tested with verification steps - Logs and API responses provided clear evidence 2. **Infrastructure Observability** - Container logs accessible and informative - API endpoints available for configuration introspection - Docker network inspection tools worked reliably 3. **GitOps Workflow** - Changes tracked in version control - Automated deployment reduced manual errors - Rollback capability available if needed ### What Could Be Improved 1. **Documentation Gaps** - Need better documentation of Authentik configuration precedence - Should document the relationship between environment variables and database config - Create runbook for Authentik outpost troubleshooting 2. **Initial Configuration** - Should have used internal URLs from the start - Better planning for Docker network topology - More thorough testing during initial setup 3. **Monitoring and Alerting** - No automated alerts when outposts fail to connect - No health checks monitoring outpost status - Could benefit from Grafana dashboard showing outpost health 4. **Testing Procedures** - Should have tested outpost connectivity before production use - Need integration tests for Authentik configuration changes - Should validate both internal and external access paths --- ## Prevention Measures ### Immediate Actions Taken 1. ✅ **Documented Working Configuration** - This RCA serves as reference for future troubleshooting - Configuration examples included in this document 2. ✅ **Verified All Outposts** - Confirmed prowlarr_outpost, radarr_outpost, sonarr_outpost, sabnzbd_outpost all functioning - Validated logs show successful authentication and proxying 3. ✅ **Updated Compose Files** - All outpost compose files now use correct internal configuration - Changes committed to version control ### Recommended Future Actions 1. **Create KBA Document** - Knowledge Base Article for Authentik outpost troubleshooting - Include configuration precedence explanation - Step-by-step diagnostic procedures 2. **Create SOP for Authentik Outpost Deployment** - Standard Operating Procedure for adding new outposts - Configuration checklist: - [ ] Set `authentik_host` in admin panel to internal URL - [ ] Set `authentik_host_insecure=true` for HTTP - [ ] Leave `authentik_host_browser` empty or set to external URL - [ ] Set Integration to "No integration active" - [ ] Match environment variables to admin panel config - [ ] Verify logs show "Loaded application" after deployment - [ ] Test external access through Traefik 3. **Implement Health Monitoring** - Add Prometheus metrics export from Authentik - Create Grafana dashboard for outpost status - Set up alerting for outpost connection failures - Monitor API endpoint `/api/v3/outposts/instances/` health 4. **Enhance Network Configuration** - Consider implementing internal DNS (Pi-hole or similar) - Resolve `*.castaldifamily.com` to `10.0.0.151` internally - Eliminates hairpin NAT issues for internal clients 5. **Automate Configuration Validation** - Script to verify compose file and admin panel config match - Pre-deployment checks for Authentik outposts - Automated testing of outpost connectivity 6. **Documentation Improvements** - Update homelab documentation with Authentik architecture - Document Docker network topology - Create troubleshooting flowchart for authentication issues --- ## References ### Related Documentation - [Traefik Configuration](../nodes/heimdall/core/compose.yaml) - [Authentik Server Configuration](../nodes/heimdall/authentik/compose.yaml) - [Prowlarr Outpost Configuration](../nodes/heimdall/prowlarr/compose.yaml) ### External Resources - [Authentik Outposts Documentation](https://docs.goauthentik.io/docs/outposts/) - [Authentik Proxy Provider](https://docs.goauthentik.io/docs/providers/proxy/) - [Docker Networking](https://docs.docker.com/network/) ### Git Commits Related to This Incident - `f770331` - Initial outpost environment variable updates (HTTPS, insecure flag) - `f10155a` - Changed AUTHENTIK_HOST to HTTP port 9000 - `7b250b9` - Attempted IP address configuration (reverted) --- ## Appendix A: Error Messages ### Typical 404 Error Log Entry ```json { "error": "404 Not Found", "event": "Failed to fetch outpost configuration, retrying in 3 seconds", "level": "error", "logger": "authentik.outpost.ak-api-controller", "timestamp": "2026-06-14T13:05:57Z" } ``` ### Successful Connection Log Entry ```json { "event": "Loaded application", "host": "prowlarr.castaldifamily.com", "level": "info", "logger": "authentik.outpost.proxyv2", "name": "Provider for Prowlarr", "timestamp": "2026-06-14T13:22:05Z" } ``` --- ## Appendix B: Diagnostic Commands ### Check Outpost Container Logs ```bash ssh chester@10.0.0.151 'docker logs prowlarr_outpost 2>&1 | tail -25' ``` ### Verify Container Environment Variables ```bash ssh chester@10.0.0.151 'docker exec prowlarr_outpost env | grep AUTHENTIK' ``` ### Query Authentik API Configuration ```bash curl -k -s \ -H "Authorization: Bearer {AUTHENTIK_TOKEN}" \ https://10.0.0.151:9443/api/v3/outposts/instances/ | jq ``` ### Test Connectivity from Inside Container ```bash ssh chester@10.0.0.151 'docker exec prowlarr_outpost wget -O- \ --header="Authorization: Bearer {AUTHENTIK_TOKEN}" \ http://authentik_server:9000/api/v3/outposts/instances/ 2>&1' ``` ### Check Docker Network Details ```bash ssh chester@10.0.0.151 'docker inspect authentik_server \ --format="{{json .NetworkSettings.Networks}}" | jq' ``` --- ## Approval and Sign-off **Prepared By:** AI Assistant (Frank v6) - Homelab Operations **Date:** 2026-06-14 **Status:** Draft **Technical Review:** Pending **Management Approval:** Pending --- *Document Version: 1.0* *Last Updated: 2026-06-14* *Next Review Date: 2026-07-14*