homelab/ansible/archive/documentation/playbooks/onboard-ansible-secrets.md

Ansible secrets onboarding playbook

Overview

This guide onboards secret management for passwords, API keys, and tokens using Ansible Vault. It defines a repeatable workflow for creating encrypted variable files, loading them safely in playbooks, and consuming secrets with idempotent Ansible modules.

What this establishes

1. Standard secret file layout

• group_vars/<group>/vault.yml for group-level secrets
• host_vars/<host>/vault.yml for host-level secrets
• Secret variable names with _pass or _secret suffixes

2. Encrypted-at-rest secret storage

• Secrets are created and edited with ansible-vault
• Plaintext secrets are not committed to Git
• Existing ignore rules in ansible/.gitignore protect vault files from accidental commits

3. Safe secret consumption patterns

• Use ansible.builtin.template, ansible.builtin.copy, and ansible.builtin.lineinfile instead of ad-hoc shell commands
• Mark sensitive tasks with no_log: true
• Set explicit file ownership and mode for rendered secret files

Prerequisites

• Ansible installed on the control node
• Access to ansible.cfg and your inventory
• A vault password strategy:
  • Interactive prompt (--ask-vault-pass) for manual runs
  • Password file (--vault-password-file) for controlled automation

Important

Do not store vault passwords in repository files or plaintext notes.

Step-by-step onboarding

Step 1: Create vault files

# Group-level secrets
ansible-vault create group_vars/docker/vault.yml

# Host-level secrets
ansible-vault create host_vars/docker-01/vault.yml

Step 2: Add secrets with naming conventions

# group_vars/docker/vault.yml
grafana_admin_pass: "replace-me"
watchtower_api_key_secret: "replace-me"

Step 3: Reference secrets in playbooks or roles

# playbooks/example.yml
- name: Configure app secrets
  hosts: docker_hosts
  become: true
  tasks:
    - name: Render application environment file
      ansible.builtin.template:
        src: templates/app.env.j2
        dest: /opt/app/.env
        owner: root
        group: root
        mode: "0600"
      no_log: true

# templates/app.env.j2
GRAFANA_ADMIN_PASSWORD={{ grafana_admin_pass }}
WATCHTOWER_API_KEY={{ watchtower_api_key_secret }}

Step 4: Run with vault decryption

# Interactive
ansible-playbook -i inventory/hosts.ini playbooks/example.yml --ask-vault-pass

# Automated (secured local file)
ansible-playbook -i inventory/hosts.ini playbooks/example.yml \
  --vault-password-file ~/.ansible/.vault-pass

Step 5: Verify idempotency and secrecy

# Syntax check
ansible-playbook -i inventory/hosts.ini playbooks/example.yml --syntax-check

# Idempotency check (run twice; second run should be unchanged)
ansible-playbook -i inventory/hosts.ini playbooks/example.yml --ask-vault-pass
ansible-playbook -i inventory/hosts.ini playbooks/example.yml --ask-vault-pass

Why module-first instead of shell

• ansible.builtin.template and ansible.builtin.copy are idempotent and track file diffs
• Explicit owner, group, and mode improve auditability
• shell can leak secrets into command history and logs if not handled carefully
• Module output is safer to control with no_log: true

Security guardrails

• Keep no_log: true on any task that reads, writes, or debugs secret values
• Never print secret variables with ansible.builtin.debug
• Scope secrets to the narrowest level possible (host before group when needed)
• Rotate credentials by updating vault values and re-running playbooks
• Prefer separate vault files per scope to limit blast radius

Troubleshooting

Decryption failed

ansible-vault view group_vars/docker/vault.yml

Use the same vault password source used during file creation.

Variable is undefined

• Confirm secret file path matches inventory group/host names
• Confirm variable names match exactly in templates and tasks
• Run with -vv and inspect which variable files loaded

Secret file committed by mistake

1. Rotate affected credentials immediately
2. Remove file from tracking
3. Rewrite Git history if secrets were pushed to remote

Integration notes

• Follow the quality checklist in Ansible quality gates
• Keep naming aligned with Naming conventions