homelab/ansible/archive/documentation/playbooks/mount_nfs_shares.md

# Mount NFS Shares

**Playbook:** `playbooks/storage/mount_nfs_shares.yml`  
**Purpose:** Configure NFS client mounts on Docker Swarm nodes for persistent storage  
**Target:** All Swarm nodes (managers + workers)

---

## Overview

This playbook configures NFS mounts from the TerraMaster NAS to Docker Swarm nodes, providing shared storage for application data and media files. It ensures all nodes have consistent access to centralized storage while maintaining the storage contract principle that NAS is not a dependency for Swarm control-plane operations.

---

## Prerequisites

### On TerraMaster NAS (10.0.0.250)

* NFS service enabled
* Two NFS exports configured:
  * `/Volume1/appdata` — Application data, configs, persistent volumes
  * `/Volume2/media` — Media files (Plex, etc.)
* NFS permissions allow access from Swarm subnet (10.0.0.0/24)

### On Swarm Nodes

* Ubuntu 24.04 LTS (Noble)
* SSH access as `chester` user with sudo privileges
* Network connectivity to TerraMaster on port 2049 (NFS)

---

## What It Does

1. **Installs NFS client** — `nfs-common` package
2. **Creates mount points** — `/mnt/homelab` and `/mnt/media`
3. **Configures fstab** — Persistent mounts survive reboots
4. **Mounts shares immediately** — Makes storage available without reboot
5. **Verifies accessibility** — Tests that mounts are readable

---

## Usage

### Run on all Swarm nodes

```bash
cd /home/chester/homelab/ansible
ansible-playbook playbooks/storage/mount_nfs_shares.yml
```

### Run with specific tags

```bash
# Only install packages and create directories
ansible-playbook playbooks/storage/mount_nfs_shares.yml --tags setup

# Only update fstab (no mount action)
ansible-playbook playbooks/storage/mount_nfs_shares.yml --tags config

# Mount without fstab changes (testing)
ansible-playbook playbooks/storage/mount_nfs_shares.yml --tags mount

# Verify existing mounts
ansible-playbook playbooks/storage/mount_nfs_shares.yml --tags verify
```

### Limit to specific nodes

```bash
# Only managers
ansible-playbook playbooks/storage/mount_nfs_shares.yml --limit swarm_managers

# Only workers
ansible-playbook playbooks/storage/mount_nfs_shares.yml --limit swarm_workers

# Single node
ansible-playbook playbooks/storage/mount_nfs_shares.yml --limit swarm-worker-1
```

---

## Configuration

### Variables

Defined in the playbook (`vars` section):

| Variable | Value | Description |
|----------|-------|-------------|
| `nfs_server` | `10.0.0.250` | TerraMaster NAS IP address |
| `nfs_mounts[0].src` | `/Volume1/appdata` | NFS export path for application data |
| `nfs_mounts[0].dest` | `/mnt/homelab` | Local mount point for app data |
| `nfs_mounts[1].src` | `/Volume2/media` | NFS export path for media |
| `nfs_mounts[1].dest` | `/mnt/media` | Local mount point for media |
| `nfs_mounts[*].opts` | `defaults` | Mount options |

### Customizing Mount Options

To change mount options (e.g., add `noatime` for performance):

```yaml
nfs_mounts:
  - src: "/Volume1/appdata"
    dest: "/mnt/homelab"
    opts: "defaults,noatime,rw"
```

Common NFS options:
- `noatime` — Don't update access times (performance)
- `hard` — Retry indefinitely if NFS server unavailable (default)
- `soft` — Fail after timeout (risky for data integrity)
- `rsize=8192,wsize=8192` — Adjust read/write buffer sizes
- `nfsvers=4` — Force NFSv4 (recommended)

---

## Using NFS Mounts in Docker

### Method 1: Bind Mounts (Current Approach)

**Docker Compose:**
```yaml
services:
  app:
    image: myapp:latest
    volumes:
      - /mnt/homelab/appdata/myapp:/data
      - /mnt/media:/media:ro  # Read-only for safety
```

**Pros:**
- Simple and transparent
- Easy to debug with standard Linux tools
- One mount serves all containers

**Cons:**
- Services coupled to host filesystem paths
- Must ensure mount exists before container starts

---

### Method 2: Docker NFS Volumes (Alternative)

**Docker Compose:**
```yaml
volumes:
  homelab_data:
    driver: local
    driver_opts:
      type: nfs
      o: addr=10.0.0.250,rw,nfsvers=4
      device: ":/Volume1/appdata"
  
  media:
    driver: local
    driver_opts:
      type: nfs
      o: addr=10.0.0.250,ro,nfsvers=4
      device: ":/Volume2/media"

services:
  app:
    image: myapp:latest
    volumes:
      - homelab_data:/data
      - media:/media:ro
```

**Pros:**
- Portable volume names (no hardcoded paths)
- Docker manages mount lifecycle
- Per-service isolation possible
- Automatic retry on NFS failure

**Cons:**
- More complex configuration
- Harder to inspect with standard tools
- Must define volumes in every compose file

---

### Recommendation

**Use bind mounts (Method 1)** for now:
- You already have working fstab configuration
- Simpler to manage across 6 nodes
- Better visibility for troubleshooting
- Can switch to Docker volumes later if needed

---

## Verification

### Check mount status

```bash
# On any Swarm node
df -h | grep mnt

# Expected output:
# 10.0.0.250:/Volume1/appdata  500G  100G  400G  20% /mnt/homelab
# 10.0.0.250:/Volume2/media    2.0T  500G  1.5T  25% /mnt/media
```

### Test write access

```bash
# On a Swarm node
sudo touch /mnt/homelab/test-write
ls -l /mnt/homelab/test-write
sudo rm /mnt/homelab/test-write
```

### Check fstab persistence

```bash
cat /etc/fstab | grep mnt
# Should show both NFS entries
```

---

## Troubleshooting

### Mount fails with "Connection refused"

**Cause:** NFS service not running or firewall blocking port 2049

**Solution:**
```bash
# Test NFS connectivity
showmount -e 10.0.0.250

# If fails, check TerraMaster NFS settings
```

---

### Mount fails with "Permission denied"

**Cause:** NFS export permissions don't allow Swarm node IPs

**Solution:** Update TerraMaster NFS export to allow `10.0.0.0/24` subnet

---

### Mount succeeds but directory is empty

**Cause:** Mounted wrong export path or path doesn't exist on NAS

**Solution:**
```bash
# List available exports
showmount -e 10.0.0.250
```

---

### Mount exists but containers can't write

**Cause:** NFS mounted read-only or wrong permissions

**Solution:**
```bash
# Check mount options
mount | grep "/mnt/homelab"

# Remount with write permissions if needed
sudo mount -o remount,rw /mnt/homelab
```

---

### Stale NFS file handle errors

**Cause:** NFS server restarted or export changed

**Solution:**
```bash
# Unmount and remount
sudo umount -f /mnt/homelab
sudo mount -a
```

---

## Safety Considerations

### Storage Contract Compliance

✅ **Compliant:**
- Mounting NFS on all nodes for data access
- Using NAS for application data (not control-plane state)
- Swarm can operate if NFS is temporarily unavailable

❌ **Violations to avoid:**
- Don't store Swarm raft data on NFS
- Don't run manager services that require NFS to stay healthy
- Don't use NFS for `/var/lib/docker` or other system paths

---

### Backup Verification

Per storage contract:
- Data on `/mnt/homelab` backed up via TerraMaster → Synology rsync
- Verify backup jobs are running: Check Synology logs
- Test restores periodically

---

## Maintenance

### Adding new NFS shares

1. Configure export on TerraMaster
2. Add entry to `nfs_mounts` list in playbook
3. Run playbook with `--tags setup,config,mount`

### Removing NFS shares

1. Unmount: `sudo umount /mnt/someshare`
2. Remove from `/etc/fstab`
3. Remove directory: `sudo rmdir /mnt/someshare`

---

## Related Documentation

- [Storage Contract](../contracts/storage.md) — NAS roles and backup policy
- [Environment Constraints](../standards/environment-constraints.md) — Network and hardware specs
- [Architecture Decisions](../../documentation/standards/architecture-decisions.md) — ADR-003 (Watchtower role)

---

## Tags Reference

| Tag | Purpose |
|-----|---------|
| `setup` | Install packages, create directories |
| `packages` | Install NFS client only |
| `filesystem` | Create mount point directories only |
| `config` | Update fstab only |
| `fstab` | Alias for `config` |
| `mount` | Execute mount operations |
| `verify` | Test mounts and display status |