Implement modular nftables architecture and Gitea SSH firewall management

- Restructure security playbook with modular nftables loader - Base rules loaded first, service rules second, drop rule last - Add Gitea self-contained firewall management (port 2222) - Add fail2ban protection for Gitea SSH brute force attacks - Update documentation with new firewall architecture - Create comprehensive Gitea deployment and testing guide This enables self-contained service roles to manage their own firewall rules without modifying the central security playbook. Each service deploys rules to /etc/nftables.d/ which are loaded before the final drop rule, maintaining the defense-in-depth security model.
2025-12-16 21:45:22 +01:00
parent 9b12225ec8
commit 2fe194ba82
12 changed files with 933 additions and 43 deletions
--- a/docs/gitea-deployment-guide.md
+++ b/docs/gitea-deployment-guide.md
@@ -0,0 +1,541 @@
+# Gitea Deployment and Testing Guide
+
+Comprehensive guide for deploying and testing Gitea Git service with SSH access on rick-infra.
+
+## Deployment
+
+### 1. Prerequisites
+
+Ensure you have the required vault variables set in your host_vars:
+
+```yaml
+# host_vars/arch-vps/vault.yml (encrypted)
+vault_gitea_db_password: "your_secure_password_here"
+```
+
+### 2. Deploy Gitea Role
+
+Run the rick-infra playbook with Gitea role:
+
+```bash
+# Deploy Gitea to arch-vps
+ansible-playbook -i inventory/hosts.yml rick-infra.yml --limit arch-vps
+
+# Or deploy only Gitea role
+ansible-playbook -i inventory/hosts.yml rick-infra.yml --tags gitea --limit arch-vps
+```
+
+### 3. Verify Deployment
+
+Check that all services are running:
+
+```bash
+# SSH into the server
+ssh root@arch-vps
+
+# Check Gitea service status
+systemctl status gitea
+
+# Check if Gitea is listening on HTTP port
+ss -tlnp | grep 3000
+
+# Check if Gitea SSH is listening
+ss -tlnp | grep 2222
+
+# Verify firewall rules
+nft list ruleset | grep 2222
+
+# Check fail2ban status
+fail2ban-client status gitea-ssh
+```
+
+Expected output:
+- Gitea service: `active (running)`
+- HTTP port 3000: listening on `127.0.0.1:3000`
+- SSH port 2222: listening on `0.0.0.0:2222`
+- nftables: Rule allowing `tcp dport 2222`
+- fail2ban: `gitea-ssh` jail active
+
+## Testing Guide
+
+### Test 1: Web Interface Access
+
+**Purpose**: Verify HTTPS access through Caddy reverse proxy
+
+```bash
+# From your local machine
+curl -I https://git.jnss.me
+```
+
+**Expected result**:
+- HTTP/2 200 OK
+- Redirects to login page
+- Valid TLS certificate
+
+**Action**: Open browser to `https://git.jnss.me` and verify web interface loads
+
+---
+
+### Test 2: Firewall Port Verification
+
+**Purpose**: Confirm port 2222 is accessible from external networks
+
+```bash
+# From your local machine (not from the server)
+nc -zv git.jnss.me 2222
+```
+
+**Expected result**:
+```
+Connection to git.jnss.me 2222 port [tcp/*] succeeded!
+```
+
+**If this fails**: The firewall rule is not active or nftables service is not running.
+
+**Troubleshooting**:
+```bash
+# On the server
+ssh root@arch-vps
+
+# Check if nftables service is running
+systemctl status nftables
+
+# List all firewall rules
+nft list ruleset
+
+# Verify Gitea rule file exists
+cat /etc/nftables.d/gitea.nft
+
+# Manually reload nftables
+systemctl reload nftables
+```
+
+---
+
+### Test 3: SSH Connection Test
+
+**Purpose**: Verify Gitea SSH server accepts connections
+
+```bash
+# From your local machine
+ssh -T -p 2222 git@git.jnss.me
+```
+
+**Expected result** (before adding SSH key):
+```
+Hi there, You've successfully authenticated, but Gitea does not provide shell access.
+If this is unexpected, please log in with password and setup Gitea under another user.
+```
+
+**OR** (if authentication fails):
+```
+git@git.jnss.me: Permission denied (publickey).
+```
+
+This is normal - it means Gitea SSH server is responding, you just need to add your SSH key.
+
+**If connection times out**: Port 2222 is blocked or Gitea SSH is not running.
+
+---
+
+### Test 4: SSH Key Setup and Authentication
+
+**Purpose**: Add SSH key to Gitea and test authentication
+
+**Step 4.1**: Create Gitea admin account
+1. Visit `https://git.jnss.me`
+2. Click "Register" (if registration is enabled) or use initial admin setup
+3. Create your user account
+
+**Step 4.2**: Generate SSH key (if needed)
+```bash
+# On your local machine
+ssh-keygen -t ed25519 -C "your_email@example.com"
+
+# View your public key
+cat ~/.ssh/id_ed25519.pub
+```
+
+**Step 4.3**: Add SSH key to Gitea
+1. Log into Gitea web interface
+2. Click your profile → Settings
+3. Click "SSH / GPG Keys" tab
+4. Click "Add Key"
+5. Paste your public key (`id_ed25519.pub` contents)
+6. Give it a name and click "Add Key"
+
+**Step 4.4**: Test SSH authentication
+```bash
+# From your local machine
+ssh -T -p 2222 git@git.jnss.me
+```
+
+**Expected result**:
+```
+Hi there, <your_username>! You've successfully authenticated with the key named <key_name>
+```
+
+---
+
+### Test 5: Repository Operations
+
+**Purpose**: Test actual Git operations over SSH
+
+**Step 5.1**: Create a test repository in Gitea web interface
+1. Click "+" → "New Repository"
+2. Name: `test-repo`
+3. Click "Create Repository"
+
+**Step 5.2**: Clone the repository
+```bash
+# From your local machine
+git clone ssh://git@git.jnss.me:2222/your_username/test-repo.git
+cd test-repo
+```
+
+**Expected result**: Repository clones successfully
+
+**Step 5.3**: Make a commit and push
+```bash
+# Create a test file
+echo "# Test Repository" > README.md
+
+# Commit and push
+git add README.md
+git commit -m "Initial commit"
+git push origin main
+```
+
+**Expected result**: 
+```
+Enumerating objects: 3, done.
+Counting objects: 100% (3/3), done.
+Writing objects: 100% (3/3), 234 bytes | 234.00 KiB/s, done.
+Total 3 (delta 0), reused 0 (delta 0), pack-reused 0
+To ssh://git.jnss.me:2222/your_username/test-repo.git
+ * [new branch]      main -> main
+```
+
+**Step 5.4**: Verify in web interface
+1. Refresh Gitea web UI
+2. Navigate to `test-repo`
+3. Verify `README.md` appears
+
+---
+
+### Test 6: fail2ban Protection
+
+**Purpose**: Verify SSH brute force protection is active
+
+**Step 6.1**: Check fail2ban status
+```bash
+# On the server
+ssh root@arch-vps
+
+# Check gitea-ssh jail
+fail2ban-client status gitea-ssh
+```
+
+**Expected result**:
+```
+Status for the jail: gitea-ssh
+|- Filter
+|  |- Currently failed: 0
+|  |- Total failed:     0
+|  `- File list:        /var/lib/gitea/log/gitea.log
+`- Actions
+   |- Currently banned: 0
+   |- Total banned:     0
+   `- Banned IP list:
+```
+
+**Step 6.2**: Simulate failed authentication (optional)
+```bash
+# From your local machine, try connecting with wrong key multiple times
+ssh -T -p 2222 -i /path/to/wrong/key git@git.jnss.me
+# Repeat this 5+ times quickly
+```
+
+**Step 6.3**: Check if IP was banned
+```bash
+# On the server
+fail2ban-client status gitea-ssh
+```
+
+**Expected result**: Your IP should appear in "Currently banned" list after 5 failed attempts.
+
+**Step 6.4**: Unban yourself (if needed)
+```bash
+# On the server
+fail2ban-client set gitea-ssh unbanip YOUR_IP_ADDRESS
+```
+
+---
+
+### Test 7: Firewall Rule Persistence
+
+**Purpose**: Verify firewall rules survive reboot
+
+**Step 7.1**: Check current rules
+```bash
+# On the server
+ssh root@arch-vps
+nft list ruleset | grep 2222
+```
+
+**Step 7.2**: Reboot server
+```bash
+# On the server
+reboot
+```
+
+**Step 7.3**: After reboot, verify rules are still active
+```bash
+# Wait for server to come back up, then SSH in
+ssh root@arch-vps
+
+# Check nftables rules again
+nft list ruleset | grep 2222
+
+# Verify Gitea SSH still accessible from outside
+exit
+
+# From local machine
+ssh -T -p 2222 git@git.jnss.me
+```
+
+**Expected result**: Port 2222 rule persists after reboot, SSH access still works.
+
+---
+
+## Troubleshooting
+
+### Issue: Connection timeout on port 2222
+
+**Symptoms**: `ssh: connect to host git.jnss.me port 2222: Connection timed out`
+
+**Diagnosis**:
+```bash
+# On server
+systemctl status gitea       # Check if Gitea is running
+ss -tlnp | grep 2222         # Check if SSH is listening
+nft list ruleset | grep 2222 # Check firewall rule
+systemctl status nftables    # Check firewall service
+```
+
+**Solutions**:
+1. **Gitea not running**: `systemctl start gitea`
+2. **Firewall rule missing**: Re-run Ansible playbook with Gitea role
+3. **nftables not running**: `systemctl start nftables`
+4. **Rule file missing**: Check `/etc/nftables.d/gitea.nft` exists
+
+---
+
+### Issue: Permission denied (publickey)
+
+**Symptoms**: SSH connection succeeds but authentication fails
+
+**Diagnosis**:
+```bash
+# Verbose SSH connection
+ssh -vvv -T -p 2222 git@git.jnss.me
+```
+
+**Solutions**:
+1. **SSH key not added to Gitea**: Add your public key in Gitea web UI
+2. **Wrong SSH key used**: Specify correct key: `ssh -i ~/.ssh/id_ed25519 -T -p 2222 git@git.jnss.me`
+3. **Key permissions wrong**: `chmod 600 ~/.ssh/id_ed25519`
+
+---
+
+### Issue: fail2ban not protecting Gitea
+
+**Symptoms**: `fail2ban-client status gitea-ssh` shows jail doesn't exist
+
+**Diagnosis**:
+```bash
+# Check if filter exists
+ls -la /etc/fail2ban/filter.d/gitea-ssh.conf
+
+# Check if jail is configured
+grep -A 10 "gitea-ssh" /etc/fail2ban/jail.local
+
+# Check fail2ban logs
+journalctl -u fail2ban | grep gitea
+```
+
+**Solutions**:
+1. **Jail not configured**: Re-run Ansible playbook with Gitea role
+2. **fail2ban not running**: `systemctl start fail2ban`
+3. **Log file not found**: Check Gitea is logging to `/var/lib/gitea/log/gitea.log`
+
+---
+
+### Issue: Git clone works but push fails
+
+**Symptoms**: Can clone but `git push` gives permission error
+
+**Diagnosis**:
+- Check repository permissions in Gitea web UI
+- Verify you have write access to the repository
+
+**Solutions**:
+1. **Not repository owner**: Ask owner to give you write access
+2. **Repository is archived**: Unarchive in settings
+3. **Branch protected**: Check branch protection rules
+
+---
+
+## Verification Checklist
+
+Use this checklist to verify your Gitea deployment:
+
+- [ ] Gitea web interface accessible at `https://git.jnss.me`
+- [ ] Port 2222 accessible from external network (`nc -zv git.jnss.me 2222`)
+- [ ] SSH connection succeeds (`ssh -T -p 2222 git@git.jnss.me`)
+- [ ] SSH key added to Gitea account
+- [ ] SSH authentication works (shows username in response)
+- [ ] Can clone repository via SSH
+- [ ] Can push commits to repository
+- [ ] nftables rule for port 2222 exists and is active
+- [ ] fail2ban jail `gitea-ssh` is running
+- [ ] Gitea service auto-starts on boot
+- [ ] nftables rules persist after reboot
+
+---
+
+## Post-Deployment Configuration
+
+### Disable Public Registration (Recommended)
+
+If you don't want anyone to create accounts:
+
+1. Edit `host_vars/arch-vps/main.yml` or `group_vars/production/main.yml`
+2. Add:
+   ```yaml
+   gitea_disable_registration: true
+   ```
+3. Re-run playbook:
+   ```bash
+   ansible-playbook -i inventory/hosts.yml rick-infra.yml --tags gitea --limit arch-vps
+   ```
+
+### Configure Email (Optional)
+
+For password resets and notifications, configure SMTP:
+
+1. Edit Gitea configuration directly:
+   ```bash
+   ssh root@arch-vps
+   nano /etc/gitea/app.ini
+   ```
+
+2. Add mailer section:
+   ```ini
+   [mailer]
+   ENABLED = true
+   FROM = gitea@jnss.me
+   PROTOCOL = smtps
+   SMTP_ADDR = smtp.example.com
+   SMTP_PORT = 465
+   USER = gitea@jnss.me
+   PASSWD = your_smtp_password
+   ```
+
+3. Restart Gitea:
+   ```bash
+   systemctl restart gitea
+   ```
+
+### Enable Actions/CI (Optional)
+
+Gitea Actions provides GitHub Actions-compatible CI/CD:
+
+1. Edit `roles/gitea/templates/app.ini.j2`
+2. Add Actions section
+3. Re-run playbook
+
+---
+
+## nftables Architecture
+
+### Modular Firewall Design
+
+Rick-infra uses a modular nftables architecture that allows services to self-manage their firewall rules:
+
+**Load Order:**
+1. **Base rules** (`/etc/nftables.conf`) - Infrastructure essentials (SSH, HTTP, HTTPS, ICMP)
+2. **Service rules** (`/etc/nftables.d/[0-8]*.nft`) - Service-specific ports (e.g., `50-gitea.nft`)
+3. **Drop rule** (`/etc/nftables.d/99-drop.nft`) - Final catch-all drop
+
+**Key Files:**
+- `/etc/nftables.conf` - Base infrastructure firewall rules
+- `/etc/nftables-load.conf` - Loader script that orchestrates rule loading
+- `/etc/nftables.d/50-gitea.nft` - Gitea SSH port (2222) rule
+- `/etc/nftables.d/99-drop.nft` - Final drop rule (loaded last)
+
+**How It Works:**
+```
+┌─────────────────────────────────────────────────┐
+│ /etc/nftables-load.conf                         │
+│                                                 │
+│  1. include "/etc/nftables.conf"               │
+│     └─> Allow: SSH(22), HTTP(80), HTTPS(443)  │
+│                                                 │
+│  2. include "/etc/nftables.d/[0-8]*.nft"      │
+│     └─> 50-gitea.nft: Allow SSH(2222)         │
+│                                                 │
+│  3. include "/etc/nftables.d/99-drop.nft"     │
+│     └─> Drop all other traffic                │
+└─────────────────────────────────────────────────┘
+```
+
+This ensures service rules are evaluated **before** the drop rule, allowing each service role to be self-contained.
+
+## Security Best Practices
+
+1. **Use strong database password**: Ensure `vault_gitea_db_password` is strong
+2. **Enable 2FA**: Enable two-factor authentication in Gitea settings
+3. **Monitor fail2ban**: Regularly check banned IPs: `fail2ban-client status gitea-ssh`
+4. **Keep updated**: Run security playbook regularly for system updates
+5. **Review SSH keys**: Periodically audit SSH keys in Gitea user accounts
+6. **Backup repositories**: Regular backups of `/var/lib/gitea/repositories`
+7. **Monitor logs**: Check Gitea logs for suspicious activity: `journalctl -u gitea`
+
+---
+
+## Quick Reference Commands
+
+```bash
+# Service management
+systemctl status gitea
+systemctl restart gitea
+journalctl -u gitea -f
+
+# Firewall
+nft list ruleset | grep 2222
+systemctl restart nftables
+cat /etc/nftables.d/50-gitea.nft
+
+# fail2ban
+fail2ban-client status gitea-ssh
+fail2ban-client get gitea-ssh banned
+fail2ban-client set gitea-ssh unbanip IP_ADDRESS
+
+# Network
+ss -tlnp | grep 2222
+nc -zv git.jnss.me 2222
+
+# SSH testing
+ssh -T -p 2222 git@git.jnss.me
+ssh -vvv -T -p 2222 git@git.jnss.me  # Verbose mode
+
+# Git operations
+git clone ssh://git@git.jnss.me:2222/user/repo.git
+git remote add origin ssh://git@git.jnss.me:2222/user/repo.git
+```
+
+---
+
+**Rick-Infra Gitea Deployment Guide**  
+Self-contained Git service with automatic firewall and security management.
--- a/docs/security-hardening.md
+++ b/docs/security-hardening.md
@@ -88,31 +88,42 @@ ssh root@your-vps "journalctl -u sshd | grep -i 'failed\|invalid'"

 ### Firewall Configuration

-#### nftables Firewall Rules
+#### Modular nftables Architecture

+Rick-infra uses a **modular firewall architecture** that enables self-contained service roles:
+
+**Structure:**
+```
+/etc/nftables.conf              Base infrastructure rules (SSH, HTTP, HTTPS)
+/etc/nftables-load.conf         Orchestration script (loads rules in order)
+/etc/nftables.d/
+  ├── 50-gitea.nft             Service-specific rules (Gitea SSH port 2222)
+  └── 99-drop.nft              Final drop rule (loaded last)
+```
+
+**Load Order:**
+1. Base infrastructure rules (always allowed)
+2. Service-specific rules (00-98 prefix)
+3. Final drop rule (99-drop.nft)
+
+**Example: Current Ruleset**
 ```bash
-# Deployed firewall configuration
 table inet filter {
    chain input {
        type filter hook input priority 0; policy drop;
        
-        # Allow loopback traffic
-        iifname "lo" accept
-        
-        # Allow established connections
+        # Base infrastructure rules
+        iif "lo" accept
        ct state established,related accept
-        
-        # Allow SSH (rate limited)
-        tcp dport 22 ct state new limit rate 5/minute accept
-        
-        # Allow HTTP/HTTPS
-        tcp dport {80, 443} accept
-        
-        # Allow ICMP (rate limited)
+        tcp dport 22 ct state new accept
+        tcp dport {80, 443} ct state new accept
        icmp type echo-request limit rate 1/second accept
        
-        # Log dropped packets
-        log prefix "DROPPED: " drop
+        # Service-specific rules (loaded from /etc/nftables.d/)
+        tcp dport 2222 ct state new accept comment "Gitea SSH (Port 2222)"
+        
+        # Final drop rule (99-drop.nft)
+        counter drop comment "Drop all other traffic"
    }
    
    chain forward {
@@ -129,15 +140,38 @@ table inet filter {

 ```bash
 # Check firewall status
-ssh root@your-vps "nft list ruleset"
+nft list ruleset

-# Monitor dropped connections
-ssh root@your-vps "journalctl -k | grep DROPPED"
+# View service-specific rules
+ls -la /etc/nftables.d/
+cat /etc/nftables.d/50-gitea.nft

-# Temporary rule addition (emergency access)
-ssh root@your-vps "nft add rule inet filter input tcp dport 8080 accept"
+# Reload firewall (after manual changes)
+systemctl restart nftables
+
+# Test configuration syntax
+nft -c -f /etc/nftables-load.conf
+
+# Add service rule (example for future services)
+# Create /etc/nftables.d/60-newservice.nft with your rules
+echo 'add rule inet filter input tcp dport 8080 accept comment "New Service"' > /etc/nftables.d/60-newservice.nft
+systemctl restart nftables
 ```

+#### Adding New Service Ports
+
+When deploying new services that need firewall access:
+
+1. Create rule file: `/etc/nftables.d/XX-servicename.nft` (XX = 00-98)
+2. Add rule: `add rule inet filter input tcp dport PORT accept comment "Service Name"`
+3. Restart nftables: `systemctl restart nftables`
+
+**Naming Convention:**
+- `00-19`: Infrastructure services
+- `20-79`: Application services
+- `80-98`: Custom/temporary rules
+- `99`: Drop rule (reserved)
+
 ### Intrusion Detection (fail2ban)

 #### fail2ban Configuration