---
layout: post
title: SSH concentrator with port knocking behind NAT
date: 2026-10-12 21:46
author: rick
categories: freebsd, security
draft: false
toc: true
---

## Introduction

I run a backup machine at a remote location: stuck behind NAT, no
control over the upstream firewall. What works for me is a reverse SSH
tunnel to a machine I do control, with port knocking in front of the
tunnel endpoint. An earlier attempt at knocking here had been running
for years without doing anything at all, so the first job with the new
setup was to test it end to end.

Note: hostnames, users, addresses and the knock sequence in this post
are placeholders - substitute your own.

## Architecture

The remote machine (`remote`, FreeBSD, somewhere behind NAT) keeps a
reverse tunnel open to the production router:

```dot
digraph concentrator {
  rankdir=LR;
  node [shape=box, style=rounded];
  ws [label="workstation"];
  conc [label="concentrator\nsshd :1022\n+ knockd + pf"];
  remote [label="remote\nFreeBSD, behind NAT"];

  remote -> conc [label="knock v4+v6, then:\nssh -T -R 0:127.0.0.1:22"];
  conc -> remote [label="sshd assigns a dynamic port,\nlisted by 'tunnels'"];
  ws -> conc [label="ssh -J, port from 'tunnels'"];
  ws -> remote [label="session via the concentrator", style=dashed];
}
```

By using `-R 0:` the client requests a dynamic port: no port-busy
headaches when several machines connect around the same time. Each tunnel
key gets a forced command in `authorized_keys`
(`command="/usr/local/sbin/tunneld <machinename>",restrict,port-forwarding`),
which tags every session with a machine label - and, thanks to
`restrict,port-forwarding`, can do nothing else.

From there, access to the machine is a simple jump:

```sh
workstation$ ssh -J admin@gw.example.net -p <port> admin@localhost
```

The port is listed in the `tunnels` output on the router.

## Accessing the machine, without thinking about ports

The dynamic reverse port changes on every (re)connect, so the jump above
needs you to look up the port first. That does not survive a half-awake
sysadmin at 3 AM, so the workstation carries a small helper script and an
ssh alias that discovers the port per connection:

```sh
# ~/.ssh/config
Host remote
    User admin
    HostName localhost
    ProxyCommand /usr/local/bin/remote-proxy
    HostKeyAlias remote-tunnel
    StrictHostKeyChecking accept-new
    ServerAliveInterval 30
    ServerAliveCountMax 3
```

```sh
#!/bin/sh
# ProxyCommand: find the current reverse port on the concentrator and
# connect through it (the port changes on every tunnel (re)connect).

PORT=$(ssh -o BatchMode=yes -n gw.example.net /usr/local/sbin/tunnels 2>/dev/null | awk '/^remote[[:space:]]/ {print $3}')
if [ -z "$PORT" ]; then
    echo "no ACTIVE tunnel found on the concentrator" >&2
    exit 1
fi
exec ssh -o BatchMode=yes -W 127.0.0.1:"$PORT" gw.example.net
```

Three details worth their own paragraph:

- `HostKeyAlias` pins the remote machine's host key under one stable name.
  Without it, every new dynamic port would present itself as a new
  `[localhost]:port` host, and the first connection to each port would ask
  for confirmation.
- The helper ssh inside the ProxyCommand needs `-n`: without it, it eats
  the outer ssh's transport stdin and the connection dies with
  "Bad packet length". That took an evening to find; the debugging
  lessons from that evening are worth a post of
  [their own](/blog/2026/10/16/ai-agent-measurement-hygiene/).
- `ServerAliveInterval` makes a dead tunnel (rather than a stuck one)
  end your session within about ninety seconds.

## Port knocking, two lessons learned

Port 1022 is only reachable from a pf table that stays valid for 30
minutes after a knock: a random sequence of 5 ports (30000-64000), over
both IPv4 and IPv6, kept identical on the router and all clients.

### Lesson 1: verify that your security actually works

The port knocking turned out to have been dead for years: knockd ran its
commands through ipfw, but ipfw was not loaded on the router. Nothing
worked, and nobody noticed - nothing depended on it anymore, which is
precisely why it could rot unnoticed. Only when we actually triggered a
knock and checked its effect did this come to light. A running daemon is
not proof that the protection works.

### Lesson 2: filtered knock ports trip the sequence

knockd discards an attempt as soon as a packet fails to hit the next
expected port. A TCP stack that repeats its SYN three times against a
FILTERED port resets the entire sequence. Two ways out, each with a
trade-off:

- let the knock ports pass in pf, so the kernel answers with an RST
  immediately and the client does not retry
- or drop silently (scanners see 'filtered' everywhere, no information
  leak) and use short connect timeouts on the client side, so no
  retransmits occur in the first place

The concentrator deliberately picks the silent variant; on a
Linux/WSL2 client without raw-socket privileges and without IPv6 the
knock is implemented with `nc -z -w 0.3` connects, pinned to the IPv4
address.

## Why not a VPN, or Tailscale?

Fair question, because NAT traversal is exactly what WireGuard,
Tailscale and ZeroTier are good at. The honest comparison:

- **Plain VPN (WireGuard/IPsec)** solves site-to-site links with stable
  endpoints. Here the remote end cannot accept inbound traffic at all
  (NAT), and the home end sits on a consumer PPPoE link with a dynamic
  address. Both are solvable - the remote end can dial out - but at the
  cost of a second always-on encrypted network next to sshd, its own key
  and endpoint management, and a whole subsystem to harden and update on
  both ends. sshd and pf were already there, already hardened, already
  in the logs.
- **Tailscale/ZeroTier** handle NAT traversal elegantly, including
  relays for impossible cases - but they put a third-party coordination
  service and account between me and my production router. Access to my
  own infrastructure would depend on somebody else's availability,
  security posture and goodwill. Self-hosted alternatives (Headscale)
  exist, but running extra infrastructure is precisely what this setup
  tries to avoid: there is nothing to operate beyond sshd.
- **Least privilege**: the forced-command, restricted tunnel keys can
  do exactly one thing - forward a session to that machine's own loopback
  sshd. A VPN, by design, makes entire subnets one route away. And every
  tunnel session here is a normal, labelled sshd session in the standard
  auth logs - auditability for free.

The trade-offs I accepted: SSH-in-SSH overhead on a single TCP stream
(fine for management traffic, wrong for bulk transfers), and port
knocking is by nature security-through-obscurity. The knock is an extra
layer, not the gatekeeper - the real gate is the per-machine key with
its forced command.

## The files, sanitized

The complete set of scripts. Values in *angle
brackets* and addresses are generic: the real knock sequence, hostnames
and users differ. Comments translated to English.

### The concentrator (router)

The per-machine key in `authorized_keys` - one line per machine, the
forced command carries the machine name:

```conf
command="/usr/local/sbin/tunneld remote",restrict,port-forwarding ssh-ed25519 AAAA... remote-tunnel
```

`knockd.conf` on the router (and, byte-identical, on every client - the
sequence lives in the file, clients parse it out). The sequence below is
an invented placeholder, not the production one:

```conf
[options]
 logfile = /var/log/knockd.log
 interface = <wan-interface>

[openSSH]
 sequence    = 45678,53147,58264,49201,61983
 seq_timeout = 60
 command     = /usr/local/sbin/knock-ssh-open %IP%
 command_6   = /usr/local/sbin/knock-ssh-open %IP%
 tcpflags    = syn

[closeSSH]
 sequence    = 61983,49201,58264,53147,45678
 seq_timeout = 60
 command     = /usr/local/sbin/knock-ssh-close %IP%
 command_6   = /usr/local/sbin/knock-ssh-close %IP%
 tcpflags    = syn
```

The knock commands are thin pfctl wrappers around the `sshknock` table
(the table is `persist`, so it survives knockd restarts):

```sh
#!/bin/sh
# knock-ssh-open: open SSH (port 1022) from $1 for 30 minutes.
IP="$1"
[ -z "$IP" ] && exit 1

/sbin/pfctl -t sshknock -T add "$IP" >/dev/null
/usr/bin/logger -t knockd "SSH access opened for $IP (30 minutes, then automatic)"

# Close automatically after 30 minutes; the timer detaches from knockd
# so the entry survives a knockd restart.
nohup sh -c "sleep 1800; /sbin/pfctl -t sshknock -T delete '$IP' \
    >/dev/null 2>&1; /usr/bin/logger -t knockd \
    'SSH access for $IP closed automatically after 30 min'" \
    >/dev/null 2>&1 &
```

```sh
#!/bin/sh
# knock-ssh-close: close SSH access from $1 immediately; also stops the
# running auto-close timer.
IP="$1"
[ -z "$IP" ] && exit 1

pkill -f "knock.*$IP" 2>/dev/null
pkill -f "sleep 1800.*$IP" 2>/dev/null
/sbin/pfctl -t sshknock -T delete "$IP" >/dev/null 2>&1
```

`tunneld`, the forced command, tags every tunnel session with its
machine name for as long as the session lives:

```sh
#!/bin/sh
# tunneld <machinename> - forced command per tunnel key.
# Registers the machine name under the sshd session PID and keeps
# running as long as the tunnel lives; sshd terminates it when the
# connection drops.

name=${1:-unknown}
dir=/var/run/tunneld
mkdir -p "$dir"

# find the sshd session PID (walk up from our own process)
p=$$
while :; do
    p=$(ps -o ppid= -p "$p" 2>/dev/null | tr -d " ")
    [ -z "$p" ] && break
    comm=$(ps -o comm= -p "$p" 2>/dev/null)
    case "$comm" in sshd*) break;; esac
    [ "$p" = "1" ] && { p=""; break; }
done
tag="$dir/${p:-$$}"
printf "%s\n" "$name" > "$tag"

trap 'rm -f "$tag"' EXIT
trap 'exit 0' TERM INT HUP
while :; do sleep 60; done
```

And `tunnels`, the session overview (machine, PID, dynamic port,
client address, uptime, active/stale):

```sh
#!/bin/sh
# tunnels - overview of the reverse SSH sessions on this concentrator.
# A session with a listener but without a client connection is STALE:
# its forward port no longer responds.

USER=gate
TAGDIR=/var/run/tunneld
tmp=$(mktemp -t tunnels)
trap 'rm -f "$tmp"' 0 1 2 15

sockstat | awk -v u="$USER" '$1==u && $5 ~ /^tcp/' > "$tmp"

pids=$(awk '{print $3}' "$tmp" | sort -un)
total=0; active=0; stale=0

printf "%-10s %-7s %-6s %-50s %-11s %s\n" MACHINE PID PORT CLIENT UPTIME STATUS
for pid in $pids; do
    machine=$(cat "$TAGDIR/$pid" 2>/dev/null || echo "?")
    ports=$(awk -v p="$pid" '$3==p && $7=="*:*"{n=$6; sub(/.*:/,"",n); print n}' "$tmp" | sort -u | paste -s -d, -)
    client=$(awk -v p="$pid" '$3==p && $7!="*:*"{print $7; exit}' "$tmp")
    age=$(ps -o etime= -p "$pid" 2>/dev/null | tr -d " ")
    total=$((total+1))
    if [ -n "$client" ]; then
        status="ACTIVE"; active=$((active+1))
    else
        client="-"; status="STALE"; stale=$((stale+1))
    fi
    [ -z "$ports" ] && ports="-"
    printf "%-10s %-7s %-6s %-50s %-11s %s\n" "$machine" "$pid" "$ports" "$client" "${age:-?}" "$status"
done

echo "---"
echo "Total: $total session(s) | $active active | $stale stale"
```

### The client (remote machine)

The tunnel loop, `sshtunnel.sh`. Knock over both families, connect,
wait for the ssh to die, sleep, repeat. Since DNS at remote sites can
be flaky, a failed lookup falls back to the concentrator's fixed WAN
addresses - with `HostKeyAlias`, so the known_hosts check keeps
matching the hostname:

```sh
#!/bin/sh
# Reverse SSH tunnel (concentrator): remote -> gateway
# The reverse port is assigned dynamically by sshd (-R 0:...), so
# there is never a 'port busy'; the actual port shows in 'tunnels' on
# the concentrator. The machine name is registered by the forced
# command (tunneld).

CONF=/usr/local/etc/knockd.conf
HOST=gw.example.net
FALLBACK_V4=203.0.113.10
FALLBACK_V6=2001:db8:7ae:154::1
SSHPORT=1022
KEY=/home/admin/.ssh/gateway-tunnel
LOG=/var/log/sshtunnel.log

# extract the knock sequence from [openSSH]; commas to spaces for knock(1)
KSEQ=$(awk '/^\[openSSH\]/{f=1;next} /^\[/{f=0} f && $1=="sequence"{sub(/^[^=]*=[[:space:]]*/,"");print;exit}' "$CONF" | tr "," " ")

SSH_PID=""
cleanup() {
    [ -n "$SSH_PID" ] && kill "$SSH_PID" 2>/dev/null
    exit 0
}
trap cleanup TERM INT

while :; do
    [ -n "$KSEQ" ] || break

    # DNS can be unreliable at remote sites: fall back to fixed WAN
    # addresses of the concentrator.
    if getent hosts "$HOST" > /dev/null 2>&1; then
        TARGETS="$HOST"
        /usr/local/bin/knock -4 -d 200 "$HOST" $KSEQ 2>/dev/null
        /usr/local/bin/knock -6 -d 200 "$HOST" $KSEQ 2>/dev/null
    else
        echo "$(date '+%Y-%m-%d %H:%M:%S') DNS lookup $HOST failed; fallback to fixed WAN addresses" >> "$LOG"
        TARGETS="$FALLBACK_V4 $FALLBACK_V6"
        /usr/local/bin/knock -4 -d 200 "$FALLBACK_V4" $KSEQ 2>/dev/null
        /usr/local/bin/knock -6 -d 200 "$FALLBACK_V6" $KSEQ 2>/dev/null
    fi

    for TARGET in $TARGETS; do
        ALIAS="-o HostKeyAlias=${HOST}"
        [ "$TARGET" = "$HOST" ] && ALIAS=""
        /usr/bin/ssh -T -p "$SSHPORT" -i "$KEY" \
            -o BatchMode=yes \
            -o StrictHostKeyChecking=yes \
            $ALIAS \
            -o ConnectTimeout=15 \
            -o ServerAliveInterval=30 -o ServerAliveCountMax=3 \
            -o ExitOnForwardFailure=yes \
            -R 0:127.0.0.1:22 \
            "gate@${TARGET}" 2>>"$LOG" &
        SSH_PID=$!
        wait "$SSH_PID"
        SSH_PID=""
    done
    sleep 30 &
    wait $!
done
```

The rc.d glue that keeps it running (and, in `stop_post`, also kills
the inner ssh - note the character-class trick that keeps pkill from
matching its own command line):

```sh
#!/bin/sh
# PROVIDE: sshtunnel
# REQUIRE: NETWORKING DAEMON
# KEYWORDS: nojail

. /etc/rc.subr

name="sshtunnel"
rcvar="sshtunnel_enable"
command="/usr/sbin/daemon"
command_args="-f -P /var/run/sshtunnel.pid /usr/local/sbin/sshtunnel.sh"
pidfile="/var/run/sshtunnel.pid"

stop_post() {
    pkill -f "/usr/local/sbin/sshtunnel.s[h]" 2>/dev/null
    pkill -f "gateway-tunne[l]" 2>/dev/null
}

load_rc_config "$name"
run_rc_command "$1"
```

Restarting it from the workstation needs care: a plain
`service sshtunnel restart` *through the tunnel* dies halfway, because
the stop step kills the transport that carries its own start step.
Two safe ways - a detached restart:

```sh
workstation$ ssh remote 'nohup sh -c "service sshtunnel restart" >/dev/null 2>&1 &'
```

or, if only a reconnect is needed, killing just the inner ssh and
letting the loop reconnect on its own:

```sh
workstation$ ssh remote 'pkill -f "gateway-tunne[l]"'
```

## Wrap-up

Everything you need to build your own is above. The main take-away:
always measure whether the knock actually arrives and the sequence
completes - on the client as well as on the router.