rickvanderzwet.nl

Personal notes on technology, sustainability and DIY projects

SSH concentrator with port knocking behind NAT

12-10-2026 · author: rick · freebsd, security

Introduction

I run a backup machine at a remote location: stuck behind NAT, no control over the upstream firewall. What works for me is a reverse SSH tunnel to a machine I do control, with port knocking in front of the tunnel endpoint. An earlier attempt at knocking here had been running for years without doing anything at all, so the first job with the new setup was to test it end to end.

Note

hostnames, users, addresses and the knock sequence in this post are placeholders - substitute your own.

Architecture

The remote machine (remote, FreeBSD, somewhere behind NAT) keeps a reverse tunnel open to the production router:

concentrator ws workstation conc concentrator sshd :1022 + knockd + pf ws->conc ssh -J, port from 'tunnels' remote remote FreeBSD, behind NAT ws->remote session via the concentrator conc->remote sshd assigns a dynamic port, listed by 'tunnels' remote->conc knock v4+v6, then: ssh -T -R 0:127.0.0.1:22

By using -R 0: the client requests a dynamic port: no port-busy headaches when several machines connect around the same time. Each tunnel key gets a forced command in authorized_keys (command="/usr/local/sbin/tunneld <machinename>",restrict,port-forwarding), which tags every session with a machine label - and, thanks to restrict,port-forwarding, can do nothing else.

From there, access to the machine is a simple jump:

workstation$ ssh -J admin@gw.example.net -p <port> admin@localhost

The port is listed in the tunnels output on the router.

Accessing the machine, without thinking about ports

The dynamic reverse port changes on every (re)connect, so the jump above needs you to look up the port first. That does not survive a half-awake sysadmin at 3 AM, so the workstation carries a small helper script and an ssh alias that discovers the port per connection:

# ~/.ssh/config
Host remote
    User admin
    HostName localhost
    ProxyCommand /usr/local/bin/remote-proxy
    HostKeyAlias remote-tunnel
    StrictHostKeyChecking accept-new
    ServerAliveInterval 30
    ServerAliveCountMax 3
#!/bin/sh
# ProxyCommand: find the current reverse port on the concentrator and
# connect through it (the port changes on every tunnel (re)connect).

PORT=$(ssh -o BatchMode=yes -n gw.example.net /usr/local/sbin/tunnels 2>/dev/null | awk '/^remote[[:space:]]/ {print $3}')
if [ -z "$PORT" ]; then
    echo "no ACTIVE tunnel found on the concentrator" >&2
    exit 1
fi
exec ssh -o BatchMode=yes -W 127.0.0.1:"$PORT" gw.example.net

Three details worth their own paragraph:

Port knocking, two lessons learned

Port 1022 is only reachable from a pf table that stays valid for 30 minutes after a knock: a random sequence of 5 ports (30000-64000), over both IPv4 and IPv6, kept identical on the router and all clients.

Lesson 1: verify that your security actually works

The port knocking turned out to have been dead for years: knockd ran its commands through ipfw, but ipfw was not loaded on the router. Nothing worked, and nobody noticed - nothing depended on it anymore, which is precisely why it could rot unnoticed. Only when we actually triggered a knock and checked its effect did this come to light. A running daemon is not proof that the protection works.

Lesson 2: filtered knock ports trip the sequence

knockd discards an attempt as soon as a packet fails to hit the next expected port. A TCP stack that repeats its SYN three times against a FILTERED port resets the entire sequence. Two ways out, each with a trade-off:

The concentrator deliberately picks the silent variant; on a Linux/WSL2 client without raw-socket privileges and without IPv6 the knock is implemented with nc -z -w 0.3 connects, pinned to the IPv4 address.

Why not a VPN, or Tailscale?

Fair question, because NAT traversal is exactly what WireGuard, Tailscale and ZeroTier are good at. The honest comparison:

The trade-offs I accepted: SSH-in-SSH overhead on a single TCP stream (fine for management traffic, wrong for bulk transfers), and port knocking is by nature security-through-obscurity. The knock is an extra layer, not the gatekeeper - the real gate is the per-machine key with its forced command.

The files, sanitized

The complete set of scripts. Values in angle brackets and addresses are generic: the real knock sequence, hostnames and users differ. Comments translated to English.

The concentrator (router)

The per-machine key in authorized_keys - one line per machine, the forced command carries the machine name:

command="/usr/local/sbin/tunneld remote",restrict,port-forwarding ssh-ed25519 AAAA... remote-tunnel

knockd.conf on the router (and, byte-identical, on every client - the sequence lives in the file, clients parse it out). The sequence below is an invented placeholder, not the production one:

[options]
 logfile = /var/log/knockd.log
 interface = <wan-interface>

[openSSH]
 sequence    = 45678,53147,58264,49201,61983
 seq_timeout = 60
 command     = /usr/local/sbin/knock-ssh-open %IP%
 command_6   = /usr/local/sbin/knock-ssh-open %IP%
 tcpflags    = syn

[closeSSH]
 sequence    = 61983,49201,58264,53147,45678
 seq_timeout = 60
 command     = /usr/local/sbin/knock-ssh-close %IP%
 command_6   = /usr/local/sbin/knock-ssh-close %IP%
 tcpflags    = syn

The knock commands are thin pfctl wrappers around the sshknock table (the table is persist, so it survives knockd restarts):

#!/bin/sh
# knock-ssh-open: open SSH (port 1022) from $1 for 30 minutes.
IP="$1"
[ -z "$IP" ] && exit 1

/sbin/pfctl -t sshknock -T add "$IP" >/dev/null
/usr/bin/logger -t knockd "SSH access opened for $IP (30 minutes, then automatic)"

# Close automatically after 30 minutes; the timer detaches from knockd
# so the entry survives a knockd restart.
nohup sh -c "sleep 1800; /sbin/pfctl -t sshknock -T delete '$IP' \
    >/dev/null 2>&1; /usr/bin/logger -t knockd \
    'SSH access for $IP closed automatically after 30 min'" \
    >/dev/null 2>&1 &
#!/bin/sh
# knock-ssh-close: close SSH access from $1 immediately; also stops the
# running auto-close timer.
IP="$1"
[ -z "$IP" ] && exit 1

pkill -f "knock.*$IP" 2>/dev/null
pkill -f "sleep 1800.*$IP" 2>/dev/null
/sbin/pfctl -t sshknock -T delete "$IP" >/dev/null 2>&1

tunneld, the forced command, tags every tunnel session with its machine name for as long as the session lives:

#!/bin/sh
# tunneld <machinename> - forced command per tunnel key.
# Registers the machine name under the sshd session PID and keeps
# running as long as the tunnel lives; sshd terminates it when the
# connection drops.

name=${1:-unknown}
dir=/var/run/tunneld
mkdir -p "$dir"

# find the sshd session PID (walk up from our own process)
p=$$
while :; do
    p=$(ps -o ppid= -p "$p" 2>/dev/null | tr -d " ")
    [ -z "$p" ] && break
    comm=$(ps -o comm= -p "$p" 2>/dev/null)
    case "$comm" in sshd*) break;; esac
    [ "$p" = "1" ] && { p=""; break; }
done
tag="$dir/${p:-$$}"
printf "%s\n" "$name" > "$tag"

trap 'rm -f "$tag"' EXIT
trap 'exit 0' TERM INT HUP
while :; do sleep 60; done

And tunnels, the session overview (machine, PID, dynamic port, client address, uptime, active/stale):

#!/bin/sh
# tunnels - overview of the reverse SSH sessions on this concentrator.
# A session with a listener but without a client connection is STALE:
# its forward port no longer responds.

USER=gate
TAGDIR=/var/run/tunneld
tmp=$(mktemp -t tunnels)
trap 'rm -f "$tmp"' 0 1 2 15

sockstat | awk -v u="$USER" '$1==u && $5 ~ /^tcp/' > "$tmp"

pids=$(awk '{print $3}' "$tmp" | sort -un)
total=0; active=0; stale=0

printf "%-10s %-7s %-6s %-50s %-11s %s\n" MACHINE PID PORT CLIENT UPTIME STATUS
for pid in $pids; do
    machine=$(cat "$TAGDIR/$pid" 2>/dev/null || echo "?")
    ports=$(awk -v p="$pid" '$3==p && $7=="*:*"{n=$6; sub(/.*:/,"",n); print n}' "$tmp" | sort -u | paste -s -d, -)
    client=$(awk -v p="$pid" '$3==p && $7!="*:*"{print $7; exit}' "$tmp")
    age=$(ps -o etime= -p "$pid" 2>/dev/null | tr -d " ")
    total=$((total+1))
    if [ -n "$client" ]; then
        status="ACTIVE"; active=$((active+1))
    else
        client="-"; status="STALE"; stale=$((stale+1))
    fi
    [ -z "$ports" ] && ports="-"
    printf "%-10s %-7s %-6s %-50s %-11s %s\n" "$machine" "$pid" "$ports" "$client" "${age:-?}" "$status"
done

echo "---"
echo "Total: $total session(s) | $active active | $stale stale"

The client (remote machine)

The tunnel loop, sshtunnel.sh. Knock over both families, connect, wait for the ssh to die, sleep, repeat. Since DNS at remote sites can be flaky, a failed lookup falls back to the concentrator's fixed WAN addresses - with HostKeyAlias, so the known_hosts check keeps matching the hostname:

#!/bin/sh
# Reverse SSH tunnel (concentrator): remote -> gateway
# The reverse port is assigned dynamically by sshd (-R 0:...), so
# there is never a 'port busy'; the actual port shows in 'tunnels' on
# the concentrator. The machine name is registered by the forced
# command (tunneld).

CONF=/usr/local/etc/knockd.conf
HOST=gw.example.net
FALLBACK_V4=203.0.113.10
FALLBACK_V6=2001:db8:7ae:154::1
SSHPORT=1022
KEY=/home/admin/.ssh/gateway-tunnel
LOG=/var/log/sshtunnel.log

# extract the knock sequence from [openSSH]; commas to spaces for knock(1)
KSEQ=$(awk '/^\[openSSH\]/{f=1;next} /^\[/{f=0} f && $1=="sequence"{sub(/^[^=]*=[[:space:]]*/,"");print;exit}' "$CONF" | tr "," " ")

SSH_PID=""
cleanup() {
    [ -n "$SSH_PID" ] && kill "$SSH_PID" 2>/dev/null
    exit 0
}
trap cleanup TERM INT

while :; do
    [ -n "$KSEQ" ] || break

    # DNS can be unreliable at remote sites: fall back to fixed WAN
    # addresses of the concentrator.
    if getent hosts "$HOST" > /dev/null 2>&1; then
        TARGETS="$HOST"
        /usr/local/bin/knock -4 -d 200 "$HOST" $KSEQ 2>/dev/null
        /usr/local/bin/knock -6 -d 200 "$HOST" $KSEQ 2>/dev/null
    else
        echo "$(date '+%Y-%m-%d %H:%M:%S') DNS lookup $HOST failed; fallback to fixed WAN addresses" >> "$LOG"
        TARGETS="$FALLBACK_V4 $FALLBACK_V6"
        /usr/local/bin/knock -4 -d 200 "$FALLBACK_V4" $KSEQ 2>/dev/null
        /usr/local/bin/knock -6 -d 200 "$FALLBACK_V6" $KSEQ 2>/dev/null
    fi

    for TARGET in $TARGETS; do
        ALIAS="-o HostKeyAlias=${HOST}"
        [ "$TARGET" = "$HOST" ] && ALIAS=""
        /usr/bin/ssh -T -p "$SSHPORT" -i "$KEY" \
            -o BatchMode=yes \
            -o StrictHostKeyChecking=yes \
            $ALIAS \
            -o ConnectTimeout=15 \
            -o ServerAliveInterval=30 -o ServerAliveCountMax=3 \
            -o ExitOnForwardFailure=yes \
            -R 0:127.0.0.1:22 \
            "gate@${TARGET}" 2>>"$LOG" &
        SSH_PID=$!
        wait "$SSH_PID"
        SSH_PID=""
    done
    sleep 30 &
    wait $!
done

The rc.d glue that keeps it running (and, in stop_post, also kills the inner ssh - note the character-class trick that keeps pkill from matching its own command line):

#!/bin/sh
# PROVIDE: sshtunnel
# REQUIRE: NETWORKING DAEMON
# KEYWORDS: nojail

. /etc/rc.subr

name="sshtunnel"
rcvar="sshtunnel_enable"
command="/usr/sbin/daemon"
command_args="-f -P /var/run/sshtunnel.pid /usr/local/sbin/sshtunnel.sh"
pidfile="/var/run/sshtunnel.pid"

stop_post() {
    pkill -f "/usr/local/sbin/sshtunnel.s[h]" 2>/dev/null
    pkill -f "gateway-tunne[l]" 2>/dev/null
}

load_rc_config "$name"
run_rc_command "$1"

Restarting it from the workstation needs care: a plain service sshtunnel restart through the tunnel dies halfway, because the stop step kills the transport that carries its own start step. Two safe ways - a detached restart:

workstation$ ssh remote 'nohup sh -c "service sshtunnel restart" >/dev/null 2>&1 &'

or, if only a reconnect is needed, killing just the inner ssh and letting the loop reconnect on its own:

workstation$ ssh remote 'pkill -f "gateway-tunne[l]"'

Wrap-up

Everything you need to build your own is above. The main take-away: always measure whether the knock actually arrives and the sequence completes - on the client as well as on the router.

Download as: plain text