When the measurement lies: debugging with an AI agent

Introduction¶

I maintain a small FreeBSD infrastructure: a home router, a backup server on a remote site behind NAT, reverse SSH tunnels to reach it, and port knocking in front of the tunnel endpoint - the setup described in my previous post. Most of the day-to-day work on it I now do together with an AI coding agent: Mistral Vibe 2.26.1, the CLI coding agent of Mistral AI (a European initiative I am happy to support), running its default model configuration. This evening produced a bug hunt with a twist that is worth writing down, because the most interesting failure was not in the network. It was in how an AI agent measures.

The short version of the hunt: after a configuration change, the agent's port-knock over IPv6 no longer worked. The agent ran ten-plus tcpdump experiments, concluded that the packets "never arrive", went through several wrong theories about the ISP, and only stopped when I ran one command myself, interactively, on the console - and the packets showed up instantly. Same interface, same filter, same seconds. The agent's measurement pipeline had been lying to it, four different ways.

If you build or operate AI agents, this is the post for you. The checklist below now lives in the AGENTS.md of the repository that the agent reads at the start of every session - which is exactly how operational experience should flow back into agent behavior.

The hunt, compressed¶

The knocking daemon on the router (knockd) is supposed to open the firewall for a client that hits five ports in the right order. IPv4 worked; the IPv6 variant did not. From the agent's measurements:

• tcpdump on the router "captured nothing" for the knock packets;
• the knocking daemon logged nothing;
• therefore, the agent concluded, inbound IPv6 must be broken somewhere upstream.

Then, at my own terminal:

router# tcpdump -ni ng0 'ip6 and tcp dst port 54321'
21:23:01.218943 IP6 2001:db8:2877:1::1.43336 > 2001:db8:7ae:154::1.54321: Flags [S], seq ...

There they were. Not only did the packets arrive - they passed the exact same pcap filter expression the agent had used. After that single data point, the real cause (a userspace parsing bug in the knocking daemon for IPv6 on loopback-type interfaces, while the kernel-side filter matched fine) was found in an hour.

Every "zero" the agent had measured was an artifact of its own measurement pipeline. Four distinct ways:

Lesson 1: stdio buffering eats evidence¶

The general rule: stdout to a terminal is line-buffered; stdout to a file or a pipe is block-buffered (typically 4-64 KiB). An agent does not run commands on a terminal. It redirects to files, it kills processes with timeouts, and it reads the files afterwards. When a timeout 40 tcpdump ... > /tmp/x.log is killed by SIGTERM, the packet lines and even the closing statistics can still be sitting in the stdio buffer - the file says "nothing", the packet was there.

This is not a tcpdump quirk; it hits any tool. The fixes are per tool:

On Linux, stdbuf -oL is a generic wrapper; FreeBSD base does not have it, so there you use the tool's own flag. And when you run ssh host cmd > file, remember that the buffering of the remote command counts.

The rule that saved us: never trust an empty result unless the output contains the closing footer or statistics that prove the tool actually finished its accounting. A tcpdump log without a "N packets captured" line is not a measurement; it is a buffer.

Lesson 2: pretty-filtering hid the errors¶

The agent read its own capture files with grep -v '^tcpdump:' ... to strip the noise - and thereby also stripped the error lines from its reading. Every subsequent conclusion was built on output it had pre-filtered. Rule: read raw first, filter for presentation second. Errors live exactly in the lines you did not want to see.

Lesson 3: filter terms and negative greps¶

Two smaller but equally effective traps:

• tcpdump's filter keyword is ip6. The agent used inet6, which is not a filter term; depending on invocation the failure is silent, and "zero results" looked like "zero traffic".
• An earlier ifconfig | grep -v '::1' was meant to skip the loopback address, but it removes every address ending in ::1 - including the global address under investigation. The agent "proved" the router had no global IPv6 address by filtering that address out.

Rule: grep for what you want, not against what you do not want, and validate every filter expression against a known-positive case before you build on its output.

Lesson 4: cross the timing, and the interface¶

A capture that returns zero proves nothing unless you also prove that the probe was sent and arrived within the capture window. The ground truth here was the firewall log (pflog): it showed the knock packets arriving in windows where the agent's captures were not even running yet, or had already been killed. Timestamps on both ends, every time.

And: the WAN traffic rides a VLAN subinterface. Filters on the parent NIC see only tagged frames and never match. Measure at the level where the packets actually appear.

The process rules¶

The network bug is fixed. What I want to keep is the process that replaced the fumbling. All of it is now in the repository's AGENTS.md, so the agent starts every future session with it:

• Positive control before believing a negative. When a measurement returns empty, first prove the instrument can capture something (known traffic), then theorize about the network.
• Raw before pretty (lesson 2).
• Explicit approval before every mutation: the agent presents the files it will change, the exact commands, the expected output and the rollback - and waits. This caught several blast-radius issues before they happened.
• Never restart a service over its own transport path. Early in the evening the agent restarted the tunnel service through the tunnel: the stop-step killed the transport that carried the start-step, and the remote machine stayed unreachable until someone on site power cycled it. Expensive lesson, now a written rule.
• Record methodology fixes immediately, not at session close. The moment a measurement artifact is discovered, it goes into AGENTS.md. The next step in the same session already benefits.
• Commit per topic, during the session. Smaller diffs, readable history, no pile of uncommitted work.

The rules, ready to copy¶

The list above in copy-paste form, generalized from my repository's AGENTS.md. Drop it into yours - any agent that reads instructions at session start will benefit on the first run:

# Measurement and process hygiene

- stdout to a file or pipe is block-buffered. Use the tool's unbuffered
  flag (tcpdump -l, grep --line-buffered, sed -u, awk fflush(), python -u)
  and only trust output if the tool's closing footer or statistics is
  actually present. A log without the final stats line is a buffer, not
  a measurement.
- Read raw output first; apply presentation filters second. Error lines
  are exactly the lines you filtered away.
- Grep for what you want, never against what you do not want, and
  validate every filter expression against a known-positive case before
  building conclusions on it.
- A negative result only counts if you can prove the probe was sent and
  arrived within the measurement window. Cross-check timestamps against
  ground truth on the receiving side (firewall logs).
- Measure at the network level where the packets actually appear (VLAN,
  tunnel, sub-interface), not at an arbitrary parent interface.
- Helper commands inside a ProxyCommand need their stdin redirected (-n).
- Never restart a service over its own transport path: the stop step
  kills the transport that carries the start step. Detach the restart,
  or use a second path.
- Before every mutation on a production system: present the files to
  change, the exact commands, the expected output and the rollback; wait
  for explicit approval.
- Record fixes for methodology errors immediately in AGENTS.md, not at
  the end of the session.
- Commit per topic, during the session.

What agent builders can take from this¶

Two things, I think.

First: agent runtimes should make these failure modes impossible or at least loud. A process runner that keeps stdout line-buffered, always reads the final buffer after a process is killed, and warns when an expected footer or statistics line is missing, would have prevented three of the four false negatives outright. Cheaper agents are nice; honester agents are better.

Second: the loop back into the model is text, so publish the text. The lessons above only help future sessions because they live in a file the agent reads. The AGENTS.md convention (operational instructions checked into the repository) is a surprisingly effective channel to feed field experience back into agent behavior - and anything published openly feeds the next generation of models, European or otherwise. That is why this post exists.

And the most valuable data point of the evening was the human one: an operator looking over the agent's shoulder and saying your measurement must be wrong - check the instrument. The best division of labor I have found so far: the agent runs the experiments, the human guards the method.

Wrap-up¶

Final score: one real bug (the knocking daemon's IPv6 parsing on loopback-typed interfaces), four measurement artifacts that hid it, one expensive self-inflicted outage, and a checklist that turns all of it into something the next session - and hopefully the next agent - starts with. The full checklist lives in the repository; consider this post the long-form advertisement for keeping one yourself.