At around 4:00AM Thursday something happened to the configuration and now I'm seeing an even weirder issue. I cranked up the amount of diffs to keep in config history, but it's a bit late for that. The traffic is flowing from our remote host properly, but there are no rules anywhere for the port forward.
Nothing shows for pfctl -sn | grep 9996, pfctl -sr | grep 9996, or grep 9996 /cf/conf/config.xml, but here's the tcpdumps(w.x.y.z being remote ip and a.b.c.d being our WAN ip):
[2.2.2-RELEASE][admin@pfSense.localdomain]/root: tcpdump -i bge0_vlan3 dst port 9996
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on bge0_vlan3, link-type EN10MB (Ethernet), capture size 65535 bytes
14:06:58.109928 IP w.x.y.z.37625 > a.b.c.d.9996: UDP, length 1464
14:06:58.110272 IP w.x.y.z.37625 > a.b.c.d.9996: UDP, length 1464
14:06:58.110768 IP w.x.y.z.37625 > a.b.c.d.9996: UDP, length 1464
14:06:58.110951 IP w.x.y.z.37625 > a.b.c.d.9996: UDP, length 1464
14:06:58.111289 IP w.x.y.z.37625 > a.b.c.d.9996: UDP, length 1464
14:06:58.111784 IP w.x.y.z.37625 > a.b.c.d.9996: UDP, length 1464
14:06:58.112125 IP w.x.y.z.37625 > a.b.c.d.9996: UDP, length 1464
14:06:58.112284 IP w.x.y.z.37625 > a.b.c.d.9996: UDP, length 1464
14:06:58.571766 IP w.x.y.z.37625 > a.b.c.d.9996: UDP, length 1464
14:06:58.572108 IP w.x.y.z.37625 > a.b.c.d.9996: UDP, length 1464
[2.2.2-RELEASE][admin@pfSense.localdomain]/root: tcpdump -i bge0_vlan4 dst port 9996
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on bge0_vlan4, link-type EN10MB (Ethernet), capture size 65535 bytes
14:07:03.110049 IP w.x.y.z.37625 > 10.0.0.10.9996: UDP, length 1464
14:07:03.110200 IP w.x.y.z.37625 > 10.0.0.10.9996: UDP, length 1464
14:07:03.110541 IP w.x.y.z.37625 > 10.0.0.10.9996: UDP, length 1464
14:07:03.110723 IP w.x.y.z.37625 > 10.0.0.10.9996: UDP, length 1464
14:07:03.111061 IP w.x.y.z.37625 > 10.0.0.10.9996: UDP, length 1464
14:07:03.111402 IP w.x.y.z.37625 > 10.0.0.10.9996: UDP, length 1464
14:07:03.111559 IP w.x.y.z.37625 > 10.0.0.10.9996: UDP, length 1464
14:07:03.111898 IP w.x.y.z.37625 > 10.0.0.10.9996: UDP, length 1464
14:07:03.112237 IP w.x.y.z.37625 > 10.0.0.10.9996: UDP, length 1464
Here are the states for the port:
bge0_vlan3 udp 10.0.0.10:9996 (a.b.c.d:9996) <- w.x.y.z:37625 NO_TRAFFIC:SINGLE
bge0_vlan4 udp w.x.y.z:37625 -> 10.0.0.10:9996 SINGLE:NO_TRAFFIC
It's "working" now, but if the connection drops I don't think it will start back up again.
EDIT: Yeah, resetting states killed it. I re-added the rule with destination set to WAN address instead of any and it's working now. That's probably all it was.