Where did my packet go ? One host can't come through were others can

torbjorn

Hey, everyone.

I am hosting a website that is used for display commercials, so the clients are only contacting our site, and they do so very regularly.
The clients all hit an IP that is CARP'ed between two hosts, and that usually works great.

# pfctl -s state | grep "THAT_IP" | wc -l
   30158

So there are lots of active clients.

However, sometimes I get a new web client going, and it can't reach the web server.
Right now I have one hitting the pfSense firewall, and I can see it using tcpdump


# tcpdump host 80.203.250.74
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on em0, link-type EN10MB (Ethernet), capture size 96 bytes
09:33:14.736148 IP 74.80-203-250.nextgentel.com.52120 > THAT_IP.http: S 1959785677:1959785677(0) win 65535 <mss 1452,nop,nop,sackok="">09:33:16.486928 IP 74.80-203-250.nextgentel.com.52121 > THAT_IP.http: S 3281597869:3281597869(0) win 65535 <mss 1452,nop,nop,sackok="">09:33:17.688801 IP 74.80-203-250.nextgentel.com.52120 > THAT_IP.http: S 1959785677:1959785677(0) win 65535 <mss 1452,nop,nop,sackok="">09:33:19.547423 IP 74.80-203-250.nextgentel.com.52121 > THAT_IP.http: S 3281597869:3281597869(0) win 65535 <mss 1452,nop,nop,sackok="">09:33:23.703288 IP 74.80-203-250.nextgentel.com.52120 > THAT_IP.http: S 1959785677:1959785677(0) win 65535 <mss 1452,nop,nop,sackok="">09:33:25.569497 IP 74.80-203-250.nextgentel.com.52121 > THAT_IP.http: S 3281597869:3281597869(0) win 65535 <mss 1452,nop,nop,sackok="">^C
6 packets captured
141196 packets received by filter
0 packets dropped by kernel</mss></mss></mss></mss></mss></mss>

As one can see, it tries to SYN with the server located at THAT_IP, but the SYN packet never reaches the web server.

Does anyone have any clue as to what is going on here ?
I can't see any funny ICMP stuff coming from that client, and right now I have about a hundred other clients going with no problems.

Any limits in pf I'm hitting ?
Some per-host stuff ?

I would be grateful for any help anyone can offer ..

– Torbjørn / Nextline