Some http traffic not going out LAN interface (xn0) but it did come in the enc0 interface.

Danb · Mar 13, 2019, 12:15 AM

I have two pfsense machines (aws instances), one is a 2.4.3 and one is a 2.4.4 version. I have an IPSec tunnel established between them but the traffic is not completely flowing correct between the two. I get the 3 way handshake and the HTTP GET/ but when the server sends back the HTTP 200 responses, they make it to the enc0 interface but don't go out the xn0 interface.

Client - 10.0.1.59 - port 80 -> pfsense01 (10.0.1.144) -> IPSec tunnel to remote pfsense02 (10.120.1.196) -> server (10.120.1.192) port 80.

Here is a tcpdump snippet on the enc0 interface on pfsense01

16:51:07.431412 (authentic,confidential): SPI 0xc5f5038f: IP 10.0.1.159.38356 > 10.120.1.192.80: Flags [S], seq 1592776385, win 26883, options [mss 8961,sackOK,TS val 3698361128 ecr 0,nop,wscale 7], length 0
16:51:07.432767 (authentic,confidential): SPI 0xc44804f6: IP 10.120.1.192.80 > 10.0.1.159.38356: Flags [S.], seq 644580954, ack 1592776386, win 26847, options [mss 8961,sackOK,TS val 1737786209 ecr 3698361128,nop,wscale 7], length 0
16:51:07.433159 (authentic,confidential): SPI 0xc5f5038f: IP 10.0.1.159.38356 > 10.120.1.192.80: Flags [.], ack 1, win 211, options [nop,nop,TS val 3698361128 ecr 1737786209], length 0
16:51:07.433173 (authentic,confidential): SPI 0xc5f5038f: IP 10.0.1.159.38356 > 10.120.1.192.80: Flags [P.], seq 1:77, ack 1, win 211, options [nop,nop,TS val 3698361128 ecr 1737786209], length 76: HTTP: GET / HTTP/1.1
16:51:07.434272 (authentic,confidential): SPI 0xc44804f6: IP 10.120.1.192.80 > 10.0.1.159.38356: Flags [.], ack 77, win 210, options [nop,nop,TS val 1737786211 ecr 3698361128], length 0
16:51:07.434664 (authentic,confidential): SPI 0xc44804f6: IP 10.120.1.192.80 > 10.0.1.159.38356: Flags [P.], seq 1:3760, ack 77, win 210, options [nop,nop,TS val 1737786211 ecr 3698361128], length 3759: HTTP: HTTP/1.1 200 OK
16:51:07.640626 (authentic,confidential): SPI 0xc44804f6: IP 10.120.1.192.80 > 10.0.1.159.38356: Flags [P.], seq 1:3760, ack 77, win 210, options [nop,nop,TS val 1737786416 ecr 3698361128], length 3759: HTTP: HTTP/1.1 200 OK
16:51:07.848088 (authentic,confidential): SPI 0xc44804f6: IP 10.120.1.192.80 > 10.0.1.159.38356: Flags [P.], seq 1:3760, ack 77, win 210, options [nop,nop,TS val 1737786624 ecr 3698361128], length 3759: HTTP: HTTP/1.1 200 OK
16:51:08.288126 (authentic,confidential): SPI 0xc44804f6: IP 10.120.1.192.80 > 10.0.1.159.38356: Flags [P.], seq 1:3760, ack 77, win 210, options [nop,nop,TS val 1737787064 ecr 3698361128], length 3759: HTTP: HTTP/1.1 200 OK
16:51:09.120235 (authentic,confidential): SPI 0xc44804f6: IP 10.120.1.192.80 > 10.0.1.159.38356: Flags [P.], seq 1:3760, ack 77, win 210, options [nop,nop,TS val 1737787896 ecr 3698361128], length 3759: HTTP: HTTP/1.1 200 OK
16:51:10.380709 (authentic,confidential): SPI 0xc5f5038f: IP 10.0.1.159.38356 > 10.120.1.192.80: Flags [F.], seq 77, ack 1, win 211, options [nop,nop,TS val 3698361865 ecr 1737786211], length 0
16:51:10.382346 (authentic,confidential): SPI 0xc44804f6: IP 10.120.1.192.80 > 10.0.1.159.38356: Flags [F.], seq 3760, ack 78, win 210, options [nop,nop,TS val 1737789159 ecr 3698361865], length 0
16:51:10.382843 (authentic,confidential): SPI 0xc5f5038f: IP 10.0.1.159.38356 > 10.120.1.192.80: Flags [R], seq 1592776463, win 0, length 0

Here is the same traffic (most of it) that is on the pfsense01 xn0 (LAN) interface:

16:51:07.431398 IP 10.0.1.159.38356 > 10.120.1.192.80: Flags [S], seq 1592776385, win 26883, options [mss 8961,sackOK,TS val 3698361128 ecr 0,nop,wscale 7], length 0
16:51:07.432779 IP 10.120.1.192.80 > 10.0.1.159.38356: Flags [S.], seq 644580954, ack 1592776386, win 26847, options [mss 8961,sackOK,TS val 1737786209 ecr 3698361128,nop,wscale 7], length 0
16:51:07.433155 IP 10.0.1.159.38356 > 10.120.1.192.80: Flags [.], ack 1, win 211, options [nop,nop,TS val 3698361128 ecr 1737786209], length 0
16:51:07.433171 IP 10.0.1.159.38356 > 10.120.1.192.80: Flags [P.], seq 1:77, ack 1, win 211, options [nop,nop,TS val 3698361128 ecr 1737786209], length 76: HTTP: GET / HTTP/1.1
16:51:07.434281 IP 10.120.1.192.80 > 10.0.1.159.38356: Flags [.], ack 77, win 210, options [nop,nop,TS val 1737786211 ecr 3698361128], length 0
16:51:10.380696 IP 10.0.1.159.38356 > 10.120.1.192.80: Flags [F.], seq 77, ack 1, win 211, options [nop,nop,TS val 3698361865 ecr 1737786211], length 0
16:51:10.382357 IP 10.120.1.192.80 > 10.0.1.159.38356: Flags [F.], seq 3760, ack 78, win 210, options [nop,nop,TS val 1737789159 ecr 3698361865], length 0
16:51:10.382840 IP 10.0.1.159.38356 > 10.120.1.192.80: Flags [R], seq 1592776463, win 0, length 0

They are the same until the server sends back the http 200 responses to the GET.
Any idea why I am getting this response?

Danb · Mar 13, 2019, 4:19 PM

This appears to be an MSS issue. I chnaged the index.html file down to a couple of words and the curl works. Will see if I can resolve by changing the MSS settings.