PF sending damaged packet to pflog: truncated-ip and bad-hlen ("bytes missing!")
-
"tcpdump -nettti pflog0" is getting a damaged packet from PF, but PF sees the packet just fine. It looks like the packet is missing 4 bytes somewhere near the start of the packet.
As an example, here's a packet that PF is dropping that's going 192.168.255.21 > 10.254.254.1. "tcpdump -nevvvttti pflog0" says this about that packet:
00:00:00.236482 rule 10..16777216/0(match): block out on re2: IP6 , wrong link-layer encapsulationtruncated-ip - 16324 bytes missing! (tos 0x2d,ECT(1), ttl 192, id 16134, offset 26216, flags [none], proto unknown (168), length 16384, options (traceroute [bad length 5]), bad cksum ff15 (->baa5)!)
10.254.254.1 > 23.12.236.104: ip-proto-168As you can see, the destination address is now in the source address field, which means that those bytes have shifted 4 bytes to the left. That's confirmed by the TTL field being populated by 192 and the IP proto field being populated by 168 (both from the real source address 192.168.255.21).
I can capture the packet just fine using tcpdump on the interface:
15:11:07.986903 00:1f:f3:46:0f:cc > 00:0d:b9:34:1c:fe, ethertype IPv4 (0x0800), length 78: (tos 0x0, ttl 64, id 26157, offset 0, flags [DF], proto TCP (6), length 64)
192.168.255.21.5900 > 10.254.254.1.60520: Flags [S.], cksum 0x7642 (correct), seq 1376072758, ack 2130060610, win 65535, options [mss 1460,nop,wscale 3,nop,nop,TS val 578270979 ecr 1519883763,sackOK,eol], length 0
0x0000: 4500 0040 662d 4000 4006 0bcd c0a8 ff15 E..@f-@.@…....
0x0010: 0afe fe01 170c ec68 5205 3436 7ef6 2542 .......hR.46~.%B
0x0020: b012 ffff 7642 0000 0204 05b4 0103 0303 ....vB..........
0x0030: 0101 080a 2277 b703 5a97 95f3 0402 0000 ...."w..Z.......Looking in filter.log I see that PF processed the undamaged packet (the addresses are identified) correctly):
Oct 24 15:11:08 netgate filterlog: 10,16777216,,1000000104,re2,match,block,out,4,0x0,,63,26157,0,DF,6,tcp,64,192.168.255.21,10.254.254.1,5900,60520,0,SA,1376072758,2130060610,65535,,mss;nop;wscale;nop;nop;TS;sackOK;eol
I believe this started when I upgraded one of the last times. I'm on 2.2.4-RELEASE (amd64) built on Sat Jul 25 19:59:52 CDT 2015 FreeBSD 10.1-RELEASE-p15.
I tried disabling TSO, LRO and offloaded checksums, but (following a reboot) that didn't fix the problem.
Any idea what is going wrong?
-
"00:00:00.236482 rule 10..16777216/0(match): block out on re2: IP6 , wrong link-layer encapsulationtruncated-ip "
This is saying it's an IP6, but tcpdump is saying IPV4. Isn't IP4/IP6 part of headers before IP addresses?
Do you have (want) IP6 enabled? Is VLAN enabled anywhere?
-
@mer:
This is saying it's an IP6, but tcpdump is saying IPV4.
It says "IP6", not "IPv6". They're both tcpdump, and they're both processing the exact same packet, just in the 2nd case it has been damaged. Presumably it thinks it's IPv4 in both cases because it's outputting IPv4 addresses.
@mer:
Isn't IP4/IP6 part of headers before IP addresses?
Yes, the first 4 bits of the packet.
@mer:Do youhave (want) IP6 enabled? Is VLAN enabled anywhere?
Yes it's enabled, it doesn't matter to me if it is. No, no VLAN.
-
Usually that "damaged packet" message is a red herring due to incomplete tcpdump parameters/processing. Increase the snaplen on tcpdump so it gets the whole entry.
Try something like this:
tcpdump -s 256 -v -S -l -n -e -ttt -i pflog0
That's what the old log processor used to use before we moved to our current filterlog daemon that doesn't rely on tcpdump.
-
More or less the same thing, and this time with 10.3-RELEASE:
$ sudo tcpdump -s 256 -v -S -l -n -e -ttt -i pflog0
tcpdump: WARNING: pflog0: no IPv4 address assigned
tcpdump: listening on pflog0, link-type PFLOG (OpenBSD pflog file), capture size 256 bytes
00:00:00.000000 rule 10..16777216/0(match): block out on re2: IP10 truncated-ip - 16324 bytes missing! (tos 0x78, ttl 192, id 16134, offset 11272, flags [DF, rsvd], proto unknown (168), length 16384, options (unknown 106,unknown 3,NOP,NOP,unknown 8 [bad length 10]), bad cksum ff15 (->e77c)!)
10.254.254.1 > 23.12.192.56: ip-proto-168
00:00:00.952885 rule 10..16777216/0(match): block out on re2: IP5 truncated-ip - 16324 bytes missing! (tos 0x7d,ECT(1), ttl 192, id 16134, offset 54248, flags [none], proto unknown (168), length 16384, options (unknown 106 [bad length 21]), bad cksum ff15 (->35fc)!)
10.254.254.1 > 23.12.192.56: ip-proto-168but the packets aren't actually deformed, as they can be seen using tcpdump on the interface itself:
$ sudo tcpdump -i re2 -s 0 -n -v port 5900
tcpdump: listening on re2, link-type EN10MB (Ethernet), capture size 65535 bytes
18:42:18.472049 IP (tos 0x0, ttl 64, id 44408, offset 0, flags [DF], proto TCP (6), length 64)
192.168.255.21.5900 > 10.254.254.1.49208: Flags [S.], cksum 0xa575 (correct), seq 1779813320, ack 365575002, win 65535, options [mss 1460,nop,wscale 3,nop,nop,TS val 1249855937 ecr 802858585,sackOK,eol], length 0
18:42:19.424994 IP (tos 0x0, ttl 64, id 22653, offset 0, flags [DF], proto TCP (6), length 64)
192.168.255.21.5900 > 10.254.254.1.49208: Flags [S.], cksum 0xa56c (correct), seq 1779813320, ack 365575002, win 65535, options [mss 1460,nop,wscale 3,nop,nop,TS val 1249855946 ecr 802858585,sackOK,eol], length 0Again, it looks like "tcpdump -nettti pflog0" is getting a damaged packet from PF, with exactly 4 bytes chopped off near (or at) the beginning of the packet.
-
Why are you monitoring the pflog interface directly on 2.3 (or even 2.2)? It's not necessary. Everything uses filterlog or the actual filter.log now, not tcpdump directly.
-
Hi, just wanted to mention this could be related to:
https://forum.pfsense.org/index.php?topic=114353.0