UDP fragmented packet loss / IPv6 / VoIP / pfSense version inconsistency
-
I am running pfSense 2.4.2 using FreeBSD 11.1-RELEASE-p4 on a Mini-ITX based system with Celeron N2930 processor. I am based in he UK and use the BT Infinity service. I have been trialling SIP for some months and found everything worked well using IPv6. I am using SipGate. Recently I became aware I was no longer receiving incoming calls which were always diverted to voicemail. I scanned firewall logs etc. but could find no sign of blocked packets.
Trying to progress the problem, and initially suspecting Snort/pfBlockerNG were implicated, I built a separate instance of pfSense on an old PC 'out of the skip'. More specifically this is an i386 machine and so is using pfSense 2.3.5. This works correctly and incoming SIP calls 'ring' and connect as expected.
Having failed to find the 'killer' difference I spotted the packet capture tool which exposed the problem. It appears my production AMD64 2.4.2 system is dropping the SIP INVITE udp packets and issuing "ICMPV6 packet too big" messages back to the SIP server.
Here is a snip from packet capture:
11:56:03.998498 IP6 2001:ab7::5 > 2a00:23c5:d007:8700:7e2f:80ff:fe20:ce8a: frag (0|1440) 5060 > 5060: UDP, bad length 1475 > 1432
11:56:03.998623 IP6 2001:ab7::5 > 2a00:23c5:d007:8700:7e2f:80ff:fe20:ce8a: frag (1440|43)
11:56:03.998735 IP6 2a00:23c5:d007:8700:230:18ff:fec9:ce4a > 2001:ab7::5: ICMP6, packet too big, mtu 1500, length 124The equivalent packet capture from the i386 2.3.5 system shows the SIP INVITE is fragmented but is not dropped and no "ICMPV6 packet too big" is sent.
I have tried switching off scrubbing, setting MTU and MSS, disabling hardware checksum offloading but all to no avail. Scanning various online forums gives the impression this could be a FreeBSD bug but also that it has been fixed.
Given my SIP installation originally worked I feel it stopped working when I upgraded to 2.4.0 in October 2017.
Any help / advice gratefully received.
Mike
-
The equivalent packet capture from the i386 2.3.5 system shows the SIP INVITE is fragmented but is not dropped and no "ICMPV6 packet too big" is sent.
If it was fragmented, it wasn't IPv6. Only the source is allowed to fragment with IPv6. How big are those packets according to packet capture? You may have to export the capture to Wireshark to see. Why is the source not responding to the "too big" and reducing the packet size? The bad length 1475 > 1432, while the system has a 1500 MTU is strange.
-
Attached is screenshot of Wireshark showing the exchange of packets (not the same sample as previously quoted but all such exchanges are very similar). Wireshark claims them all to be V6. My knowledge is strictly limited [ever day I ponder this problem I just realise how little I understand :(]. I do not understand why the packet is deemed too big either. If the sender (SipGate) processed the too big message would it not just fragment them anyway? I've attached the .cap file as others may gain more insight using Wireshark than I. The capture filtered traffic to/from the SipGate network. I've not seen any related IPv4 traffic.
Given this behaviour does not occur with 2.3.5 i386 (and different hardware) I'm inclined to think this is a bug but I wouldn't want to call it between pfSense / FreeBSD and the hardware (NIC).
[packetcapture (3).cap.zip](/public/imported_attachments/1/packetcapture (3).cap.zip) -
One thing I see, the fragment is 1492 and the original invite is 98 bytes, which is a total of 1590. What's the MTU of the device that's sending that?
Regardless, the mandatory function on IPv6 is to send a too big message. Fragmenting along the route is prohibited. So, this fragment has to be coming from the source. Even with IPv4, a router can only fragment a packet that's passing through it. It cannot create a fragment on the input. That has to be done upstream.
-
Not sure if there is a basic misunderstanding here (which might be mine :)). The packet capture is on the WAN port so the Invite packet is arriving fragmented from SipGate. The issue then is that instead of processing the packet and sending it to the LAN side it's being dropped [so the Invite is never delivered and the phone doesn't 'ring']. I can ask SipGate tech support what the MTU is on their server and wouldn't be surprised to learn its 1500. That's perhaps why a 1590 byte Invite is being fragmented.
Why does my 2.3.5 i386 instance of pfSense process the packet? Surely the mandatory function on IPv6 is to send a too big message is precisely the point any my 2.4.2 pfSense box isn't. That's rather disappointing at the very least…
-
^^^^
That too big message is part of Path MTU Discovery, which is mandatory on IPv6 and often used on IPv4. The main reason for PMTUD is to reduce the processing load on routers and also to reduce the buffering requirements at the destination. Perhaps a chat with someone at SipGate can help. I don't know why pfSense 2.4.2 isn't passing what should be a valid packet. Incidentally, I just experimented with oversize pings to pfSense and get the same sort of fragment as shown in your capture. So, it would appear that fragment is generated at the source. According to your capture, pfSense is sending the too big message, as expected.Did you do a capture on the LAN side? Does it show anything of that Invite?
-
One thing I've notice. In the System > Advanced > System Tunables page there's a value net.inet.udp.maxdgram. On my system, it's 57344. I wonder if that or some other value is causing pfSense to not pass larger datagrams (IP supports up to 64K bytes in a datagram). This is unrelated to the MTU size.
-
JKnott - thanks for responding to me on this issue - I appreciate the help. :)
I understand the purpose of the 'too big' message but get the impression that the packets that trigger it being sent are dropped pending a re-transmission with a smaller size. (The retransmission doesn't happen - I see a lot of speculation that ICMP packets are often filtered out for security reasons).
My contention is the 'too big' message is being sent in error - the result of a failure to process the fragments correctly perhaps. I just ran a capture on the LAN port: with an outbound call. All the SIP and RTP packets are seen as expected. However the incoming Invite packets never make it to the LAN side.
Since 2.3.5 manages to process the fragmented Invite packets (arriving from SipGate on the WAN port) and do not trigger ICMPV6 too big messages, I can assume they are processed correctly (confirmed by the phone actually ringing).
I think System > Advanced > System Tunables value net.inet.udp.maxdgram has nothing to do with this problem as it's the same for both the working (2.3.5 i386) and non working (2.4.2 AMD64) version.
Is there any way to check the IP parameters at the FreeBSD level? Sometimes the bug can be between pfSense and the O/S so the parameters for BSD are being set up incorrectly. If that's not the case then this might be a bug in BSD 11.1-RELEASE-p4. Were it a glitch in the hardware I can't see how it ever worked (which it did pre 2.4.0 AMD64).
-
FWIW, I just tried an experiment. First off, I tried pinging, with both IPv4 & IPv6, with oversize packets to a computer on the local LAN. Both worked fine. I then connected one computer to another interface on my firewall and tried again. This time IPv4 worked, but IPv6 didn't. I can see the fragmented IPv6 pings leaving one computer, but not arriving on the other. So, it appears pfSense/FreeBSD is not passing packets that have been fragmented by the source, but does pass IPv4.
Looks like it's time to file a bug report, as it should be passing both IPv4 and IPv6 fragments.
-
JKnott - thanks for confirming the problem. Are you going to file a bug report or do you expect me to? I'm happy not to…. ;D
-
I'll let you have the honours. :)
-
#8165
:)
-
Looks like https://redmine.pfsense.org/issues/8165 is closed to early. We still see problems with IPv6 fragments, in our case with local created ones which simply disappear. Depending on certificates and keysizes used Strongswan will use "oversized" UDP packets in the IKEv2 connection etsablishment. If the remote side does not support IKEv2 Fragmentation (Windows older than Version 10 /1803) the packet is never leaving the pfsense box if IPv6 is used. A Capture done at the WAN Interface show that this packet is simply missing and therefore the handshake never completes. This is still the case on latest 2.4.4-RELEASE-p1.
