IP fragmentation, MTU issues using PPPoE Vodafone sure signal
-
I was searching for a slightly different issue and came across this page so thought i'd register and give some hopefully useful input.
Hi Freddy,
Thanks for this. This is interesting information, partly because FTTC is coming to my area in a few months, and it tells me that upgrading to it should cure this issue! ;)I haven't tried it but it should be possible to trace the Ethernet between the router and the separate VDSL modem for Infinity (it is just an Ethernet connection after all) so you might be able to capture the PPPoE packets there to see if you are at least receiving the first, larger fragment there. This would help narrow down where the problem is occurring.
I was considering doing this between my pfSense and Vigor modem. The problem is I don't have a hub any more (my test hub died) and I haven't had time to mess around setting up a switch with mirror ports as it involves making quite a few changes to my network. If I get a chance to do it I will post results.
Cheers,
Keith
-
You can capture from pfSense itself!?
-
@ermal:
You can capture from pfSense itself!?
That's what I did above for the PPP connection. What I hadn't tried (because it hadn't occurred to me to try) was capturing directly from the OPT interface that the modem is connected to. I've tried this now and it seems to work.
However, there's no easy way to filter just the suresignal stuff at capture time as all of the packets are (of course) PPPoE between my IP and the ISPs gateway IP, so I need to do a bit of messing around with wireshark.
Will report back…
Cheers,
Keith
-
I was considering doing this between my pfSense and Vigor modem. The problem is I don't have a hub any more (my test hub died) and I haven't had time to mess around setting up a switch with mirror ports as it involves making quite a few changes to my network. If I get a chance to do it I will post results.
I know what you mean. I had to go to eBay to pick up an Ethernet hub for testing - 25 quid later for a really old Nortel 8 port business hub … but it does the job!
I find there is confusion about PPPoE MTU settings out there. The actual RFC 2516 states the PPP payload MTU should be no more than 1492 octets, so when you add the PPPoE/PPP headers (6 + 2) and Ethernet MAC (18) you end up with a 1518 octet Ethernet packet, which makes sense. In this case, the MTU for a router that is forwarding the IP/PPPoE/PPP PDU between two interfaces would require a minimum MTU of 1500 octets to avoid fragmentation.
Some people seem to treat 1492 as the maximum overall MTU value for the entire Ethernet frame and then deduct the Ethernet MAC and PPP/PPPoE headers from that. So you end up with 1492 - 6 - 2 - 18 = 1466 octets or so as the 'recommended' MTU value to use for the payload. This shouldn't really cause any problems if you were to set it, until you come up against a sender that doesn't want to play ball with changing its MTU.
The SeGW's that are used for a system like the VF SS service are pretty hefty pieces of hardware designed to simultaneously support hundreds of thousands of high speed IPsec tunnels … to be able to do this highly computational stuff efficiently a lot of processing is done in canned hardware implementations (or NPs) so it's maybe not in their interest to have to worry about fragmenting packets as well for the occasional customer that has a lower MTU set as this would mean having to further increase complexity.
Interestingly, I see that the Don't Fragment bit on packets received from the SeGW is not set so ICMP messages for "destination unreachable, datagram too big but DF bit set" may not be generated back to the SeGW anyway.
-
I was considering doing this between my pfSense and Vigor modem. The problem is I don't have a hub any more (my test hub died) and I haven't had time to mess around setting up a switch with mirror ports as it involves making quite a few changes to my network. If I get a chance to do it I will post results.
I know what you mean. I had to go to eBay to pick up an Ethernet hub for testing - 25 quid later for a really old Nortel 8 port business hub … but it does the job!
Actually I found I could do it direct from pfSense as suggested by ermal above. This may not apply to you (Freddy) as you haven't said that you are using pfSense. But for those that are:
To do this you need to create a new OPT interface using the physical interface that your modem is connected to. You should already have a "WAN" interface set up for PPPoE. Fo instance, my modem is connected to rl0 so this appears as using Network port PPPOE0(rl0). You can then set another interface up using just rl0. In my case I had already done that to get tot the modem web/telnet interface.
You can then put packet capture on this interface.
I find there is confusion about PPPoE MTU settings out there. The actual RFC 2516 states the PPP payload MTU should be no more than 1492 octets, so when you add the PPPoE/PPP headers (6 + 2) and Ethernet MAC (18) you end up with a 1518 octet Ethernet packet, which makes sense. In this case, the MTU for a router that is forwarding the IP/PPPoE/PPP PDU between two interfaces would require a minimum MTU of 1500 octets to avoid fragmentation.
In normal circumstances, 1492 is the largest MTU supported by PPPoE, as you suggest and RFC 2516 state. However, it is possible to break past the normal Ethernet frame limitation in order to accommodate the extra 8 octets. This breaks traditional Ethernet rules, but there is a precedent for this in Jumbo frames. I have seen this referred to as "baby jumbo frames" as the frames are only slightly bigger. This is covered by RFC 4638. Apparently the BT VDSL service will support this as long as your equipment will support it. Some devices, like the Firebrick, do support this.
I'm not sure what the situation with pfSense would be here. I'm guessing pfSense doesn't currently support RFC 4638, but I'm also guessing that the dependency is largely on the actual hardware it is deployed on and the drivers for the Ethernet ports, but it seems like it would be a desirable capability to have on pfSense.
In my case I would also need my modem to support it.
Some people seem to treat 1492 as the maximum overall MTU value for the entire Ethernet frame and then deduct the Ethernet MAC and PPP/PPPoE headers from that. So you end up with 1492 - 6 - 2 - 18 = 1466 octets or so as the 'recommended' MTU value to use for the payload. This shouldn't really cause any problems if you were to set it, until you come up against a sender that doesn't want to play ball with changing its MTU.
Yes I have seen many variations of this. In theory PPP should do the right thing and negotiate the best MTU that both ends support. I have no feelings on whether this is the case in practice or not.
Note that some ISPs (especially, I think, in the US) offer a service which is PPPoEoA, in contrast to what we are talking about here which is a PPPoA service bridged to PPPoE. With PPPoEoA you have an extra layer in which uses an additional 24 octets. in this case you end up with an MTU of 1468.
FYI, the reason I went with 1456 is that it's supposed to be a more efficient frame size for ATM based networks (as ADSL typically is). The reason is ATM has fixed sized cells which the IP packet is encapsulated into. If the packet is bigger than one cell, it uses multiple cells. If the packet doesn't exactly fill the cells, you end up with cells containing empty padding. 1456 is supposed to be the magic number which completely fills the ATM cells with no padding. Packets above this will result in a cell per packet which is mostly empty. However, I have seen other descriptions which suggest that 1478 is the optimum if your service uses VC based multiplexing (which I believe mine does).
The SeGW's that are used for a system like the VF SS service are pretty hefty pieces of hardware designed to simultaneously support hundreds of thousands of high speed IPsec tunnels … to be able to do this highly computational stuff efficiently a lot of processing is done in canned hardware implementations (or NPs) so it's maybe not in their interest to have to worry about fragmenting packets as well for the occasional customer that has a lower MTU set as this would mean having to further increase complexity.
This might be the case, but there seems to be a significant number of customers impacted by this issue. It may well be worth their while tweaking the MTU on the SeGW's to 1492 to cure the issue. It seems their devices are having to fragment these packets into two fragments already anyway. I don't see that fragmenting into two packets with different sizes will make any real difference to anything. Of course there is the possibility that other packets which would have fit into the 1500 octet MTU would now get fragmented, but this is statistically fairly unlikely. And the savings on handling the support calls alone might make this worthwhile.
Interestingly, I see that the Don't Fragment bit on packets received from the SeGW is not set so ICMP messages for "destination unreachable, datagram too big but DF bit set" may not be generated back to the SeGW anyway.
OK, this is VERY useful information. I was unable to determine if this was the case or not as I wasn't seeing these packets. In this case the fragmentation should be done within the ISPs network, and it suggests this isn't being done properly.
Cheers,
Keith
-
OK, results of PPPoE sniffing. Full packet dump below but, basically, these packets aren't appearing on the PPP layer either (unless the modem is filtering them out during bridging, which I find pretty unlikely).
No. Time Source Destination Protocol Info 1317 17.077093 <my-wan-ip> 212.183.133.179 ISAKMP IKE_SA_INIT Frame 1317 (474 bytes on wire, 474 bytes captured) Ethernet II, Src: Fabiatec_08:99:2e (00:04:a7:08:99:2e), Dst: Draytek_38:b5:c2 (00:50:7f:38:b5:c2) PPP-over-Ethernet Session Point-to-Point Protocol Internet Protocol, Src: <my-wan-ip> (<my-wan-ip> ), Dst: 212.183.133.179 (212.183.133.179) User Datagram Protocol, Src Port: isakmp (500), Dst Port: isakmp (500) Internet Security Association and Key Management Protocol No. Time Source Destination Protocol Info 1319 17.431679 212.183.133.179 <my-wan-ip> ISAKMP IKE_SA_INIT Frame 1319 (555 bytes on wire, 555 bytes captured) Ethernet II, Src: Draytek_38:b5:c2 (00:50:7f:38:b5:c2), Dst: Fabiatec_08:99:2e (00:04:a7:08:99:2e) PPP-over-Ethernet Session Point-to-Point Protocol Internet Protocol, Src: 212.183.133.179 (212.183.133.179), Dst: <my-wan-ip> (<my-wan-ip> ) User Datagram Protocol, Src Port: isakmp (500), Dst Port: isakmp (500) Internet Security Association and Key Management Protocol No. Time Source Destination Protocol Info 1320 17.875322 <my-wan-ip> 212.183.133.179 ISAKMP IKE_AUTH Frame 1320 (514 bytes on wire, 514 bytes captured) Ethernet II, Src: Fabiatec_08:99:2e (00:04:a7:08:99:2e), Dst: Draytek_38:b5:c2 (00:50:7f:38:b5:c2) PPP-over-Ethernet Session Point-to-Point Protocol Internet Protocol, Src: <my-wan-ip> (<my-wan-ip> ), Dst: 212.183.133.179 (212.183.133.179) User Datagram Protocol, Src Port: 5646 (5646), Dst Port: ipsec-nat-t (4500) UDP Encapsulation of IPsec Packets Internet Security Association and Key Management Protocol No. Time Source Destination Protocol Info 1323 17.962236 212.183.133.179 <my-wan-ip> IP Fragmented IP protocol (proto=UDP 0x11, off=1480, ID=d5cf) Frame 1323 (68 bytes on wire, 68 bytes captured) Ethernet II, Src: Draytek_38:b5:c2 (00:50:7f:38:b5:c2), Dst: Fabiatec_08:99:2e (00:04:a7:08:99:2e) PPP-over-Ethernet Session Point-to-Point Protocol Internet Protocol, Src: 212.183.133.179 (212.183.133.179), Dst: <my-wan-ip> (<my-wan-ip> ) Data (16 bytes) 0000 95 f4 16 30 bf 05 82 6e 7e 9f 8b 46 6f 8c a9 20 ...0...n~..Fo.. No. Time Source Destination Protocol Info 1342 21.878812 <my-wan-ip> 212.183.133.179 ISAKMP IKE_AUTH Frame 1342 (514 bytes on wire, 514 bytes captured) Ethernet II, Src: Fabiatec_08:99:2e (00:04:a7:08:99:2e), Dst: Draytek_38:b5:c2 (00:50:7f:38:b5:c2) PPP-over-Ethernet Session Point-to-Point Protocol Internet Protocol, Src: <my-wan-ip> (<my-wan-ip> ), Dst: 212.183.133.179 (212.183.133.179) User Datagram Protocol, Src Port: 5646 (5646), Dst Port: ipsec-nat-t (4500) UDP Encapsulation of IPsec Packets Internet Security Association and Key Management Protocol No. Time Source Destination Protocol Info 1345 21.970402 212.183.133.179 <my-wan-ip> IP Fragmented IP protocol (proto=UDP 0x11, off=1480, ID=dca9) Frame 1345 (68 bytes on wire, 68 bytes captured) Ethernet II, Src: Draytek_38:b5:c2 (00:50:7f:38:b5:c2), Dst: Fabiatec_08:99:2e (00:04:a7:08:99:2e) PPP-over-Ethernet Session Point-to-Point Protocol Internet Protocol, Src: 212.183.133.179 (212.183.133.179), Dst: <my-wan-ip> (<my-wan-ip> ) Data (16 bytes) 0000 95 f4 16 30 bf 05 82 6e 7e 9f 8b 46 6f 8c a9 20 ...0...n~..Fo.. No. Time Source Destination Protocol Info 1391 29.083752 <my-wan-ip> 212.183.133.179 ISAKMP IKE_AUTH Frame 1391 (514 bytes on wire, 514 bytes captured) Ethernet II, Src: Fabiatec_08:99:2e (00:04:a7:08:99:2e), Dst: Draytek_38:b5:c2 (00:50:7f:38:b5:c2) PPP-over-Ethernet Session Point-to-Point Protocol Internet Protocol, Src: <my-wan-ip> (<my-wan-ip> ), Dst: 212.183.133.179 (212.183.133.179) User Datagram Protocol, Src Port: 5646 (5646), Dst Port: ipsec-nat-t (4500) UDP Encapsulation of IPsec Packets Internet Security Association and Key Management Protocol No. Time Source Destination Protocol Info 1394 29.162119 212.183.133.179 <my-wan-ip> IP Fragmented IP protocol (proto=UDP 0x11, off=1480, ID=e936) Frame 1394 (68 bytes on wire, 68 bytes captured) Ethernet II, Src: Draytek_38:b5:c2 (00:50:7f:38:b5:c2), Dst: Fabiatec_08:99:2e (00:04:a7:08:99:2e) PPP-over-Ethernet Session Point-to-Point Protocol Internet Protocol, Src: 212.183.133.179 (212.183.133.179), Dst: <my-wan-ip> (<my-wan-ip> ) Data (16 bytes) 0000 95 f4 16 30 bf 05 82 6e 7e 9f 8b 46 6f 8c a9 20 ...0...n~..Fo.. No. Time Source Destination Protocol Info 1482 42.050727 <my-wan-ip> 212.183.133.179 ISAKMP IKE_AUTH Frame 1482 (514 bytes on wire, 514 bytes captured) Ethernet II, Src: Fabiatec_08:99:2e (00:04:a7:08:99:2e), Dst: Draytek_38:b5:c2 (00:50:7f:38:b5:c2) PPP-over-Ethernet Session Point-to-Point Protocol Internet Protocol, Src: <my-wan-ip> (<my-wan-ip> ), Dst: 212.183.133.179 (212.183.133.179) User Datagram Protocol, Src Port: 5646 (5646), Dst Port: ipsec-nat-t (4500) UDP Encapsulation of IPsec Packets Internet Security Association and Key Management Protocol No. Time Source Destination Protocol Info 1485 42.152763 212.183.133.179 <my-wan-ip> IP Fragmented IP protocol (proto=UDP 0x11, off=1480, ID=000d) Frame 1485 (68 bytes on wire, 68 bytes captured) Ethernet II, Src: Draytek_38:b5:c2 (00:50:7f:38:b5:c2), Dst: Fabiatec_08:99:2e (00:04:a7:08:99:2e) PPP-over-Ethernet Session Point-to-Point Protocol Internet Protocol, Src: 212.183.133.179 (212.183.133.179), Dst: <my-wan-ip> (<my-wan-ip> ) Data (16 bytes) 0000 95 f4 16 30 bf 05 82 6e 7e 9f 8b 46 6f 8c a9 20 ...0...n~..Fo..</my-wan-ip></my-wan-ip></my-wan-ip></my-wan-ip></my-wan-ip></my-wan-ip></my-wan-ip></my-wan-ip></my-wan-ip></my-wan-ip></my-wan-ip></my-wan-ip></my-wan-ip></my-wan-ip></my-wan-ip></my-wan-ip></my-wan-ip></my-wan-ip></my-wan-ip></my-wan-ip></my-wan-ip></my-wan-ip></my-wan-ip></my-wan-ip></my-wan-ip></my-wan-ip></my-wan-ip></my-wan-ip></my-wan-ip></my-wan-ip>Cheers,
Keith
-
As an aside, I raised a support ticket with my ISP on this.
They are suggesting that my network is "too complex" which is a ridiculous statement.
I'm using a standard off-the-shelf DSL modem and PPPoE. They do not provide any router or modem themselves, and they advertise it as a "business" service (and charge accordingly), and business customers are far more likely to be using a PPPoE device instead of a 40 quid consumer-grade router.
I'm going to make my feelings known, but I suspect I will be changing ISP in the near future.
Cheers,
Keith
-
In normal circumstances, 1492 is the largest MTU supported by PPPoE, as you suggest and RFC 2516 state. However, it is possible to break past the normal Ethernet frame limitation in order to accommodate the extra 8 octets. This breaks traditional Ethernet rules, but there is a precedent for this in Jumbo frames. I have seen this referred to as "baby jumbo frames" as the frames are only slightly bigger. This is covered by RFC 4638. Apparently the BT VDSL service will support this as long as your equipment will support it. Some devices, like the Firebrick, do support this.
I'm not sure what the situation with pfSense would be here. I'm guessing pfSense doesn't currently support RFC 4638, but I'm also guessing that the dependency is largely on the actual hardware it is deployed on and the drivers for the Ethernet ports, but it seems like it would be a desirable capability to have on pfSense.
Having checked into this, neither the Realtek (rt) or Agere (et) drivers/hardware for BSD do not seem to support MTUs larger than 1500, so it's probably not supported on my hardware.
Cheers,
Keith
-
Fascinating stuff!
Do you mind if I ask who your ISP is?
Steve
-
I would rather not say at the moment, as I still have a ticket open, and I would rather give them the chance to redeem themselves before posting negative publicity which can be directly attributed to them.
The thing that bothers me is I'm pointing out a flaw in their network, and they don't seem to understand that I'm trying to help them. Quite frankly I'm not that bothered about getting the SureSignal device working (I got it sent to me for free by my Vodafone Business Account Manager). What I do care about is having a professional, competent, and clueful ISP, especially when I have been paying a considerably premium to get a "business grade" service.
They keep saying things like "We don't support PPPoE Authentication" but then promote and sell a premium version of their service "for business use", as far as I'm concerned these two statements are mutually incompatible.
It's not really impacting me in any practical way at the moment, but if they've still not sorted or at least acknowledged it in a few months time when FTTC is supposed to be in my area, I'll be using switching to a new provider. At that point I will be only too happy to share who they are.
Cheers,
Keith
-
That seems fair.
If your ISP is llu type then I think you have a chance. If you are reliant on btw then good luck!
I haven't experinced any problems here with either the vigor 120 or the fttc modem but I don't have a vss. It would be interesting to run a test but I'm not sure what might show up the problem.Steve
-
That seems fair.
If your ISP is llu type then I think you have a chance. If you are reliant on btw then good luck!It is (apparently) LLU, which is why it is so annoying to me that they don't seem willing to entertain checking this or at least passing it along to someone in their team who is more clueful.
Ultimately, the answer might come back as "yes we know this is an issue but we are unable/unwilling to support this because of xxxx", and that's an answer I would accept. There may be good technical/operation reasons.
Incidentally, I have heard that people who use a Be line have had more success. I believe this is down to their use of PPPoE on their backhaul circuits which means they have an MTU of 1492 across most of their access network anyway and, therefore, know how to configure their kit to support this properly.
Cheers,
Keith
