Performance issue



  • Hiya again,

    I'd say I'm quite experienced in troubleshooting issues but I'm really scratching my head over this one. In short, I'm using Ubuntu boxes to terminate a VTI route based IPSec tunnel. The performance is great until I route through PfSense, it was working fine in the release before 2.3 so something is wrong.

    The setup is as follows:

    Site A

    • PfSense running in ESX(i)

    • WAN interface is connected to a bridged cable modem

    • DMZ interface has a Ubuntu box behind it

    • WAN speed is 200/40 Mbit/s

    Site B

    • PfSense running on Atom D2700 Board

    • WAN interface is connected to FttH through a managed switch

    • DMZ interface has a Ubuntu box behind it

    • WAN speed is 500/500 Mbit/s

    Setup

    The Ubuntu boxes are both behind NAT. Port 500 and 4500 UDP are forwarded from the WAN on both sites. The boxes establish a tunnel using VTI interfaces and use packet marks with 0.0.0.0/0 selectors for a fully route-based tunnel. This works like a charm, the PfSense boxes just have a /16 route for the other site pointing to the directly connected Ubuntu gateway.

    Both PfSense boxes have a LAN interface too serving clients, this is irrelevant to the problem but when I ping from my LAN the routing path is:

    Client sends traffic to the LAN IP of PfSense
    PfSense routes the packet to the DMZ address of the Ubuntu Box
    Ubuntu encapsulates the packet in NAT-T and sends it out through its default gateway out to the internet

    Packet arrives on the other side, PfSense NAT's it to the Ubuntu Box
    Ubuntu decapsulates the packet and forwards it to PfSense
    Pfsense routes it out to the LAN interface where the destination sits

    The performance on the 2.2.6 was great.. 200 Mbit/s in one direction and 40 Mbit/s in the other thus hitting my WAN limits with PLENTY of CPU power to spare.

    Problem and observations

    Ok, the interesting bit that got me scratching my head.

    • When I run iPerf between the ubuntu boxes the performance is a solid 200/40. No CPU on WAN link problem.

    • When I deploy another machine in the DMZ segment and use the ubuntu box as a gateway again 200/40. So routing through the Ubuntu Box is not a problem.

    • When I iPerf from the PfSense box on my side it's sluggish, like 50/20 Mbit/s. The problem must lie in PfSense. I'm not even routing through PfSense; just iPerf'ing FROM PfSense. Same behaviour when using a client behind PfSense.

    • Ok I thought, maybe a problem with my box. Let's run iPerf from the other box straight to the Ubuntu box on the other side.. The same behaviour is displayed. Poor throughput.

    Ok so we have established that:

    Ubuntu1 –> Ubuntu2 = Fast
    Ubuntu1 <-- Ubuntu2 = Fast
    Pfsense --> Ubuntu1--> Ubuntu2 = Slow
    Ubuntu1 <-- Ubuntu2 <-- PfSense = Slow

    Right, ok. So when iPerf'ing from/to either of the PfSense boxes the transfer is slow. The only similarity is that they both run PfSense 2.3, one box is a VM the other is HW. Different cables and switches but the same problem.

    • Hmm ok could it be a performance issue between PfSense and the Ubuntu box maybe? To find out I did an iPerf between the Ubuntu boxes and their local PfSense. All directions 1Gbit/s no sweat.

    • Did some packet capturing and confirmed the MTU bottleneck on the VTI interfaces is honoured by PMTU. The packets do not exceed 1400 bytes and they do not fragment on the WAN interfaces. Both 1500 WAN links.

    • The only oddity is that I see many "Duplicate ACK's" and "Out-of-orders" but ONLY when coming from or through a 2.3 box

    So to recap

    • Performance from PfSense to Ubuntu is 1Gbit/s on both sites

    • Performance from Ubuntu to Ubuntu through IPSec is 200/40

    • Performance from a client in the DMZ lan to the remote Ubuntu box through the local Ubuntu box is 200/40

    • Performance from PfSense to the remote Ubuntu box throught the local Ubuntu box is SLOW

    • This worked fine before the update

    • Problem is identical on both sides despide completely different hardware

    So if the tunnel is performing fine. The performance from a Windows machine through the tunnel is fine why is PfSense showing this issue whilst local routing performance is fine?

    What changed in 2.3 that could cause this strange behaviour? Sorry for the long read but I've been hammering at this for a while now and I can't get it solved.

    This is the relevant portion of the topology:

    This traffic is fast both ways using the inside tunnel addresses both ubuntu boxes achieve 200/40 INSIDE the tunnel.

    This traffic from PfSense to the remote Ubuntu box using the local directly attached Ubuntu box is SLOW:

    This too is slow:

    It's not the tunnel or routing through the ubuntu box because this is FAST:

    The pics might be confusing but there is no loopback routing weirdness here because:

    The traffic coming out of PfSense going to Ubuntu is normal TCP/UDP routed based on a /16 static route.
    The traffic coming out of ubuntu back to PfSense is encapsulated NAT-T traffic destined for a WAN IP which is routed out the WAN.

    I think this is a problem introduced with 2.3 because it was working on 2.2.6 and the problem occurs on 2 completely different PfSense machines. Anyone who can shed some light on this please?

    :-[ :-[ :-[



  • I have replaced the PfSense Box with a Ubuntu 15.10 purely using IPTables and the problem persists.. PfSense is not on the list of suspects anymore. I think some regression maybe got into Ubuntu as it was working before, maybe it was introduced around the same time I upgraded to 2.3..

    I'll do more investigation..



  • Hi,

    just had a very short look over your posting. I would give it a try to lower the MTU size on the servers end. Just try to lower it to something like 1260. Then try again.
    Hade scenarios where performance was slow and lowering the MTU helped - or try with setting MMS-clamping to adjust windows size in tcp connections. Not working for udp.



  • I have checked the packet dumps as this was my first thought but the packets leaving the Ubuntu Box never exceed the WAN MTU. The packets coming into the Ubuntu Box never exceed the 1400 MTU limit on the VTI. It seems that PMTU is doing it's thing. I have also created a mangle rule (MSS Clamping) to keep the size down which gives me exactly the same result. What I have not done is lower the MTU of the client behind the VPN as I had confirmed the MTU size was being honored.

    I will give this a try this evening and report back. Thanks.


  • Rebel Alliance Global Moderator

    "When I deploy another machine in the DMZ segment and use the ubuntu box as a gateway again 200/40"

    This is a problematic configuration to be sure.  Not only are you hairpinning, you most likely than not also have a asynchronous routing condition

    Lets look at your drawing. I see 2 problem with how your data flows in your testing.




  • @johnpoz:

    "When I deploy another machine in the DMZ segment and use the ubuntu box as a gateway again 200/40"

    This is a problematic configuration to be sure.  Not only are you hairpinning, you most likely than not also have a asynchronous routing condition

    Lets look at your drawing. I see 2 problem with how your data flows in your testing.

    Thanks for the reply,

    No I don't think it hairpins because the PfSense box has a static route for the remote subnet pointing to the ubuntu machine. The Ubuntu machine does not route the session back to PfSense but Encapsulates the traffic in ESP. That ESP is a whole different conversation between the Ubuntu box and the remote external side.

    So data flows in this scenario:

    PfSense (172.21.12.1/28) –> Ubuntu (172.21.12.2/28)
    Ubuntu routes the traffic into the VTI adapter which encapsulates the traffic. Traffic in response comes out of VTI and is decapsulated and routed back to PfSense:
    Ubuntu (172.21.12.2/28) --> PfSense (172.21.12.1/28)

    The IPSec tunnel is a whole seperate IP conversation:

    Ubuntu (172.21.12.2/28) <--> PfSense (172.21.12.1/28) <--> WAN <--> PfSense (10.0.1.1/28) <--> Ubuntu (10.0.1.2/28)

    In my opinion this is a valid routing path and no asymmetric routing is taking place. PfSense sends data to 172.21.12.2 and receives replies on the stream from the same IP over the same path. The tunnel traffic is sent from 172.21.12.2 to 172.21.12.1 and out to the internet, the reverse path is the same.

    For the testing with the seperate boxes in the /28 subnet I obviously created a static route for the remote subnet pointing to 172.21.12.2 as gateway, else the traffic would go from the client to pfsense to ubuntu but the reverse path would be different as ubuntu will reply straight back to the client because it is in the same subnet. I even doubt if a TCP conversation would even work.

    I have attached an image to clarify, thanks for the suggestion but this config should be valid from a routing perspective. In the image I have seperated the 2 conversations. Your image shows a reply magically breaking out of the tunnel, that packet is still encapsulated and the destination is not the PfSense. It will go the the Ubuntu box first. There it is decapsulated and routed further, in this case to PfSense. It cannot reply directly as the packet is inside a UDP/4500 ESP packet.

    I am now looking into the ESX offloading settings but I have not found a resolution yet. I think this is a problem in VMWare, not with the routing/tunnel config now.


  • Rebel Alliance Global Moderator

    "No I don't think it hairpins"

    So these connection are 2 different physical interfaces?  You talk to ubuntu box out one interface and the traffic comes back to pfsense on another interface?  If not then its a hairpin, and you can not reach full speed of the interface plain and simple.  This is bad design to start with.  While this sort of thing can work, its going to be a performance hit, and your doing the same thing on the ubuntu box, your going in its interface, just to come back out the same interface.  This is hairpinning.

    I missed the part in your OP about ubuntu creating a esp tunnel to the other ubuntu box. Sorry was just looking at picture.

    But to be honest what is the point of this? Why are you creating tunnels between your ubuntu machines - why not just create the tunnel/vpn between pfsense boxes?

    Where you say your windows machine is fast.. What interface is it talking to ubuntu on?  You say that is fast..  What is on the esxi box what is not, just pfsense is on the esxi host?



  • The perfomance should be more than sufficient, the WAN speed is 200Mbit/s at most, even a full duplex link can handle this easily. 2 NIC's is better but this should be fine. Both the Ubuntu machine and PfSense are VM's and the bandwidth is not limited by physical carrier, it can do multi gigabits only limited by the CPU as the packets between PfSense and Ubuntu do not traverse a physical NIC.

    Hairpinning is NAT terminology. What you are talking about is sending and receiving on the same wire, but even then a physical gigabit connection can handle 1 gbit/s in both directions simultaneously. I can send data to a ubuntu box at 1 gbit/s and the box can send it out at the same rate topping 2 gbit/s when you add in/out traffic together. Minus the bandwith required for ACK's. But again this is not a limitation as both machines are VM's topping at something like 20Gbit/s CPU limited  :P

    Reason 1 is:
    I want to use VTI tunnels as they are fully routable and work like a charm, I can use OSPF or other routing protocols instead of pesky IPSec policies. Unfortunately this is not yet available in PfSense.

    Reason 2 is :
    https://forum.pfsense.org/index.php?topic=99995.0

    Don't get me wrong, initially it looked like PfSense was causing an issue but I now think it lies in the VMWare environment. It did work fine before, for quite some time actually so this setup has proven to work fine hence I don't think the problem is caused by going back and forth over the same "wire".

    Your replies are appreciated, a bit of discussion can lead to deeper analysis of a problem  ;D I'll add a second NIC, add some static routes and see how that works out.


  • Rebel Alliance Global Moderator

    "even a full duplex link can handle this easily."

    Still bad design plan and simple…

    No hairpin is not a "NAT" term.. Yes you can hairpin with NAT, ie NAT "reflection".  The term hairpin means in and out same interface.. And it should be avoided if possible.  When you have multiple vlans on the same physical interface and vlan A talks to vlan B this is a hairpin, and not best for performance.  If possible if you have vlans that send a lot of traffic to each other, these vlans should be on different physical interfaces at the device making the routing decision.

    You say your windows machine is fast, the way you drew it - looks to be coming in different path than the interface you have your vti on?  Is that the case?  Again you state this is hosted on VM, what interface in vm are physical in the drawing what are virtual?

    This is esxi, where is your vmkern?  Same interface?