XG-7100 efficiency low?



  • I try to use multiple pipes with speed of 1gig. It works fine on a single port and single 1gig but once I use more eg. additional gig to another port and test using multiple laptop off multiple ports(each port gets different gateway/pipe) I can go over 1gig total, why? Isnt this model suppose to go close to 5gig total on all ports?

    Enable PowerD to maximum



  • Hi,

    Possible issue : check out this question https://forum.netgate.com/topic/144915/what-is-wrong-with-my-pfsense-limiters and you'll get the picture ;)



  • The thing is, I don't use limiters or shapers on those interfaces I test



  • It would be really bad if that unit is able to do only 1gig as oppose to min 2.5gig x2.
    I bought few units XG-7100 and tons of SG-4860 just to discover this now?



  • It's my understanding that the backplane of the switch supports up to 5Gbps (2x 2.5Gbps) so with large amounts of traffic on the switch ports, it doesn't get overloaded. I'm also assuming that if one of the fiber ports is used as a WAN port, the high-speed backplane on the switch itself is easily able to talk to the fiber ports at full speed.

    So, I believe the only way you are going to get higher than 1Gbps on a single port on the XG-7100 is to use the SFP+ ports. You could also add an additional card, with high speed fiber ports, into the expansion slot. I would consult Netgate, however, on what the best approach is to doing it this way.

    The SG-4860, by the way, will never support faster than 1Gbps on any port.

    Jeff



  • I am not talking about speed from one port but 2 diffrenet LANs and 2 diffrenet WANs if you read my first post you will get picture.



  • @mke said in XG-7100 efficiency low?:

    I am not talking about speed from one port but 2 diffrenet LANs and 2 diffrenet WANs if you read my first post you will get picture.

    Um no, your first post wasn't that clear.

    You now say 2 different LANs and 2 different WANs. What ports on the XG-7100 are you using for the 2 LANs and what ports for the 2 WANs? To get anything faster than 1Gbps, I believe you have to use the SFP+ ports. If you're not, and you are simply using the built-in switch ports, you only get 1Gbps to each port. Even if you bond (LAGG) ports together, you don't get any faster than 1Gbps. You simply get redundancy, not additional speeds.

    Jeff



  • Let me give you more clarification. I don't use any SFP and dont need to test anything more that a single gig off single port.

    I use only all standard copper ports

    1. port1 - wan 1 - isp1 1gbs
    2. port2 - wan 2 - isp2 1gbs
    3. port3 - lan1 with a gateway pointing to ISP1
    4. port4 - lan2 with a gateway pointing to ISP2

    laptop 1 connected to lan1 port3
    laptop 2 connected to lan2 port4

    Now I do speed test on two laptops at the same time trying to see what is the throughput capability of xg7100. single laptop of single port is gig, fine, but once you connect the second trying to engage second pipe I am getting not more that gig total looking at two screens of 2 laptops.


  • LAYER 8 Global Moderator

    So your laptops are only see 500mbps vs 1g when you run both..

    But when you run 1 you see 1g, you run 2 you see 1g, run both at same time you only see 500mbps each is what your saying.

    You sure your routing them correctly via your 2 isps?



  • Yep LAN-1 goes via ISP1 , LAN-2 goest via ISP2 there is not much room for mistake here when choosing the gateway, the result is less or more as you said, laptops are fighting for what looks like total 1gig combined on both sometimes 300 vs 700 sometimes 500 vs 500. Which is really weird and inefficient. PowerD set to max, No heavy NAT, no limiters on tested networks. I do have squid but it is disabled.


  • LAYER 8 Global Moderator

    And these are 2 completely different isp.. Or you connected 2 different devices to the same?

    What your hinting at is the routing can only do 1 gig total... Or that the backplain of the switch can only move 1 gig total? Which makes no sense.

    You don't have ports 3 and 4 bridged do you?

    So lan 1 is like 192.168.1/24, lan 2 is is 192.168.2/24 for example

    And you have 2 different isp giving you 2 different public IPs completely different from each other.

    So you validated via states that the traffic is actually going out 2 different wans?


  • Netgate Administrator

    You have the switch internal interfaces in the default load-balance lagg mode?

    Steve



  • @johnpoz

    Correct 2 completely different ISPs. Correct sth is wrong indicating total throughput problem over gig. No bridging. Correct WANs independent separate subnets, LANs also on separate subnets tested directly off untagged ports on xg7100. Before I did test I made sure that my public facing IP is different on each connection.

    @stephenw10
    Each tested vlan on lan and wan side is working with default lagg0


  • LAYER 8 Global Moderator

    @mke said in XG-7100 efficiency low?:

    Before I did test I made sure that my public facing IP is different on each connection.

    And you validated that your routing is working as you assume.. Via checking the state tables and traffic flow over both wan side interfaces..

    Something is off that is for sure - the thing is way more capable of just 1 gig ;)

    Now that @stephenw10 sure can get to the bottom of the issue.. I don't have a XG7100 to play with :( or would be happy to duplicate your testing...



  • I did not checked state tables, only did "what is my public IP" check in the browser.
    I have opened ticket with support, sent them status dump for my unit. Waiting for the issue to be potentially replicated, let's see they said it may be days.


  • LAYER 8 Global Moderator

    Support never shares anything about any specific issues that get moved to support tickets, even when the thread was started in the forum..

    So hope you will share what the issue is, when figure it out. Best I could do would be to duplicate the setup you are using for testing with my sg4860.. But that wouldn't be much help, since doesn't have any switch ports.



  • Will share info whenever they come with some answer.


  • LAYER 8 Global Moderator

    Great.. Got to be something stupid ;) I don't think it will be "days" either..


  • LAYER 8 Rebel Alliance

    Is your problem fixed?

    -Rico


  • LAYER 8 Global Moderator

    I had heard a snip from Chris that they were about to discuss with the ticket opener.. But that is all I got out of him ;) Hope the OP comes back and let us know some info about this myself.


  • Netgate Administrator

    I setup a lab to test this. The LAGG appears to be working very well.

    Lab Details:
    HOST-1 (SG-5100)
    DUT (XG-7100)
    HOST-2 (SG-5100)

    HOST-1
    IX0 -> LAGG0.4091 / PORT 1 (ETH1)
    IX1 -> LAGG0.4090 / PORT 2 (ETH2)
    IX2 -> LAGG0.4091 / PORT 3 (ETH3)
    IX3 -> LAGG0.4090 / PORT 4 (ETH4)

    HOST-2:
    IX0 -> LAGG0.3091 / PORT 5 (ETH5)
    IX1 -> LAGG0.3090 / PORT 6 (ETH6)
    IX2 -> LAGG0.3091 / PORT 7 (ETH7)
    IX3 -> LAGG0.3090 / PORT 8 (ETH8)

    DUT-NETWORK
    LAGG0.4090 = WAN_1
    LAGG0.4091 = LAN_1
    LAGG0.3090 = WAN_2
    LAGG0.3091 = LAN_2

    Each WAN and LAN has two IPs assigned from two different networks - for a total of 4 WAN IPs and 4 LAN IPs.

    LAN-1 = 1.1.1.1/30, 3.3.3.1/30

    • 16.0.0.0/18 -> 1.1.1.2 (HOST-1.IX0)
    • 16.0.64.0/18 -> 3.3.3.2 (HOST-1.IX2)

    LAN-2 = 5.5.5.1/30, 7.7.7.1/30

    • 16.0.128.0/18 -> 5.5.5.2 (HOST-2.IX0)
    • 16.0.192.0/18 -> 7.7.7.2 (HOST-2.IX2)

    WAN-1 = 2.2.2.1/30, 4.4.4.1/30

    • 48.0.0.0/18 -> 2.2.2.2 (HOST-1.IX1)
    • 48.0.64.0/18 -> 4.4.4.2 (HOST-1.IX3)

    WAN-2 = 6.6.6.1/30, 8.8.8.1/30

    • 48.0.128.0/18 -> 6.6.6.2 (HOST-1.IX2)
    • 48.0.192.0/18 -> 8.8.8.2 (HOST-1.IX4)

    For UDP traffic, I used the latest trex build to generate traffic.
    For TCP traffic, I used the latest iperf3 build to generate traffic.

    Sending 1500 byte UDP packets, I consistently get around 4.8 Gbps (highest was around 4.86 Gbps / 405 Kpps).
    Sending TCP packets with iperf over 1500 MTU, I get close to 4 Gbps.

    The results of each were the same under the following scenarios (TCP performed a little better with PF disabled):
    PF disabled, PF enabled, NAT disabled, NAT enabled, static routes to local WAN, policy routes to external WAN.

    In all scenarios, the results line up for both unidirectional and bidirectional (in the case of bidirectional, the same result as unidirectional but the result applies to both RX and TX for each ethernet switched interface).


  • LAYER 8 Global Moderator

    Yeah those numbers seem what you would think.. So the question now is helping the OP figure out what is going on in his testing.. Prob have to prove to him that something is not wrong with his hardware..

    Or what could be in his config that could be causing the problem.


  • LAYER 8 Netgate

    I cannot duplicate @mke's findings:

    Clients sending (uploading)

    Simultaneous iperf3 -c 172.18.208.1 -P4 -t60 and iperf3 -c 172.18.209.1 -P4 -t60

    XG-2758 igb1 <-> XG7100 lagg0.4082 <-> lagg0.4083 <-> MacBook Pro 882,881,928 (897Mb/sec)
    XG-2758 igb2 <-> XG7100 lagg0.4084 <-> lagg0.4085 <-> Proxmox VM 908,876,895 (893Mb/sec)

    Servers sending (downloading)

    Simultaneous iperf3 -R -c 172.18.208.1 -P4 -t60 and iperf3 -R -c 172.18.209.1 -P4 -t60

    XG-2758 igb1 <-> XG7100 lagg0.4082 <-> lagg0.4083 <-> MacBook Pro 924,924,926 (925Mb/sec)
    XG-2758 igb2 <-> XG7100 lagg0.4084 <-> lagg0.4085 <-> Proxmox VM 931,932,926 (930Mb/sec)

    MacBook downloading, VM uploading

    Simultaneous iperf3 -R -c 172.18.208.1 -P4 -t60 and iperf3 -c 172.18.209.1 -P4 -t60

    XG-2758 igb1 <-> XG7100 lagg0.4082 <-> lagg0.4083 <-> MacBook Pro 925,878,899 (901Mb/sec)
    XG-2758 igb2 <-> XG7100 lagg0.4084 <-> lagg0.4085 <-> Proxmox VM 860,917,903 (893Mb/sec)

    iperf3 servers running on same XG-2758. This is far from a perfect test environment but it is sufficient to duplicate what is being asserted and I was not successful in doing so.



  • They are still investigating this but so far cannot replicate the problem. Since I have more than one XG-7100 I did more testing but not with iperf but real pipes at two different locations, result was the same(struggling to go over 1gig) and I even did video on this and sent them but can't post since it shows my IPs and I don't have time to do editing.


  • LAYER 8 Global Moderator

    So can you duplicate their iperf testing?



  • What I would love to do is to do iperf across real links with multiple sites with mix of xg7100 and SG-8860 all with gig pipes, not sure if will be able and right now I have very limited time.


  • LAYER 8 Global Moderator

    do you have any of the switch ports still open - you could use those without disruption of your active links.



  • I don't have access to those devices since they are in different locations so physically it is problematic to do testing right now.



  • I got finally an answer after multiple test using multiple XG7100, different routers, pipes, core switches, combinations. It ended up that testing in the real world using same website is misleading. I did try different speed tests but wanted to use that same particular on both laptops because for higher speeds it gave me very good results(single laptop) that reflected situation however while testing multiple computers it showed bottleneck on their side, sick.


  • LAYER 8 Netgate

    Thank you for coming back and reporting your findings.

    That is why one should not rely on external test sites when testing device performance.


  • LAYER 8 Global Moderator

    @Derelict said in XG-7100 efficiency low?:

    That is why one should not rely on external test sites when testing device performance.

    You don't say ;) heheheeh


Log in to reply