Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Very slow traffic from other VM's through pfSense on XenServer

    Scheduled Pinned Locked Moved Virtualization
    47 Posts 23 Posters 57.0k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • M
      mortenchristensen
      last edited by

      I have 2 XenServers, one with XenServer 6.2 and one with Xenserver Creedence beta 3.

      Both have a pfSense 2.2 RC as router/firewall and a couple of Ubuntu Linux VM's and a windows-VM.

      Traffic through both the physical xenserver-box and the virtual pfSense firewall goes at expected speeds.

      But traffic from the other VM's on the same xenserver through the pfSense out on wan/internet goes very, very slow.
      It goes so bad they cannot update themselve with apt-get.

      When I try with iperf from a linux VM through the pfSense's WAN the speed is 3,82 KBits/sec.
      The VM's and pfSense are connected with an internal single-server network (as OPT1), and tests to iperf server run on pfSense from a linux VM shows gigabit-speed.

      One of the pfSense' has xen-tools installed. The other has not. I cannot se improvements with the tools installed.

      One of the XenServers can get several public IP'numbers. On that I now have installed VM's with both an IPCop firewall and a Zentyal firewall.
      When one of those new firewall-VMs' is default gateway for the ordinary VM's on the XenServer, their wan/internet-speed is normal.

      Anobody with experience on XenServer as hypervisor, that can give me in a direction to experiment in to get traffic from VM's on the same Xenserver through pfSense up at useful performance ?

      1 Reply Last reply Reply Quote 0
      • C
        cmb
        last edited by

        Try disabling hardware checksum offloading under System>Advanced, Networking. TSO and LRO should also be disabled, though they likely already are since that's the default for those.

        Which type of NIC is showing up in the VM? re0, em0, xn0?

        1 Reply Last reply Reply Quote 0
        • M
          mortenchristensen
          last edited by

          Sorry.

          Tried to disable hardware checksum offloading. The other 2 were disabled by default.

          Did not improve the problem.

          NIC's in the pfSense VM are nx0 to nx3

          1 Reply Last reply Reply Quote 0
          • M
            mortenchristensen
            last edited by

            New test with a pfSense 2.1.
            Here internet-traffic from other VM's on the same Xenserver is normal.

            The problem seems to be new in pfSense 2.2.

            1 Reply Last reply Reply Quote 0
            • C
              cmb
              last edited by

              2.1x wouldn't have xn NICs, it's specific to that. Can you force it to e1000 NICs on 2.2 and see?

              1 Reply Last reply Reply Quote 0
              • M
                mortenchristensen
                last edited by

                2.1x wouldn't have xn NICs, it's specific to that. Can you force it to e1000 NICs on 2.2 and see?

                On my 2.1.5 the nic's are called re. Can you give me some hints on, where abd how to change the driver ?

                1 Reply Last reply Reply Quote 0
                • P
                  phadm
                  last edited by

                  Hi,

                  i have the same problem with RC 2.2 (XenServer 6.2, SP1016, different platforms and nics) . The problem is the offload engine. If you route traffic between virtual hosts, you get tcp retransmissions, only a few sessions survive….

                  You have to disable the offload function at the VIF at the XenServer.
                  First identify the uuid of the VIF's:

                  xe vm-vif-list uuid=VMUUID

                  And disable the offload settings:

                  xe vif-param-set uuid=VIFUUID other-config:ethtool-gso="off"
                  xe vif-param-set uuid=VIFUUID other-config:ethtool-ufo="off"
                  xe vif-param-set uuid=VIFUUID other-config:ethtool-tso="off"
                  xe vif-param-set uuid=VIFUUID other-config:ethtool-sg="off"
                  xe vif-param-set uuid=VIFUUID other-config:ethtool-tx="off"
                  xe vif-param-set uuid=VIFUUID other-config:ethtool-rx="off"

                  shutdown / start the VM.

                  And now the disadvantage, whitout offload engine the TCP throughput falls on GBIT level over the vswitch. With offload I reach over 371 MBps with fetch, download the xencenter.iso from dom0 via http, whitout 98 MBps.

                  So who has a better solution, bring it on !!

                  1 Reply Last reply Reply Quote 0
                  • DerelictD
                    Derelict LAYER 8 Netgate
                    last edited by

                    This all worked for me on the test stack I use which is now all 2.2-RELEASE.  I don't really care about performance much in this application, but before I did this it was useless.  Thanks much.

                    Chattanooga, Tennessee, USA
                    A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                    DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                    Do Not Chat For Help! NO_WAN_EGRESS(TM)

                    1 Reply Last reply Reply Quote 0
                    • M
                      mortenchristensen
                      last edited by

                      ___First identify the uuid of the VIF's:
                      xe vm-vif-list uuid=VMUUID

                      And disable the offload settings:
                      xe vif-param-set uuid=VIFUUID other-config:ethtool-gso="off"
                      xe vif-param-set uuid=VIFUUID other-config:ethtool-ufo="off"
                      xe vif-param-set uuid=VIFUUID other-config:ethtool-tso="off"
                      xe vif-param-set uuid=VIFUUID other-config:ethtool-sg="off"
                      xe vif-param-set uuid=VIFUUID other-config:ethtool-tx="off"
                      xe vif-param-set uuid=VIFUUID other-config:ethtool-rx="off"

                      shutdown / start the VM___

                      Used this on both a XenServer 6.5 and a 6.2 later upgraded to 6.5. On both it has given other VM's internet-access again.

                      Run the xe commands on a Xenserver Private Network, so I hope the speed degrade will only occur on traffic that involves that net.
                      I think, both the pfSense VM and the other VM's need to be restartet to get useful speed.

                      1 Reply Last reply Reply Quote 0
                      • A
                        apollo13
                        last edited by

                        @phadm:

                        You have to disable the offload function at the VIF at the XenServer.
                        First identify the uuid of the VIF's:

                        Which VIF? Local or WAN or both?

                        Thanks,
                        Florian

                        1 Reply Last reply Reply Quote 0
                        • DerelictD
                          Derelict LAYER 8 Netgate
                          last edited by

                          I did it on all.

                          Chattanooga, Tennessee, USA
                          A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                          DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                          Do Not Chat For Help! NO_WAN_EGRESS(TM)

                          1 Reply Last reply Reply Quote 0
                          • J
                            jpenninkhof
                            last edited by

                            This helped me too. I only did this for my LAN port.

                            In my setup it seemed to be sufficient to execute:
                            xe vif-param-set uuid=VIFUUID other-config:ethtool-tx="off"
                            xe vif-param-set uuid=VIFUUID other-config:ethtool-rx="off"

                            1 Reply Last reply Reply Quote 0
                            • A
                              apollo13
                              last edited by

                              @jpenninkhof:

                              This helped me too. I only did this for my LAN port.

                              In my setup it seemed to be sufficient to execute:
                              xe vif-param-set uuid=VIFUUID other-config:ethtool-tx="off"
                              xe vif-param-set uuid=VIFUUID other-config:ethtool-rx="off"

                              I can confirm that the LAN port should be enough. On a related note, did someone install the XenServer Tools in the VM?

                              1 Reply Last reply Reply Quote 0
                              • C
                                corotte
                                last edited by

                                Hi,

                                updated my XenServer 6.2 to 6.5 a few day ago with my VM pfsense 2.1.5 with no issue

                                updated pfsense to 2.2 WITH XENTOOLS (xe-guest-utilties 6.0.2_3) and got the same issue !

                                installed xentool using that method http://blog.feld.me/posts/2014/07/pfsense-on-citrix-xenserver/ (Thanks feld !)

                                look like issue remain even with Xentools :/

                                anyone can confirm ?

                                StefanAlbrechtS 1 Reply Last reply Reply Quote 0
                                • DerelictD
                                  Derelict LAYER 8 Netgate
                                  last edited by

                                  Yes.  It's broken.

                                  Chattanooga, Tennessee, USA
                                  A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                                  DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                                  Do Not Chat For Help! NO_WAN_EGRESS(TM)

                                  1 Reply Last reply Reply Quote 0
                                  • C
                                    corotte
                                    last edited by

                                    damn !

                                    but a quesiton remain … was it working well in snapshot ? was it working well with previous version of xentool ?

                                    in this thread
                                    https://forum.pfsense.org/index.php?topic=86827.0
                                    it look like to be an issue with xn nic …
                                    maybe a previous version should work ?

                                    1 Reply Last reply Reply Quote 0
                                    • DerelictD
                                      Derelict LAYER 8 Netgate
                                      last edited by

                                      No.

                                      Just disable the tx/rx like in the above until FreeBSD and/or Citrix fixes it.

                                      Chattanooga, Tennessee, USA
                                      A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                                      DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                                      Do Not Chat For Help! NO_WAN_EGRESS(TM)

                                      1 Reply Last reply Reply Quote 0
                                      • C
                                        corotte
                                        last edited by

                                        Ok

                                        didi the above fix and it finally work.

                                        Thanks folks !

                                        1 Reply Last reply Reply Quote 0
                                        • dsiminiukD
                                          dsiminiuk
                                          last edited by

                                          My Internet speed normally is 20 Mb/s down and 2 Mb/s up.

                                          I deployed pfSense 2.2-RELEASE X64 in XenServer 6.5

                                          Without modification, the pfSense 2.2 would only muster 5 Mb/s down, and 0.06 Mb/s up. Painful.

                                          I applied the changes to the LAN side VIF and the upload speed went back to full 2 Mb/s. The WAN speed did not improve.

                                          I applied the changes to the WAN side VIF and the upload speed went back up to 20 Mb/s.

                                          Eureka!

                                          1 Reply Last reply Reply Quote 0
                                          • A
                                            Andy_
                                            last edited by

                                            It's just the tx-offload setting that needs to be changed, rx-offload is fixed-on.

                                            I can confirm the problem and fix with Debian Wheezy/Xen 4.1.4 dom0.

                                            ethtool -K ${dev} tx off in vif-bridge online did the trick.

                                            The issue wasn't submitted to freebsd-bugs so far, now it is:
                                            https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=197344

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.