Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Very slow traffic from other VM's through pfSense on XenServer

    Scheduled Pinned Locked Moved
    Virtualization
    23
    47
    55.3k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • P
      phadm
      last edited by

      Hi,

      i have the same problem with RC 2.2 (XenServer 6.2, SP1016, different platforms and nics) . The problem is the offload engine. If you route traffic between virtual hosts, you get tcp retransmissions, only a few sessions survive….

      You have to disable the offload function at the VIF at the XenServer.
      First identify the uuid of the VIF's:

      xe vm-vif-list uuid=VMUUID

      And disable the offload settings:

      xe vif-param-set uuid=VIFUUID other-config:ethtool-gso="off"
      xe vif-param-set uuid=VIFUUID other-config:ethtool-ufo="off"
      xe vif-param-set uuid=VIFUUID other-config:ethtool-tso="off"
      xe vif-param-set uuid=VIFUUID other-config:ethtool-sg="off"
      xe vif-param-set uuid=VIFUUID other-config:ethtool-tx="off"
      xe vif-param-set uuid=VIFUUID other-config:ethtool-rx="off"

      shutdown / start the VM.

      And now the disadvantage, whitout offload engine the TCP throughput falls on GBIT level over the vswitch. With offload I reach over 371 MBps with fetch, download the xencenter.iso from dom0 via http, whitout 98 MBps.

      So who has a better solution, bring it on !!

      1 Reply Last reply Reply Quote 0
      • DerelictD
        Derelict LAYER 8 Netgate
        last edited by

        This all worked for me on the test stack I use which is now all 2.2-RELEASE.  I don't really care about performance much in this application, but before I did this it was useless.  Thanks much.

        Chattanooga, Tennessee, USA
        A comprehensive network diagram is worth 10,000 words and 15 conference calls.
        DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
        Do Not Chat For Help! NO_WAN_EGRESS(TM)

        1 Reply Last reply Reply Quote 0
        • M
          mortenchristensen
          last edited by

          ___First identify the uuid of the VIF's:
          xe vm-vif-list uuid=VMUUID

          And disable the offload settings:
          xe vif-param-set uuid=VIFUUID other-config:ethtool-gso="off"
          xe vif-param-set uuid=VIFUUID other-config:ethtool-ufo="off"
          xe vif-param-set uuid=VIFUUID other-config:ethtool-tso="off"
          xe vif-param-set uuid=VIFUUID other-config:ethtool-sg="off"
          xe vif-param-set uuid=VIFUUID other-config:ethtool-tx="off"
          xe vif-param-set uuid=VIFUUID other-config:ethtool-rx="off"

          shutdown / start the VM___

          Used this on both a XenServer 6.5 and a 6.2 later upgraded to 6.5. On both it has given other VM's internet-access again.

          Run the xe commands on a Xenserver Private Network, so I hope the speed degrade will only occur on traffic that involves that net.
          I think, both the pfSense VM and the other VM's need to be restartet to get useful speed.

          1 Reply Last reply Reply Quote 0
          • A
            apollo13
            last edited by

            @phadm:

            You have to disable the offload function at the VIF at the XenServer.
            First identify the uuid of the VIF's:

            Which VIF? Local or WAN or both?

            Thanks,
            Florian

            1 Reply Last reply Reply Quote 0
            • DerelictD
              Derelict LAYER 8 Netgate
              last edited by

              I did it on all.

              Chattanooga, Tennessee, USA
              A comprehensive network diagram is worth 10,000 words and 15 conference calls.
              DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
              Do Not Chat For Help! NO_WAN_EGRESS(TM)

              1 Reply Last reply Reply Quote 0
              • J
                jpenninkhof
                last edited by

                This helped me too. I only did this for my LAN port.

                In my setup it seemed to be sufficient to execute:
                xe vif-param-set uuid=VIFUUID other-config:ethtool-tx="off"
                xe vif-param-set uuid=VIFUUID other-config:ethtool-rx="off"

                1 Reply Last reply Reply Quote 0
                • A
                  apollo13
                  last edited by

                  @jpenninkhof:

                  This helped me too. I only did this for my LAN port.

                  In my setup it seemed to be sufficient to execute:
                  xe vif-param-set uuid=VIFUUID other-config:ethtool-tx="off"
                  xe vif-param-set uuid=VIFUUID other-config:ethtool-rx="off"

                  I can confirm that the LAN port should be enough. On a related note, did someone install the XenServer Tools in the VM?

                  1 Reply Last reply Reply Quote 0
                  • C
                    corotte
                    last edited by

                    Hi,

                    updated my XenServer 6.2 to 6.5 a few day ago with my VM pfsense 2.1.5 with no issue

                    updated pfsense to 2.2 WITH XENTOOLS (xe-guest-utilties 6.0.2_3) and got the same issue !

                    installed xentool using that method http://blog.feld.me/posts/2014/07/pfsense-on-citrix-xenserver/ (Thanks feld !)

                    look like issue remain even with Xentools :/

                    anyone can confirm ?

                    StefanAlbrechtS 1 Reply Last reply Reply Quote 0
                    • DerelictD
                      Derelict LAYER 8 Netgate
                      last edited by

                      Yes.  It's broken.

                      Chattanooga, Tennessee, USA
                      A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                      DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                      Do Not Chat For Help! NO_WAN_EGRESS(TM)

                      1 Reply Last reply Reply Quote 0
                      • C
                        corotte
                        last edited by

                        damn !

                        but a quesiton remain … was it working well in snapshot ? was it working well with previous version of xentool ?

                        in this thread
                        https://forum.pfsense.org/index.php?topic=86827.0
                        it look like to be an issue with xn nic …
                        maybe a previous version should work ?

                        1 Reply Last reply Reply Quote 0
                        • DerelictD
                          Derelict LAYER 8 Netgate
                          last edited by

                          No.

                          Just disable the tx/rx like in the above until FreeBSD and/or Citrix fixes it.

                          Chattanooga, Tennessee, USA
                          A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                          DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                          Do Not Chat For Help! NO_WAN_EGRESS(TM)

                          1 Reply Last reply Reply Quote 0
                          • C
                            corotte
                            last edited by

                            Ok

                            didi the above fix and it finally work.

                            Thanks folks !

                            1 Reply Last reply Reply Quote 0
                            • dsiminiukD
                              dsiminiuk
                              last edited by

                              My Internet speed normally is 20 Mb/s down and 2 Mb/s up.

                              I deployed pfSense 2.2-RELEASE X64 in XenServer 6.5

                              Without modification, the pfSense 2.2 would only muster 5 Mb/s down, and 0.06 Mb/s up. Painful.

                              I applied the changes to the LAN side VIF and the upload speed went back to full 2 Mb/s. The WAN speed did not improve.

                              I applied the changes to the WAN side VIF and the upload speed went back up to 20 Mb/s.

                              Eureka!

                              1 Reply Last reply Reply Quote 0
                              • A
                                Andy_
                                last edited by

                                It's just the tx-offload setting that needs to be changed, rx-offload is fixed-on.

                                I can confirm the problem and fix with Debian Wheezy/Xen 4.1.4 dom0.

                                ethtool -K ${dev} tx off in vif-bridge online did the trick.

                                The issue wasn't submitted to freebsd-bugs so far, now it is:
                                https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=197344

                                1 Reply Last reply Reply Quote 0
                                • ?
                                  A Former User
                                  last edited by

                                  Interesting - only appears to apply to virtual interfaces.

                                  My pfSense VM is running in xen 4.2 (Centos 6.6 dom0) and has no speed issues, but I'm using pci-passthrough to give 2 dedicated hardware NICs (off a dual-port Intel card) to pfSense for LAN/WAN  (so that DMZ/intranet are physically separate too).

                                  1 Reply Last reply Reply Quote 0
                                  • B
                                    bananaboy
                                    last edited by

                                    Thanks johnkeats for putting that up here. It really helped me sort this out.

                                    One thing to note is disabling tx offload using ethtool -K does not persist across guest reboots or live migration because the dom-id and assigned vif changes, while xe vif-param-set other-config:ethtool-tx="off" does.

                                    Is there any downside to using the vif-param-set option, or are the two basically equivalent?

                                    1 Reply Last reply Reply Quote 0
                                    • B
                                      bennymundz
                                      last edited by

                                      @johnkeates:

                                      You only need to disable checksum offloading on the hypervisor side of pfSense's interface.

                                      Any interface that does DomU-DomU communication on pfSense's side produces un-checksummed packets which get dropped by PF in BSD.

                                      sudo ethtool -K $interface tx off

                                      where $interface is the VIF on the Xen Dom0 side is enough. Setting TX off on the bridge forces the Dom0 to calculate ALL checksums on ALL packets no matter where the come from or where they are going. This is not a smart idea since it creates a lot of calculations where they might not be needed. So if the pfSense DomU is on vif123.0 you run: sudo ethtool -K vif123.0 tx off

                                      Sorry noob question here,

                                      I am using a Xen implementation on a unraid distribution, when you say Dom0 side are you talking about the VIF that is spun up with the PFsense VM ? Like when i ifconfig to list my interfaces I just don't really know how to identify the interface you are referring to.

                                      Sorry for the noob question again

                                      1 Reply Last reply Reply Quote 0
                                      • DerelictD
                                        Derelict LAYER 8 Netgate
                                        last edited by

                                        It's all here:

                                        https://forum.pfsense.org/index.php?topic=85797.msg475906#msg475906

                                        I recently just rebuilt my test stack and all I did was the tx and rx on every NIC which is still probably more than is necessary but it worked.

                                        Chattanooga, Tennessee, USA
                                        A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                                        DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                                        Do Not Chat For Help! NO_WAN_EGRESS(TM)

                                        1 Reply Last reply Reply Quote 0
                                        • B
                                          bennymundz
                                          last edited by

                                          @johnkeates:

                                          You only need to disable checksum offloading on the hypervisor side of pfSense's interface.

                                          Any interface that does DomU-DomU communication on pfSense's side produces un-checksummed packets which get dropped by PF in BSD.

                                          sudo ethtool -K $interface tx off

                                          where $interface is the VIF on the Xen Dom0 side is enough. Setting TX off on the bridge forces the Dom0 to calculate ALL checksums on ALL packets no matter where the come from or where they are going. This is not a smart idea since it creates a lot of calculations where they might not be needed. So if the pfSense DomU is on vif123.0 you run: sudo ethtool -K vif123.0 tx off

                                          Thank you for taking the time to explain this, i turned the TX off on the pfsense vif and all was good. Happy days

                                          1 Reply Last reply Reply Quote 0
                                          • B
                                            BBMitch
                                            last edited by

                                            Hello all…

                                            Thanks for the information - sure helped us solve this but I have some more information that wasn't clear to me from all posted here.

                                            This issue only seems to apply where Pf is communicating with hosts within the same xen host (dom0).

                                            We use xenserver 6.2 fwiw. We have two xen dom0 - pf was natting for two services - one on dom0-a and one on dom0-b

                                            pf itself was located on dom0-b
                                            The dom0-a service worked perfectly after the update to 2.2.2 - the dom0-b service did not.

                                            For people new to xenserver / for completeness, we used:
                                            xe vm-list
                                            #then find the uuid of your pf vm
                                            xe vif-list vm-uuid={uuid of the vm from above}
                                            #note the uuid of the vif - not the network you want to change!
                                            #for each vif you can check the status:
                                            xe vif-param-get uuid={uuid of vif} param-name=other-config
                                            xe vif-param-set uuid={uuid of vif} other-config:ethtool-tx="off"

                                            For what it's worth I was able to turn off tx on only the LAN interface (which nats for the dom0-b service).

                                            I tried but did not need to keep offload off for the WAN interface which seems to get proper checksum as it leaves the dom0 through the physical nic.

                                            Once complete you need to reboot the pf vm. the setting will persist across reboots.

                                            Hope that helps someone else :-)

                                            Mitch

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post