Very slow traffic from other VM's through pfSense on XenServer

[mortenchristensen]

Sorry.

Tried to disable hardware checksum offloading. The other 2 were disabled by default.

Did not improve the problem.

NIC's in the pfSense VM are nx0 to nx3

[mortenchristensen]

New test with a pfSense 2.1.
Here internet-traffic from other VM's on the same Xenserver is normal.

The problem seems to be new in pfSense 2.2.

[cmb]

2.1x wouldn't have xn NICs, it's specific to that. Can you force it to e1000 NICs on 2.2 and see?

[mortenchristensen]

2.1x wouldn't have xn NICs, it's specific to that. Can you force it to e1000 NICs on 2.2 and see?

On my 2.1.5 the nic's are called re. Can you give me some hints on, where abd how to change the driver ?

[phadm]

Hi,

i have the same problem with RC 2.2 (XenServer 6.2, SP1016, different platforms and nics) . The problem is the offload engine. If you route traffic between virtual hosts, you get tcp retransmissions, only a few sessions survive….

You have to disable the offload function at the VIF at the XenServer.
First identify the uuid of the VIF's:

xe vm-vif-list uuid=VMUUID

And disable the offload settings:

xe vif-param-set uuid=VIFUUID other-config:ethtool-gso="off"
xe vif-param-set uuid=VIFUUID other-config:ethtool-ufo="off"
xe vif-param-set uuid=VIFUUID other-config:ethtool-tso="off"
xe vif-param-set uuid=VIFUUID other-config:ethtool-sg="off"
xe vif-param-set uuid=VIFUUID other-config:ethtool-tx="off"
xe vif-param-set uuid=VIFUUID other-config:ethtool-rx="off"

shutdown / start the VM.

And now the disadvantage, whitout offload engine the TCP throughput falls on GBIT level over the vswitch. With offload I reach over 371 MBps with fetch, download the xencenter.iso from dom0 via http, whitout 98 MBps.

So who has a better solution, bring it on !!

[Derelict]

This all worked for me on the test stack I use which is now all 2.2-RELEASE.  I don't really care about performance much in this application, but before I did this it was useless.  Thanks much.

[mortenchristensen]

___First identify the uuid of the VIF's:
xe vm-vif-list uuid=VMUUID

And disable the offload settings:
xe vif-param-set uuid=VIFUUID other-config:ethtool-gso="off"
xe vif-param-set uuid=VIFUUID other-config:ethtool-ufo="off"
xe vif-param-set uuid=VIFUUID other-config:ethtool-tso="off"
xe vif-param-set uuid=VIFUUID other-config:ethtool-sg="off"
xe vif-param-set uuid=VIFUUID other-config:ethtool-tx="off"
xe vif-param-set uuid=VIFUUID other-config:ethtool-rx="off"

shutdown / start the VM___

Used this on both a XenServer 6.5 and a 6.2 later upgraded to 6.5. On both it has given other VM's internet-access again.

Run the xe commands on a Xenserver Private Network, so I hope the speed degrade will only occur on traffic that involves that net.
I think, both the pfSense VM and the other VM's need to be restartet to get useful speed.

[apollo13]

@phadm:

You have to disable the offload function at the VIF at the XenServer.
First identify the uuid of the VIF's:

Which VIF? Local or WAN or both?

Thanks,
Florian

[Derelict]

I did it on all.

[jpenninkhof]

This helped me too. I only did this for my LAN port.

In my setup it seemed to be sufficient to execute:
xe vif-param-set uuid=VIFUUID other-config:ethtool-tx="off"
xe vif-param-set uuid=VIFUUID other-config:ethtool-rx="off"

[apollo13]

@jpenninkhof:

This helped me too. I only did this for my LAN port.

In my setup it seemed to be sufficient to execute:
xe vif-param-set uuid=VIFUUID other-config:ethtool-tx="off"
xe vif-param-set uuid=VIFUUID other-config:ethtool-rx="off"

I can confirm that the LAN port should be enough. On a related note, did someone install the XenServer Tools in the VM?

[corotte]

Hi,

updated my XenServer 6.2 to 6.5 a few day ago with my VM pfsense 2.1.5 with no issue

updated pfsense to 2.2 WITH XENTOOLS (xe-guest-utilties 6.0.2_3) and got the same issue !

installed xentool using that method http://blog.feld.me/posts/2014/07/pfsense-on-citrix-xenserver/ (Thanks feld !)

look like issue remain even with Xentools :/

anyone can confirm ?

[Derelict]

Yes.  It's broken.

[corotte]

damn !

but a quesiton remain … was it working well in snapshot ? was it working well with previous version of xentool ?

in this thread
https://forum.pfsense.org/index.php?topic=86827.0
it look like to be an issue with xn nic …
maybe a previous version should work ?

[Derelict]

No.

Just disable the tx/rx like in the above until FreeBSD and/or Citrix fixes it.

[corotte]

Ok

didi the above fix and it finally work.

Thanks folks !

[dsiminiuk]

My Internet speed normally is 20 Mb/s down and 2 Mb/s up.

I deployed pfSense 2.2-RELEASE X64 in XenServer 6.5

Without modification, the pfSense 2.2 would only muster 5 Mb/s down, and 0.06 Mb/s up. Painful.

I applied the changes to the LAN side VIF and the upload speed went back to full 2 Mb/s. The WAN speed did not improve.

I applied the changes to the WAN side VIF and the upload speed went back up to 20 Mb/s.

Eureka!

[Andy_]

It's just the tx-offload setting that needs to be changed, rx-offload is fixed-on.

I can confirm the problem and fix with Debian Wheezy/Xen 4.1.4 dom0.

ethtool -K ${dev} tx off in vif-bridge online did the trick.

The issue wasn't submitted to freebsd-bugs so far, now it is:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=197344

[A Former User]

Interesting - only appears to apply to virtual interfaces.

My pfSense VM is running in xen 4.2 (Centos 6.6 dom0) and has no speed issues, but I'm using pci-passthrough to give 2 dedicated hardware NICs (off a dual-port Intel card) to pfSense for LAN/WAN  (so that DMZ/intranet are physically separate too).

[bananaboy]

Thanks johnkeats for putting that up here. It really helped me sort this out.

One thing to note is disabling tx offload using ethtool -K does not persist across guest reboots or live migration because the dom-id and assigned vif changes, while xe vif-param-set other-config:ethtool-tx="off" does.

Is there any downside to using the vif-param-set option, or are the two basically equivalent?