Issue XCP-ng routed config / pfSense - slow speed & packetloss

[maverickws]

Hi all,

I have a networking related question hope you guys can help me.
My setup is the following:

• two identical hypervisors at Hetzner,
• 1 Realtek RTL8111/8168/8411 pci-e gigabit ethernet each
• both servers have a /29 network aside from their main IP
• routed setup, ip_forwarding enabled
• pfsense router vm;
• vlans configured on the xcp-ng pool mtu 1400
• all of pfsense's vif have ethtool-tx="off"

the pfsense router has the 4 networks: xn0(wan), xn1(lan/vlan10), xn2(vlan11), xn3(vlan12)
the pfsense router also has IPsec site-to-site configured and that part is working fine.

I was having one issue before when I only had 2 interfaces to the pfsense sometimes i'd simply lose connectivity to the wan ip, and on gateway monitoring it'd show huge packetloss over 70%-90% - between the vm and the host (its gateway for the /29).

pfSense interface statistics don't show errors in/out or collisions.

Now as I added the other interfaces the access to other VM's is excruciatingly slow. accessing via web fails frequently and pages take forever to load. The pfsense has plenty of resources.
IIRC I can leave pfsense's default mtu as the xcp-ng host has it configured to 1400.
Before reconfiguring the interfaces I only had the strange packetloss situation, but now the slow speed is murder. Any clues please? Thank you

[maverickws]

Hi all,

After a lot of digging I came across this known issue which I believe is the same situation I am encountering.

After coming across this known open bug: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=188261

I was looking to apply the patch but I can't find ip_fastfwd.c

Thank you.

[laptopfreek0]

@maverickws

Have you had any success getting pfsense to work better on your newer Xen setup?

[maverickws]

@laptopfreek0 hi man sorry I was away from the forums didn't see this in the meanwhile.
We have our pfSense routers working perfectly up to speed now inc. HA with CARP.

I'm not sure here but IIRC the solution to this issue was to add a DHCP option to enforce MTU 1400 on all leases. Its option 26 | unsigned 16bit integer | 1400. - This was related to the vSwitch.

[laptopfreek0]

@maverickws
Thanks for your response.

I finally figured out my issue which wasn't quite related. It seems that when using xcp-ng 8.2 it ignores the ethtool-tx="off" and ethtool-rx="off". The same problem arises if you use either 8.0 or 8.1 and do a yum update. This took me a good amount of installs and reinstalls to figure this out. I hope that xcp-ng figures out what they managed to break that ignores the other-config on the newer installations, and makes pfsense useless on newer installs. Maybe this can help you avoid the pitfall if you plan to do updates to the hypervisor in the future.

[maverickws]

@laptopfreek0 hi mate, well to be honest I have 8.2 and I didn't come across that issue. Actually all you really need is ethtool-tx="off" that's how we have it and runs without issues.

[dave.opc]

@maverickws

have you checked iperf3 speeds between pfsense and xcp-ng itself?
Mine is bad. Additionally from pfsense to xcp-ng it has many retries during transfer