pfSense VM on Proxmox: PPPoE only works when parent NIC is PCI passthrough — virtual NIC breaks LAN→WAN traffic

w0w

@netblues
Did you file this issue on Redmine already?

netblues

@w0w No, I havent.

Steven said would try to replicate the issue localy.

Perhaps a redmine is now appropriate.

stephenw10

Mmm, your report was only for policy routed traffic. Given this new data that could just be your setup though.

@w0w You say clients can ping DNS servers, is that locally or over the PPPoE?

This feels like it might be an MTU/MSS issue if the virtual NIC is reporting the wrong value somehow.

netblues

@stephenw10
As the op says, it only happens on latest beta, which is also the case in what I see.
And looking at interface status everything mtu related looks fine my side too.

w0w

@stephenw10 said in pfSense VM on Proxmox: PPPoE only works when parent NIC is PCI passthrough — virtual NIC breaks LAN→WAN traffic:

You say clients can ping DNS servers, is that locally or over the PPPoE?

8.8.8.8

@stephenw10 said in pfSense VM on Proxmox: PPPoE only works when parent NIC is PCI passthrough — virtual NIC breaks LAN→WAN traffic:

This feels like it might be an MTU/MSS issue if the virtual NIC is reporting the wrong value somehow.

I have been played with the MTU/MSS values without any luck.

I also tried almost all sysctl hw.vtnet settings

hw.vtnet.altq_disable: 1
hw.vtnet.lro_mbufq_depth: 0
hw.vtnet.lro_entry_count: 128
hw.vtnet.rx_process_limit: 1024
hw.vtnet.tso_maxlen: 65535
hw.vtnet.mq_max_pairs: 32
hw.vtnet.mq_disable: 0
hw.vtnet.lro_disable: 1
hw.vtnet.tso_disable: 1
hw.vtnet.fixup_needs_csum: 0
hw.vtnet.csum_disable: 1

What I did not try are those tunables... this will be next

dev.vtnet.X.rxcsum=0
dev.vtnet.X.txcsum=0
dev.vtnet.X.tso=0

w0w

said in pfSense VM on Proxmox: PPPoE only works when parent NIC is PCI passthrough — virtual NIC breaks LAN→WAN traffic:

dev.vtnet.X.rxcsum=0
dev.vtnet.X.txcsum=0
dev.vtnet.X.tso=0

Failed also.

marcosm

Would you share the content/output of the following when it's working and when it's not?

Generated OpenVPN config, e.g.: /var/etc/openvpn/server1/config.ovpn
Filter rules: pfctl -a '*' -se; pfctl -a '*' -sn; pfctl -a '*' -sr

You can upload it here:
https://nc.netgate.com/nextcloud/s/8CQAsHwwooTRAPt

netblues

@marcosm Since I'm the only one reporting the issue with openvpn,
I have uploaded the requested info.

However, testing further reveals that dco enabled client connection doesn't work only if the vpn is established over pppoe internet connection.

if the dco enabled openvp connection uses dhcp wan, then openvpn works fine.

So, opevpn client without dco works over pppoe connection from a non virtual pc, while at the same time , the same pc can only ping anything on the Internet but fails on anything else.

netblues

Just tried the new 2611 rc version.
The issue remains unchanged.

w0w

@netblues said in pfSense VM on Proxmox: PPPoE only works when parent NIC is PCI passthrough — virtual NIC breaks LAN→WAN traffic:

2611 rc version.

It looks like it’s working for me — I can reach the Internet through PPPoE on vtnet.

netblues

@w0w Are you sure?It is definitely NOT fixed here.

Cab you revert to previus rc and check that you have the issue there?

w0w

@netblues
Testing all variants right now, I'll let you know if I'll find something

stephenw10

Hmm, this would be interesting. AFAIK no specific fix for this went in since the beta. So if it is now working it must have pulled in a change with something else.

netblues

@stephenw10 The only difference here is proxmox kvm vs redhat 9.6 kvm.
I doubt there is a difference.

I have tested with bios boot. Can do the same with uefi, but in previous rc, the issues where the same.

@w0w Are you using uefi or bios boot? (and the relevant 440fx versus q35 hardware emulation)

w0w

Also, the Ookla Speedtest in Edge shows full speed.
Some specifics… This version was installed from the online installer with the configuration restored using the same installer.
When it booted for the first time, I had to go into Routes and manually switch the default IPv4 and IPv6 gateways to the PPPoE one, because I had the multi-WAN gateway set there. Before that it wasn’t working — or more precisely, it was working via the backup WAN gateway (I have a multi-WAN setup).

After forcing the PPPoE gateway, I checked that the Internet was reachable from a client. Then I went back and set the default gateway to the multi-WAN gateway again and verified that whatismyip still showed the PPPoE IP. After that I rebooted several times — everything continued to work correctly.

And issue remains on the previous RC version.

@netblues said in pfSense VM on Proxmox: PPPoE only works when parent NIC is PCI passthrough — virtual NIC breaks LAN→WAN traffic:

@stephenw10 The only difference here is proxmox kvm vs redhat 9.6 kvm.
I doubt there is a difference.

I have tested with bios boot. Can do the same with uefi, but in previous rc, the issues where the same.

@w0w Are you using uefi or bios boot? (and the relevant 440fx versus q35 hardware emulation)

I don't know... some kind of magic.

stephenw10

Hmm, you are using if_pppoe on a lagg of vtnet rather than directly on vtnet. Possible difference. Have you always been running that?

w0w

@stephenw10
I’ve been running LAGGs in failover mode for years on literally every interface. This makes things simpler for HA and also for hardware changes.

In any case, when it wasn’t working, I tried every option—LAGG, direct connection, everything. I’m actually surprised it’s working now.

netblues

I have also tried q35 and uefi boot.
The issue remains.
Booting to anything lese than the last two beta/rc releases , with the same config works correctly.

w0w

@netblues
I can't explain this. So you are using PPPoE (over vtnet) and clients on LAN can not reach internet?

netblues

@w0w Yes.
And at the same time, clients on the same hypervisor, bound to the same bridge to lan, using virtio, can reach the Internet fine.

Also clients on the physical lan, can ping the Internet over pppoe.