pfSense VM on Proxmox: PPPoE only works when parent NIC is PCI passthrough — virtual NIC breaks LAN→WAN traffic

netblues

@stephenw10 The only difference here is proxmox kvm vs redhat 9.6 kvm.
I doubt there is a difference.

I have tested with bios boot. Can do the same with uefi, but in previous rc, the issues where the same.

@w0w Are you using uefi or bios boot? (and the relevant 440fx versus q35 hardware emulation)

w0w

Also, the Ookla Speedtest in Edge shows full speed.
Some specifics… This version was installed from the online installer with the configuration restored using the same installer.
When it booted for the first time, I had to go into Routes and manually switch the default IPv4 and IPv6 gateways to the PPPoE one, because I had the multi-WAN gateway set there. Before that it wasn’t working — or more precisely, it was working via the backup WAN gateway (I have a multi-WAN setup).

After forcing the PPPoE gateway, I checked that the Internet was reachable from a client. Then I went back and set the default gateway to the multi-WAN gateway again and verified that whatismyip still showed the PPPoE IP. After that I rebooted several times — everything continued to work correctly.

And issue remains on the previous RC version.

@netblues said in pfSense VM on Proxmox: PPPoE only works when parent NIC is PCI passthrough — virtual NIC breaks LAN→WAN traffic:

@stephenw10 The only difference here is proxmox kvm vs redhat 9.6 kvm.
I doubt there is a difference.

I have tested with bios boot. Can do the same with uefi, but in previous rc, the issues where the same.

@w0w Are you using uefi or bios boot? (and the relevant 440fx versus q35 hardware emulation)

I don't know... some kind of magic.

stephenw10

Hmm, you are using if_pppoe on a lagg of vtnet rather than directly on vtnet. Possible difference. Have you always been running that?

w0w

@stephenw10
I’ve been running LAGGs in failover mode for years on literally every interface. This makes things simpler for HA and also for hardware changes.

In any case, when it wasn’t working, I tried every option—LAGG, direct connection, everything. I’m actually surprised it’s working now.

netblues

I have also tried q35 and uefi boot.
The issue remains.
Booting to anything lese than the last two beta/rc releases , with the same config works correctly.

w0w

@netblues
I can't explain this. So you are using PPPoE (over vtnet) and clients on LAN can not reach internet?

netblues

@w0w Yes.
And at the same time, clients on the same hypervisor, bound to the same bridge to lan, using virtio, can reach the Internet fine.

Also clients on the physical lan, can ping the Internet over pppoe.

w0w

I was able to reproduce this bug: I installed 25.07.1, restored the configuration, verified that LAN clients had Internet access, and then upgraded to the latest RC. After the upgrade, the clients no longer had Internet access.
That's fun...

netblues

@w0w And most peobably can only ping too

w0w

@netblues
Yes, like only ICMP working

netblues

@w0w So we definitely have an issue here. It can't be a configuration issue, and certainly NOT a firewall rules issue.
But I remain clueless where to look. (Besides the fact that I need to revert for practical reasons, and running another pf plus vm in parallel for testing has licensing issues too)

w0w

@netblues
I dug a bit deeper. I compared the system that was installed from scratch with the one that was upgraded. Of course, things went a bit sideways, but overall there are noticeable differences in both libraries and some binaries, which raises some questions — although in general this could simply be a consequence of the FreeBSD version upgrade.
By the way, have you tried installing it using the Netgate installer?

netblues

@w0w So you say that by doing a clean default install with netgate installer AND restoring the config would work in latest RC?

Can't check this right now, someone might shoot me and it would be netgates fault

w0w

@netblues said in pfSense VM on Proxmox: PPPoE only works when parent NIC is PCI passthrough — virtual NIC breaks LAN→WAN traffic:

So you say that by doing a clean default install with netgate installer AND restoring the config would work in latest RC?

It worked once at least, but I didn’t try it again

netblues

@w0w Brewing... I know in a while

Nada. Issue remains the same. I installed directly into latest rc 25.11.r.20251126.1732
and restored config.

Only ping over pppoe.

w0w

@netblues

ok… thats interesting…

ifconfig vtnet0 -rxcsum -txcsum -rxcsum6 -txcsum6 -vlanhwtag -vlanhwcsum

Try this on LAN interface. Hope this works for you...

I just compared ifconfig output for both working and non working VMs and looks like working VM on every interface have options=880008<VLAN_MTU,LINKSTATE,HWSTATS>
So I applied it to the pppoe parent - still no go... and then I've tried LAN interface and it worked for me.

netblues

@w0w said in pfSense VM on Proxmox: PPPoE only works when parent NIC is PCI passthrough — virtual NIC breaks LAN→WAN traffic:

ifconfig vtnet0 -rxcsum -txcsum -rxcsum6 -txcsum6 -vlanhwtag -vlanhwcsum

Tried that.. Still , no dice. :(

w0w

@netblues
Please show your ifconfig output for LAN and pppoe parent interface.

netblues

@w0w I have now created a fresh default install
Directly install 25.11rc from netgate installer , configured everything by the gui just for a single lan, and a single pppoe connection.

Automatic outbound nat etc. No changes anywhere

ping works, everything else on physical lan fails (miserably)
pfsense (and anything on virtual) can install packages, and has full Internet

have tried disabling checksums too. No dice

ifconfig vtnet1
vtnet1: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
   description: WAN
   options=8800a8<VLAN_MTU,JUMBO_MTU,VLAN_HWCSUM,LINKSTATE,HWSTATS>
   ether d4:5d:64:08:66:46
   inet6 fe80::d65d:64ff:fe08:6646%vtnet1 prefixlen 64 scopeid 0x2
   media: Ethernet autoselect (10Gbase-T <full-duplex>)
   status: active
   nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

pppoe0: flags=10088d1<UP,POINTOPOINT,RUNNING,NOARP,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1492
   description: Ftth1G
   options=0
   inet 100.79.101.245 --> 10.106.108.100 netmask 0xffffffff
   inet6 fe80::d65d:64ff:fe08:6646%pppoe0 prefixlen 64 scopeid 0x7
   nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

As a side note, when restoring configuration to a fresh install 25.11rc, all packages got reinstalled, however the widget says no packages.
I have tried removing it and adding again. Nada.
Tried adding a new package (from gui), package got installed the widget insists. No packages installed.

Steps to reproduce. Install fresh pfplus 25.11rc, restore config that has package widget and some packages, wait for the packages reinstallation, and voila !

w0w

@netblues
You forgot to show your ifconfig LAN output.

loader.conf.local (you need to reboot after making changes)

hw.vtnet.altq_disable=1
hw.vtnet.tso_disable=1
hw.vtnet.csum_disable=1

LAN

ix0: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
	options=4813828<VLAN_MTU,JUMBO_MTU,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,HWSTATS,MEXTPG>
	ether a0:3------25
	inet6 fe80::aab8:e0ff:fe02:655a%ix0 prefixlen 64 scopeid 0x1
	media: Ethernet autoselect (10Gbase-T <full-duplex>)
	status: active
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

WAN parent

vtnet0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
	options=880008<VLAN_MTU,LINKSTATE,HWSTATS>
	ether a-----:24
	inet6 fe80::aab8:e0ff:fe02:655a%vtnet0 prefixlen 64 scopeid 0x6
	media: Ethernet autoselect (10Gbase-T <full-duplex>)
	status: active
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

Shell Output - sysctl hw.vtnet.

hw.vtnet.altq_disable: 1
hw.vtnet.lro_mbufq_depth: 0
hw.vtnet.lro_entry_count: 128
hw.vtnet.rx_process_limit: 1024
hw.vtnet.tso_maxlen: 65535
hw.vtnet.mq_max_pairs: 32
hw.vtnet.mq_disable: 0
hw.vtnet.lro_disable: 1
hw.vtnet.tso_disable: 1
hw.vtnet.fixup_needs_csum: 0
hw.vtnet.csum_disable: 1

Side note — if you have vlans on LAN you should not use -vlanhwtag posted previously, this will break vlans