Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    pfSense VM on Proxmox: PPPoE only works when parent NIC is PCI passthrough — virtual NIC breaks LAN→WAN traffic

    Scheduled Pinned Locked Moved Plus 25.11 Snapshots
    51 Posts 4 Posters 769 Views 6 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • N Online
      netblues @w0w
      last edited by

      I have also tried q35 and uefi boot.
      The issue remains.
      Booting to anything lese than the last two beta/rc releases , with the same config works correctly.

      w0wW 1 Reply Last reply Reply Quote 0
      • w0wW Away
        w0w @netblues
        last edited by

        @netblues
        I can't explain this. So you are using PPPoE (over vtnet) and clients on LAN can not reach internet?

        N 1 Reply Last reply Reply Quote 0
        • N Online
          netblues @w0w
          last edited by

          @w0w Yes.
          And at the same time, clients on the same hypervisor, bound to the same bridge to lan, using virtio, can reach the Internet fine.

          Also clients on the physical lan, can ping the Internet over pppoe.

          1 Reply Last reply Reply Quote 0
          • w0wW Away
            w0w
            last edited by

            I was able to reproduce this bug: I installed 25.07.1, restored the configuration, verified that LAN clients had Internet access, and then upgraded to the latest RC. After the upgrade, the clients no longer had Internet access.
            That's fun...

            N 1 Reply Last reply Reply Quote 1
            • N Online
              netblues @w0w
              last edited by

              @w0w And most peobably can only ping too

              w0wW 1 Reply Last reply Reply Quote 0
              • w0wW Away
                w0w @netblues
                last edited by

                @netblues
                Yes, like only ICMP working

                N 1 Reply Last reply Reply Quote 1
                • N Online
                  netblues @w0w
                  last edited by

                  @w0w So we definitely have an issue here. It can't be a configuration issue, and certainly NOT a firewall rules issue.
                  But I remain clueless where to look. (Besides the fact that I need to revert for practical reasons, and running another pf plus vm in parallel for testing has licensing issues too)

                  w0wW 1 Reply Last reply Reply Quote 0
                  • w0wW Away
                    w0w @netblues
                    last edited by

                    @netblues
                    I dug a bit deeper. I compared the system that was installed from scratch with the one that was upgraded. Of course, things went a bit sideways, but overall there are noticeable differences in both libraries and some binaries, which raises some questions — although in general this could simply be a consequence of the FreeBSD version upgrade.
                    By the way, have you tried installing it using the Netgate installer?

                    N 1 Reply Last reply Reply Quote 0
                    • N Online
                      netblues @w0w
                      last edited by

                      @w0w So you say that by doing a clean default install with netgate installer AND restoring the config would work in latest RC?

                      Can't check this right now, someone might shoot me and it would be netgates fault 😱

                      w0wW 1 Reply Last reply Reply Quote 1
                      • w0wW Away
                        w0w @netblues
                        last edited by

                        @netblues said in pfSense VM on Proxmox: PPPoE only works when parent NIC is PCI passthrough — virtual NIC breaks LAN→WAN traffic:

                        So you say that by doing a clean default install with netgate installer AND restoring the config would work in latest RC?

                        It worked once at least, but I didn’t try it again 😊

                        N 1 Reply Last reply Reply Quote 0
                        • N Online
                          netblues @w0w
                          last edited by netblues

                          @w0w Brewing... I know in a while

                          Nada. Issue remains the same. I installed directly into latest rc 25.11.r.20251126.1732
                          and restored config.

                          Only ping over pppoe.

                          w0wW 1 Reply Last reply Reply Quote 1
                          • w0wW Away
                            w0w @netblues
                            last edited by w0w

                            @netblues

                            ok… thats interesting…

                            ifconfig vtnet0 -rxcsum -txcsum -rxcsum6 -txcsum6 -vlanhwtag -vlanhwcsum

                            Try this on LAN interface. Hope this works for you...

                            I just compared ifconfig output for both working and non working VMs and looks like working VM on every interface have options=880008<VLAN_MTU,LINKSTATE,HWSTATS>
                            So I applied it to the pppoe parent - still no go... and then I've tried LAN interface and it worked for me.

                            N 1 Reply Last reply Reply Quote 0
                            • N Online
                              netblues @w0w
                              last edited by

                              @w0w said in pfSense VM on Proxmox: PPPoE only works when parent NIC is PCI passthrough — virtual NIC breaks LAN→WAN traffic:

                              ifconfig vtnet0 -rxcsum -txcsum -rxcsum6 -txcsum6 -vlanhwtag -vlanhwcsum

                              Tried that.. Still , no dice. :(

                              w0wW 1 Reply Last reply Reply Quote 0
                              • w0wW Away
                                w0w @netblues
                                last edited by

                                @netblues
                                Please show your ifconfig output for LAN and pppoe parent interface.

                                N 1 Reply Last reply Reply Quote 0
                                • N Online
                                  netblues @w0w
                                  last edited by netblues

                                  @w0w I have now created a fresh default install
                                  Directly install 25.11rc from netgate installer , configured everything by the gui just for a single lan, and a single pppoe connection.

                                  Automatic outbound nat etc. No changes anywhere

                                  ping works, everything else on physical lan fails (miserably)
                                  pfsense (and anything on virtual) can install packages, and has full Internet

                                  have tried disabling checksums too. No dice

                                  ifconfig vtnet1
vtnet1: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
   description: WAN
   options=8800a8<VLAN_MTU,JUMBO_MTU,VLAN_HWCSUM,LINKSTATE,HWSTATS>
   ether d4:5d:64:08:66:46
   inet6 fe80::d65d:64ff:fe08:6646%vtnet1 prefixlen 64 scopeid 0x2
   media: Ethernet autoselect (10Gbase-T <full-duplex>)
   status: active
   nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

pppoe0: flags=10088d1<UP,POINTOPOINT,RUNNING,NOARP,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1492
   description: Ftth1G
   options=0
   inet 100.79.101.245 --> 10.106.108.100 netmask 0xffffffff
   inet6 fe80::d65d:64ff:fe08:6646%pppoe0 prefixlen 64 scopeid 0x7
   nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

                                  As a side note, when restoring configuration to a fresh install 25.11rc, all packages got reinstalled, however the widget says no packages.
                                  I have tried removing it and adding again. Nada.
                                  Tried adding a new package (from gui), package got installed the widget insists. No packages installed.

                                  Steps to reproduce. Install fresh pfplus 25.11rc, restore config that has package widget and some packages, wait for the packages reinstallation, and voila !

                                  w0wW 1 Reply Last reply Reply Quote 0
                                  • w0wW Away
                                    w0w @netblues
                                    last edited by w0w

                                    @netblues
                                    You forgot to show your ifconfig LAN output.

                                    loader.conf.local (you need to reboot after making changes)

                                    hw.vtnet.altq_disable=1
hw.vtnet.tso_disable=1
hw.vtnet.csum_disable=1

                                    LAN

                                    ix0: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
	options=4813828<VLAN_MTU,JUMBO_MTU,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,HWSTATS,MEXTPG>
	ether a0:3------25
	inet6 fe80::aab8:e0ff:fe02:655a%ix0 prefixlen 64 scopeid 0x1
	media: Ethernet autoselect (10Gbase-T <full-duplex>)
	status: active
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

                                    WAN parent

                                    vtnet0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
	options=880008<VLAN_MTU,LINKSTATE,HWSTATS>
	ether a-----:24
	inet6 fe80::aab8:e0ff:fe02:655a%vtnet0 prefixlen 64 scopeid 0x6
	media: Ethernet autoselect (10Gbase-T <full-duplex>)
	status: active
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

                                    Shell Output - sysctl hw.vtnet.

                                    hw.vtnet.altq_disable: 1
hw.vtnet.lro_mbufq_depth: 0
hw.vtnet.lro_entry_count: 128
hw.vtnet.rx_process_limit: 1024
hw.vtnet.tso_maxlen: 65535
hw.vtnet.mq_max_pairs: 32
hw.vtnet.mq_disable: 0
hw.vtnet.lro_disable: 1
hw.vtnet.tso_disable: 1
hw.vtnet.fixup_needs_csum: 0
hw.vtnet.csum_disable: 1

                                    Side note — if you have vlans on LAN you should not use -vlanhwtag posted previously, this will break vlans

                                    N 1 Reply Last reply Reply Quote 0
                                    • N Online
                                      netblues @w0w
                                      last edited by

                                      @w0w
                                      All vlan configuration is handled at the hypervisor level.
                                      pf sees only virtual interfaces.

                                      Here is the output

                                      sysctl hw.vtnet
hw.vtnet.altq_disable: 1
hw.vtnet.lro_mbufq_depth: 0
hw.vtnet.lro_entry_count: 128
hw.vtnet.rx_process_limit: 1024
hw.vtnet.tso_maxlen: 65535
hw.vtnet.mq_max_pairs: 32
hw.vtnet.mq_disable: 0
hw.vtnet.lro_disable: 1
hw.vtnet.tso_disable: 1
hw.vtnet.fixup_needs_csum: 0
hw.vtnet.csum_disable: 1

                                      ifconfig vtnet0
vtnet0: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
   options=880028<VLAN_MTU,JUMBO_MTU,LINKSTATE,HWSTATS>
   ether 52:54:00:05:01:fb
   inet 192.168.31.3 netmask 0xffffff00 broadcast 192.168.31.255
   inet6 fe80::5054:ff:fe05:1fb%vtnet0 prefixlen 64 scopeid 0x1
   media: Ethernet autoselect (10Gbase-T <full-duplex>)
   status: active
   nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
                                      w0wW 2 Replies Last reply Reply Quote 0
                                      • w0wW Away
                                        w0w @netblues
                                        last edited by

                                        @netblues
                                        I kinda screwed up… I forgot to mention that this ix0 is part of a LAGG interface, and the LAN itself is using that LAGG. This shouldn’t really affect anything, but I’ll check if that’s the issue. Also, I don’t remember changing any settings for this NIC on the host, I think I left it as is.

                                        1 Reply Last reply Reply Quote 0
                                        • w0wW Away
                                          w0w
                                          last edited by

                                          Configured LAN to use ix0 directly — nothing changed.

                                          1 Reply Last reply Reply Quote 0
                                          • w0wW Away
                                            w0w @netblues
                                            last edited by w0w

                                            @netblues said in pfSense VM on Proxmox: PPPoE only works when parent NIC is PCI passthrough — virtual NIC breaks LAN→WAN traffic:

                                            JUMBO_MTU

                                            Hmm… I don't see any jumbo settings on my vtnet interfaces, did you change something? VM setiings? Nonstandard MTU?

                                            Now I have configured it directly for both pppoe and LAN

                                            vtnet0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
	options=880008<VLAN_MTU,LINKSTATE,HWSTATS>
	ether a-
	inet6 fe80::aab8:e0ff:fe02:655a%vtnet0 prefixlen 64 scopeid 0x5
	media: Ethernet autoselect (10Gbase-T <full-duplex>)
	status: active
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
vtnet1: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
	description: LAN
	options=880008<VLAN_MTU,LINKSTATE,HWSTATS>
	ether a-
	inet 10.0.67.2 netmask 0xffffff00 broadcast 10.0.67.255
	inet 10.0.67.5 netmask 0xffffff00 broadcast 10.0.67.255 vhid 5
	inet 10.0.70.5 netmask 0xffffff00 broadcast 10.0.70.255 vhid 10
	inet 10.0.70.11 netmask 0xffffff00 broadcast 10.0.70.255
	inet6 fe80::a236:9fff:fef8:f225%vtnet1 prefixlen 64 scopeid 0x6
	inet6 fd00:1234:abcd:1::2 prefixlen 64
	inet6 fd00:1234:abcd:1::5 prefixlen 64 vhid 12
	carp: MASTER vhid 5 advbase 5 advskew 100
	      peer 224.0.0.18 peer6 ff02::12
	carp: MASTER vhid 10 advbase 5 advskew 100
	      peer 224.0.0.18 peer6 ff02::12
	carp: MASTER vhid 12 advbase 5 advskew 100
	      peer 224.0.0.18 peer6 ff02::12
	media: Ethernet autoselect (10Gbase-T <full-duplex>)
	status: active
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

                                            Ok, I've changed MTU to 9000 in proxmox for the LAN card/bridge/vtnet

                                            vtnet1: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
	description: LAN
	options=880028<VLAN_MTU,JUMBO_MTU,LINKSTATE,HWSTATS>

                                            Still working for me… no problem.

                                            Proxmox settings for WAN parent

                                            :~# ethtool -k enp6s0f0
Features for enp6s0f0:
rx-checksumming: on
tx-checksumming: on
        tx-checksum-ipv4: off [fixed]
        tx-checksum-ip-generic: on
        tx-checksum-ipv6: off [fixed]
        tx-checksum-fcoe-crc: on [fixed]
        tx-checksum-sctp: on
scatter-gather: on
        tx-scatter-gather: on
        tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: on
        tx-tcp-segmentation: on
        tx-tcp-ecn-segmentation: off [fixed]
        tx-tcp-mangleid-segmentation: off
        tx-tcp6-segmentation: on
        tx-tcp-accecn-segmentation: off [fixed]
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: off
receive-hashing: on
highdma: on [fixed]
rx-vlan-filter: on
vlan-challenged: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: on [fixed]
tx-gre-segmentation: on
tx-gre-csum-segmentation: on
tx-ipxip4-segmentation: on
tx-ipxip6-segmentation: on
tx-udp_tnl-segmentation: on
tx-udp_tnl-csum-segmentation: on
tx-gso-partial: on
tx-tunnel-remcsum-segmentation: off [fixed]
tx-sctp-segmentation: off [fixed]
tx-esp-segmentation: on
tx-udp-segmentation: on
tx-gso-list: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off
hw-tc-offload: off
esp-hw-offload: on
esp-tx-csum-hw-offload: on
rx-udp_tunnel-port-offload: on
tls-hw-tx-offload: off [fixed]
tls-hw-rx-offload: off [fixed]
rx-gro-hw: off [fixed]
tls-hw-record: off [fixed]
rx-gro-list: off
macsec-hw-offload: off [fixed]
rx-udp-gro-forwarding: off
hsr-tag-ins-offload: off [fixed]
hsr-tag-rm-offload: off [fixed]
hsr-fwd-offload: off [fixed]
hsr-dup-offload: off [fixed]

                                            LAN

                                             ethtool -k enp6s0f1
Features for enp6s0f1:
rx-checksumming: on
tx-checksumming: on
        tx-checksum-ipv4: off [fixed]
        tx-checksum-ip-generic: on
        tx-checksum-ipv6: off [fixed]
        tx-checksum-fcoe-crc: on [fixed]
        tx-checksum-sctp: on
scatter-gather: on
        tx-scatter-gather: on
        tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: on
        tx-tcp-segmentation: on
        tx-tcp-ecn-segmentation: off [fixed]
        tx-tcp-mangleid-segmentation: off
        tx-tcp6-segmentation: on
        tx-tcp-accecn-segmentation: off [fixed]
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: off
receive-hashing: on
highdma: on [fixed]
rx-vlan-filter: on
vlan-challenged: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: on [fixed]
tx-gre-segmentation: on
tx-gre-csum-segmentation: on
tx-ipxip4-segmentation: on
tx-ipxip6-segmentation: on
tx-udp_tnl-segmentation: on
tx-udp_tnl-csum-segmentation: on
tx-gso-partial: on
tx-tunnel-remcsum-segmentation: off [fixed]
tx-sctp-segmentation: off [fixed]
tx-esp-segmentation: on
tx-udp-segmentation: on
tx-gso-list: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off
hw-tc-offload: off
esp-hw-offload: on
esp-tx-csum-hw-offload: on
rx-udp_tunnel-port-offload: on
tls-hw-tx-offload: off [fixed]
tls-hw-rx-offload: off [fixed]
rx-gro-hw: off [fixed]
tls-hw-record: off [fixed]
rx-gro-list: off
macsec-hw-offload: off [fixed]
rx-udp-gro-forwarding: off
hsr-tag-ins-offload: off [fixed]
hsr-tag-rm-offload: off [fixed]
hsr-fwd-offload: off [fixed]
hsr-dup-offload: off [fixed]
                                            N 1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.