VLAN subnet can't access pfSense

fenugurod

After days debugging this problem I'm starting to reach insanity and some help would be awesome. Maybe it's something easy that I'm missing out

Context
I'm running pfSense virtualized at Proxmox. My network is composed by a TPLink Omada switch and a few Omada APs. The NIC at the server is an Intel I350-T4 with SR-IOV enabled. Without VLANs everything is working alright and my devices have access to LAN and WAN. The problem is at the VLANs. Devices connected on a VLAN subnet doesn't have access to pfSense but I can ping, after I set a static IP given that they don't reach pfSense, each other at the VLAN. The VLAN that I'm trying to setup is called Guest and it has the ID 40.

Physical network configuration
At the switch all ports are configured to use the profile ALL.
Screenshot 2024-01-28 at 21.31.15.png

The VLAN is enabled at the switch:

This is the VLAN network:

The DHCP L2 Relay is enabled but I tested with it and without it, but the problem persists.

And this is the configuration of the ALL profile used on every port of the switch:

This is a wireless network assigned to the VLAN 40:

dmesg output on proxmox
This is the output of dmesg on Proxmox. It show that IOMMU is enabled but at the same time there are a few IOMMU feature*** inconsistent. Could this be a problem either at the hardware or configuration?

> dmesg | grep -e DMAR -e IOMMU

[    0.010929] ACPI: DMAR 0x0000000078630000 000088 (v02 INTEL  EDK2     00000002      01000013)
[    0.010957] ACPI: Reserving DMAR table memory at [mem 0x78630000-0x78630087]
[    0.069067] DMAR: IOMMU enabled
[    0.158812] DMAR: Host address width 39
[    0.158813] DMAR: DRHD base: 0x000000fed90000 flags: 0x0
[    0.158816] DMAR: dmar0: reg_base_addr fed90000 ver 4:0 cap 1c0000c40660462 ecap 29a00f0505e
[    0.158817] DMAR: DRHD base: 0x000000fed91000 flags: 0x1
[    0.158821] DMAR: dmar1: reg_base_addr fed91000 ver 5:0 cap d2008c40660462 ecap f050da
[    0.158822] DMAR: RMRR base: 0x0000007e000000 end: 0x000000807fffff
[    0.158824] DMAR-IR: IOAPIC id 2 under DRHD base  0xfed91000 IOMMU 1
[    0.158825] DMAR-IR: HPET id 0 under DRHD base 0xfed91000
[    0.158826] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[    0.160320] DMAR-IR: Enabled IRQ remapping in x2apic mode
[    0.333267] pci 0000:00:02.0: DMAR: Skip IOMMU disabling for graphics
[    0.391278] DMAR: No ATSR found
[    0.391279] DMAR: No SATC found
[    0.391280] DMAR: IOMMU feature fl1gp_support inconsistent
[    0.391280] DMAR: IOMMU feature pgsel_inv inconsistent
[    0.391281] DMAR: IOMMU feature nwfs inconsistent
[    0.391281] DMAR: IOMMU feature dit inconsistent
[    0.391282] DMAR: IOMMU feature sc_support inconsistent
[    0.391282] DMAR: IOMMU feature dev_iotlb_support inconsistent
[    0.391282] DMAR: dmar0: Using Queued invalidation
[    0.391284] DMAR: dmar1: Using Queued invalidation
[    0.391893] DMAR: Intel(R) Virtualization Technology for Directed I/O

A full dmesg output can be found here. There are some warnings there but I don't know to which extend those could be a problem. Except for this one that looks suspect:

igb 0000:05:00.3 enp5s0f3: malformed Tx packet detected and dropped, LVMMC:0x34000000

This the part caught my attention because I'm using enp5s0f3v0 as the LAN interface, which is working ok, and I'm creating a VLAN in pfSense on top of that interface.

Proxmox configuration
This is my /etc/network/interfaces config:

source /etc/network/interfaces.d/*

auto lo
iface lo inet loopback

auto enp5s0f1
iface enp5s0f1 inet static
    address 10.0.10.2/24
    gateway 10.0.10.1
    dns-nameservers 1.1.1.1
    dns-search internal

auto enp3s0
iface enp3s0 inet manual

auto enp5s0f0
iface enp5s0f0 inet manual

auto enp5s0f2
iface enp5s0f2 inet manual

auto enp5s0f3
iface enp5s0f3 inet manual

enp5s0f0v0 is used as WAN and enp5s0f3v0 LAN.

And this is my systemd service that I use to configure SR-IOV during boot:

[Unit]
Description=Script to enable NIC SR-IOV on boot

[Service]
Type=oneshot
ExecStart=/usr/bin/bash -c '/usr/bin/echo 2 > /sys/class/net/enp5s0f0/device/sriov_numvfs'
ExecStart=/usr/bin/bash -c '/usr/bin/echo 2 > /sys/class/net/enp5s0f1/device/sriov_numvfs'
ExecStart=/usr/bin/bash -c '/usr/bin/echo 2 > /sys/class/net/enp5s0f2/device/sriov_numvfs'
ExecStart=/usr/bin/bash -c '/usr/bin/echo 2 > /sys/class/net/enp5s0f3/device/sriov_numvfs'

# enp5s0f0
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set enp5s0f0 vf 0 mac a0:36:9f:7d:35:00'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set enp5s0f0 vf 1 mac a0:36:9f:7d:35:01'

# enp5s0f1
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set enp5s0f1 vf 0 mac a0:36:9f:7d:35:02'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set enp5s0f1 vf 1 mac a0:36:9f:7d:35:03'

# enp5s0f2
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set enp5s0f2 vf 0 mac a0:36:9f:7d:35:04'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set enp5s0f2 vf 1 mac a0:36:9f:7d:35:05'

# enp5s0f3
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set enp5s0f3 vf 0 mac a0:36:9f:7d:35:06'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set enp5s0f3 vf 1 mac a0:36:9f:7d:35:07'

[Install]
WantedBy=multi-user.target

pfSense configuration
Interface assignment:

LAN40 configuration:

LAN40 firewall rules:

LAN40 DHCP configuration:

fenugurod

I can confirm, the problem is 100% with SR-IOV. Yesterday I removed all the SR-IOV part and I'm passing the NIC port directly to pfSense and everything works. The question now is, where is the best place that I can go to talk about SR-IOV?

laser22

go to the proxmox forum