VLAN: 4 working, 2 failing; WAN using VLAN client IP?

MichaelAnders

Hi,
I run pfsense CE 2.6.0 (lets call it "firewall") on Proxmox (lets call it "host"). In my LAN I have several proxmox servers which use VLANs.

Setup (work-in-progress; not perfect yet, but not the focus of this topic and can be ignored to not deviate from my issue unless that is where the problem is - which I highly doubt):

gateway:
192.168.178.1
Fritzbox (very common router in Germany)

host:
Network setup (4 different physical network ports):
eno1: physical WAN port to gateway, 192.168.178.42/24, gateway 192.168.178.1
vmbr0: VLAN aware bridge from a physical network port to LAN, 192.168.100.4/24
vmbr3: VLAN aware bridge from a physical network port to LAN, used by pfsense as LAN
vmbr4: bridge from a physical network port to WAN, used by pfsense as WAN

proxmox VM on host:
net0: bridge vmbr4
net1: bridge vmbr3

Networks on firewall (pfsense):
WAN: vtnet0; 192.168.178.43/24
LAN: vtnet1; 192.1689.100.1/24
vtnet1.20; 192.168.20.1/24 ; OK
vtnet1.66; 192.168.66.1/24 ; FAIL
vtnet1.200; 192.168.200.1/24 ; OK
vtnet1.3001; 192.168.210.1/24 ; OK
vtnet1.11; 192.168.1.1/24 ; FAIL
vtnet1.3333; 192.168.33.1/24 ; OK

Everything works, except: out of the 6 VLANs I have created on the firewall, 4 work (internet access) while 2 are failing. I have no idea what is going on, but what I see in the firewall logs is VERY strange... for those that fail, WAN uses the VLAN IP?! Maybe someone here has an idea...

I have set up firewall rules as much as possible such that everything is logged.

To rule out any issues with my Unifi LAN/VLAN network, I will be using the SAME VM on the host (rules out some software/config issues), just changing the VLAN and then restarting it (DHCP is enabled). I will check two VM that work and compare them against the two which fail.

For each test, I will "ping 8.8.8.8"
If it fails, I will issue a "traceroute 8.8.8.8"
I will check the firewall logs for "8.8.8.8" - I can only see the sent packets though, not the incoming (maybe I did something wrong, no idea).

VLAN 3333 - OK:
response time: ~20ms
Firewall: Interface HA; 192.168.33.107; 8.8.8.8; ICMP
Firewall: Interface WAN; 192.168.178.43; 8.8.8.8; ICMP

VLAN 20 - OK:
response time: ~20ms
Firewall: Interface HOMENETWORK; 192.168.20.23; 8.8.8.8; ICMP
Firewall: Interface WAN; 192.168.178.43; 8.8.8.8; ICMP

VLAN 11 - FAIL:
response time: n/a
Firewall: Interface UNIFI3; 192.168.1.53; 8.8.8.8; ICMP
Firewall: Interface WAN; 192.168.1.53; 8.8.8.8; ICMP <----- VLAN IP!
traceroute:
1 192.168.1.1; 1.3 ms
(end)

VLAN 66 - FAIL:
response time: n/a
Firewall: Interface UNIFI3; 192.168.66.52; 8.8.8.8; ICMP
Firewall: Interface WAN; 192.168.66.52; 8.8.8.8; ICMP <----- VLAN IP!
traceroute:
1 192.168.66.1; 0.8 ms
(end)

Setting a static IP for the VM and defining the gateway 192.168.66.1 changes nothing - WAN still uses the same IP from the VLAN.

I have no idea why WAN IP = VLANs own VM client IP?

Maybe my pfsense config is broken somewhere, but (visually) I do not see any differences in the interfaces, vlans or dhcp servers except those that are needed.

Thanks,
Bjoern

MichaelAnders

Solved...
For some reason, "Firewall -> NAT -> Outbound" showed me an "Auto created rule for ISAKMP - ... to WAN" for one failing VLAN, but it did not add the "randomize Source port" entry automatically.

No clue why... I also seem to have had "Manual Outbound NAT rule generation." on, but then I wonder how I ended up with the above auto created rule.

In any case, I now added the needed NAT entries manuall and now finally it works :)