pfSense on Proxmox via vmbr0 - got LAN access, but no WAN/internet access - why?

newsboost

Hi all,

I've had a similar setup as this working for some months but then screwed something up and now I cannot remember what I once did to make it work and am asking for a piece of advice or ideas. My setup is that Proxmox has 16 GB RAM and running on a mini-pc with 4 LAN ports and it is the only router in my home-network. Furthermore I have 2 managed switches with a few VLANs (not that relevant here I think). All connected devices have both LAN + WAN access, except the Proxmox host has only LAN-access - not internet/WAN access. The configuration is:

Proxmox host:
I've passed through 3 of the 4 physical LAN ports through to pfSense (one WAN, one LAN VLAN trunk-port and one port for testing), that runs virtualized and created vmbr0 which I thought would enable me to access the internet once setup inside pfSense also:

proxmoxHost# ip -4 a s
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    inet 192.168.1.2/24 brd 192.168.1.255 scope global dynamic enp2s0
       valid_lft 5317sec preferred_lft 5317sec
7: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    inet 192.168.100.2/24 scope global vmbr0
       valid_lft forever preferred_lft forever
       
proxmoxHost# route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         192.168.100.1   0.0.0.0         UG    0      0        0 vmbr0
192.168.1.0     0.0.0.0         255.255.255.0   U     0      0        0 enp2s0
192.168.100.0   0.0.0.0         255.255.255.0   U     0      0        0 vmbr0

I have a suspicion that maybe something is wrong here but have never messed much with routing tables. vmbr0 should be for anything that is not LAN-access and enp2s0 is a physical cable so it acts as an access port and I can connect a laptop, if I screw up the configuration for my managed switches.

pfSense virtualized:
The vmbr0-interface that I believe should allow me to access the internet is configured:

And then inside the pfSense configuration, I've chosen that I don't use my normal VLAN subnets, in this case I use this subnet:

With the exact same configuration as I have for my other VLANs:

Furthermore, I currently for the firewall, have allowed everything on subnet 100:

The problem:

All my internet is routed through the pfSense-virtualized VM. Normally my devices have a VLAN that is connected physically via a cable that goes into the NIC for one of the 3 passed-through NIC's - and that seem to work just fine. But my Proxmox host does not have internet access - only LAN access.

If I ssh in to the pfSense VM (192.168.1.1 or 192.168.100.1) - or with any other device connected to any of the other VLANs and try to:

ping google or 192.168.100.1 it works in both cases
ping my laptop (192.168.1.54, which I used for SSH'ing into the pfSense VM) also works

However, If I ssh in to the Proxmox machine (192.168.1.2) and try to:

ping google or 192.168.100.1 I get "Destination Host Unreachable"
ping my laptop (192.168.1.54, which I used for SSH'ing into proxmox) however it works
ping pfSense on VLAN 1 (which my laptop is also on) at: 192.168.1.1 I get "Destination Host Unreachable"

If I run tcpdump on the Proxmox machine and ping it from my laptop the ping IS being received and sent out via the enp2s0-interface:

12:49:53.423348 enp2s0 In  IP 192.168.1.54 > 192.168.1.2: ICMP echo request, id 7, seq 2, length 64
12:49:53.423401 enp2s0 Out IP 192.168.1.2 > 192.168.1.54: ICMP echo reply, id 7, seq 2, length 64
12:49:54.424708 enp2s0 In  IP 192.168.1.54 > 192.168.1.2: ICMP echo request, id 7, seq 3, length 64
12:49:54.424759 enp2s0 Out IP 192.168.1.2 > 192.168.1.54: ICMP echo reply, id 7, seq 3, length 64

hmm... It's getting a bit complicated and enp2s0 is the built-in realtec NIC, which connects to the first managed switch, to an access port for VLAN 1, so it only handles VLAN 1 or subnet 192.168.1.1, it wouldn't handle subnet 192.168.100.0/24...

Anyway, I also get many of these, which I think is wrong:

12:55:21.928735 vmbr0 Out ARP, Request who-has 192.168.100.1 tell 192.168.100.2, length 28
12:55:21.928746 veth103i0 Out ARP, Request who-has 192.168.100.1 tell 192.168.100.2, length 28
12:55:21.928751 fwpr100p0 Out ARP, Request who-has 192.168.100.1 tell 192.168.100.2, length 28
12:55:21.928780 fwln100i0 B   ARP, Request who-has 192.168.100.1 tell 192.168.100.2, length 28
12:55:21.928792 tap100i0 Out ARP, Request who-has 192.168.100.1 tell 192.168.100.2, length 28
12:55:21.928803 fwbr100i0 B   ARP, Request who-has 192.168.100.1 tell 192.168.100.2, length 28

Damn, it's slightly complicated, anyway appreciate if anyone could give a few hints, it's probably just a minor mis-configuration somewhere...

If you need any info, please let me know, I'm very eager to fix this problem so I can update Proxmox and have internet access for the Proxmox host using vmbr0 - I appreciate ANY help/ideas/advice, thanks!

viragomann

@newsboost
Your vmbr0 seems to hang in the limbo. It has no slave network port assigned to it.

BTW: What the benefit to have two IPs in different subnets on Proxmox?
This could result in unwanted issues.

newsboost

@viragomann said in pfSense on Proxmox via vmbr0 - got LAN access, but no WAN/internet access - why?:

@newsboost
Your vmbr0 seems to hang in the limbo. It has no slave network port assigned to it.

UPDATE: SORRY I HAVE INTERNET NOW VIA "VMBR0" DESPITE THE ERROR MESSAGE, THANKS A LOT, THE PROBLEM SEEMS SOLVED - I'LL LEAVE THE BELOW QUESTIONS/ANSWER OPEN STILL, HOWEVER, THANKS INDEED, FRIEND!!!

Right, I've been thinking about that but got/receive errors when I tried to assign a port to it... I'm not so used to bridging, but is it correct that a bridge port always must be connected to a physical port? I guess the alternative is to bridge vmbr0 inside proxmox, to a physical network port? But in any case, I get the error below which I don't really understand:

BTW: What the benefit to have two IPs in different subnets on Proxmox?
This could result in unwanted issues.

Right. I have had something like this working for almost a year and then experimented with some stuff and forgot what I did and how or why I did things and now I cannot recreate it, because of the network errors you see in the screenshot above "command 'ifreload -a' failed: exit code 1" - this I don't understand?

So to answer the question: The problem is that my proxmox server is in a very small utility room with no keyboard, mouse or monitor plugged into. So in case I ever screw up my pfsense (have happened, although mostly in the beginning with firewall rules etc), I would have absolutely no network connection and a restart would not fix the issue as proxmox is using pfSense as its router...

So I knew/know I should at least leave one network port that is NOT passed through to pfSense. That way I can hopefully still login via ssh and fix/look at things, in case I break the pfSense VM, bringing down my home internet... So, I would like to access my Proxmox server with a port that uses a static ip of 192.168.1.xx/24, in case any emergencies happen. And for normal use I access my proxmox server using 192.168.1.2/24 because this is handed out by the pfSense VM that resides at 192.168.1.2/24. If pfSense goes down, so does 192.168.1.xx/24 - I'm trying to implement a backup-solution when/if that situation occurs...

The other thing is that I'm adding several other virtual machines, so these cannot all have a network cable as I only have a few NICs/network ports. Furthermore at the moment I would like Proxmox and the next VMs I'm making to have the DHCP-assigned IP of 192.168.100.xx/24, so all VMs and Proxmox can see each other. NB: I might probably change this in the future - or I'll implement firewall restrictions so not all VMs can see the Proxmox server. But one step first - first step is to make vmbr0 function...

I hope this explained the situation and answered the questions... And THANKS a lot for helping, I've had a break for a while but also suspected that the problem is that vmbr0 does not have a physical slave port, but it comes up with an error whenever I try to remedy that... I suspect it's a small issue, a small thing I don't understand... Got any ideas? Thanks again, for your valuable help!

viragomann

@newsboost
You cannot use a passed-through NIC on Proxmox itself. The only available NIC you can use is enp1s0f3.

So to answer the question: The problem is that my proxmox server is in a very small utility room with no keyboard, mouse or monitor plugged into. So in case I ever screw up my pfsense (have happened, although mostly in the beginning with firewall rules etc), I would have absolutely no network connection and a restart would not fix the issue as proxmox is using pfSense as its router...

So I knew/know I should at least leave one network port that is NOT passed through to pfSense. That way I can hopefully still login via ssh and fix/look at things, in case I break the pfSense VM, bringing down my home internet... So, I would like to access my Proxmox server with a port that uses a static ip of 192.168.1.xx/24, in case any emergencies happen. And for normal use I access my proxmox server using 192.168.1.2/24 because this is handed out by the pfSense VM that resides at 192.168.1.2/24. If pfSense goes down, so does 192.168.1.xx/24 - I'm trying to implement a backup-solution when/if that situation occurs...

That's not a prlausible reason to have two subnets on Proxmox.
Just connect the bridge vmbr0 to a physical NIC port and assign a static (!) IP to the bridge in Proxmox. This should be a trusted subnet of course.
So to access Proxmox in case of emergency, you have only to assign a static IP within the same subnet to a computer and connect it to the appropriate network port. Then you can access Proxmox independently from the state of pfSense.

newsboost

@viragomann said in pfSense on Proxmox via vmbr0 - got LAN access, but no WAN/internet access - why?:

@newsboost
You cannot use a passed-through NIC on Proxmox itself. The only available NIC you can use is enp1s0f3.

That makes completely sense to me and probably explains the error message, thanks! But I'm really confused now, because it seem to work, i.e. it provides VLAN 100 internet access and yet it seems that the interface is still being passed through, because enp1s0f0 = igb0 = WAN and enp1s0f1 = LAN (vlan trunk) = igb1... Are you sure this should not work, because it seem to work? And why does it work, is it kind of "undefined behaviour" perhaps? Great comment, thanks!

That's not a prlausible reason to have two subnets on Proxmox.

The explanation was not good enough... So, VLAN 1 (subnet 192.168.1.0/24) is my management VLAN and the VMs I create in Proxmox should preferably not have access to the management VLAN so I thought the safest and quickest solution would be to use another subnet for all my experimental VMs... That way, they don't have access to the more important devices/machines/printers/servers on VLAN 1... I think this is a better explanation, hopefully...

Just connect the bridge vmbr0 to a physical NIC port and assign a static (!) IP to the bridge in Proxmox. This should be a trusted subnet of course.

You're right - and I did just that and it also works:

From a logical perspective, this makes much more sense because as you wrote above and after I've been thinking about it, I think it's weird that I can bridge a NIC that has been passed through to proxmox and still get the behaviour that I wanted - but after my improved understanding and after reading your comment, now I wouldn't expect this to work any longer, but it still does... Very weird, it can bridge the NIC when passed through, apparently without internet/network problems!

So to access Proxmox in case of emergency, you have only to assign a static IP within the same subnet to a computer and connect it to the appropriate network port. Then you can access Proxmox independently from the state of pfSense.

It makes completely sense what you're writing and probably the solution could be that I should have two VMBR-interfaces:

One for emergencies, if pfSense does not respond or boot up correctly so I can plugin a network cable and ssh directly into Proxmox and
One on subnet 100, such that I can isolate all the VMs from the management VLAN and do experiments without any fear...

Is it really that bad if I put vmbr0 in the VLAN 100-subnet so the proxmox interfaces can be access on two different subnets? Because I've been testing and it seems to work completely fine on two different subnets - although perhaps I would like to later block VLAN 100 from accessing the Proxmox-interface and I can do that by adding a firewall-rule using the pfSense-interface, isn't that right?

Appreciate your comments a lot, thanks!