pfSense on PROXMOX with HomeAssistant

bearhntr · Sep 23, 2022, 2:27 PM

Thanks for the reply. I am not using VLANS at all -- I would not know where to even begin with those. Apologies for the length.

What I am trying to learn/do -- as I can do it in ESXi (but I want to use Proxmox) - is on a machine with ONE NIC (for example) -- How do I setup pfSense. I realize that the line from the cable modem has to go some place.

In my old configuration (a stand-alone pfSense box with 5 NICs - one of the NICs on the 4-port card was my LAN port, and the on-board NIC was the WAN (cable from ISP was there)). This worked great as the LAN cable went into my ORBI (which was set to AP mode) - and it handled the WiFi only The DNS/DHCP/IPv4/IPv6 was all done in pfSense.

So when I went to install Proxmox - I had put the ORBI into Router mode - until Proxmox was loaded and then pfSense on there. I had modem plugged into WAN port on ORBI and then a cable from one of its ports plugged into the on-board port on the new Proxmox box. The ORBI had a DHCP RSVP for this port to give it the IP of 192.168.10.252 (what I wanted the Proxmox to use). I did the install and during the install I chose the enp2s0 (the on-board NIC - the others were called enp1s0f0, 1, 2, 3 (the 4-port card) -- during the install nothing plugged into those).

Proxmox installed - no problems - from a WiFi computer I was able to access the GUI (https://192.168.10.252:8006). Then part of the Proxmox setup/install guide advised me to create a vmbr0 and point it to the port chosen during setup. I did this.

I then installed pfSense using the instructions I found on Netgate. I had me create 2 vmbr# for WAN and LAN. Done - installed...setup the ports in pfSense to use vtnet0 and vtnet1 (they were the vmbr1 and vmbr2 I had created. I then changed the LAN port from 192.168.1.1 (the default) to 192.168.10.254 (what I have used for pfSense since it was on its own box).

I then shut down the modem, put the ORBI back into AP mode - and moved the modem cable into the port on the 4-port that I created the vmbr1 for, then a cable from the vmbr2 LAN into the WAN port on the ORBI (again in AP mode) and rebooted it.

Modem powered on & pfSense was restarted and the WAN got an address from the ISP (both IPv4 and v6). I then configured the LAN to Track Interface >> WAN and it too got an IPv6 (in another range 2601: -- the WAN was 2001: ). There was still a cable from another port in the ORBI to the on-board NIC -- and since ORBI no longer doing DHCP - I had to set it in /etc/network/interfaces for the Proxmox.

This is the way it is setup now -- and it IS WORKING - but I am trying to figure out "WHY" I have to use 3 NIC ports to do this. When I can do the same thing in ESXi with only 2 physical NICs.

I know it has to be something stupid-simple, that I am simply not grasping (due to the way that ESXi names things and probably handles the vSwitches and Port Groups).

At some point, I may do VLANs, but right now I just doing a single IPv4 (192.168.10.xxx) and will let pfSense handle the IPv6 in "Track" WAN mode.

My issue is that when I re-build this or attempt to - I either have to have the ORBI in Router mode - or build another stand-alone pfSense (which is not a problem) -- or I will have no Internet until I get it all set back up - and then do not know what to do with that vmbr0 I created to get started.

Maybe this will help -- I put comments on everything....when I set it all up:

Gblenn · Sep 23, 2022, 3:16 PM

So, here's how I have set it up...

I'm also using a 4 port card like you, with one difference. I use the onboard NIC only for Proxomox (web UI and SSH), nothing else. Why... because I can... and I think it looks cleaner.

I have dedicated two NICs on the PCI-card to pfsense, which I happen to do using IOMMU, but that is not necessary.
The other two ports on the 4 port card I use for any other VM's on Proxmox.

So when I am setting things up, the Proxmox machine is connected to my LAN only with the onboard NIC. And during the setup of pfsense I typically would use two computers. One "master" where I have internet access throughout the process and access the Proxmox UI to create and configure the pfsense VM, assigning ports and running through the initial setup process from the Console window in Proxmox.

Then I have a laptop which I connect to pfsense LAN (vmbr2) in your case, looking at that earlier picture. Once I have gone through the setup on the master pc in the Proxmox Console, it will provide an IP to the laptop and I can access the web UI to finalize the configuration. To do that I usually load a configuration backup that I know is working. After restarting pfsense I would connect also the WAN port (disconnecting whatever other router/fw I happened to be using, in your case ORBI).

It's no more complicated to do this with only two NICs in use. Then you would have to assign the pfsense LAN to the same port you use to access Proxmox UI. They will be recognized by the switches from their different MAC addresses. And you could even put all of your VM's on that same port as well, leaving all 4 NIC's on the PCI card to be failover or loadbalancing WAN ports for pfsense... if you like...

bearhntr · Sep 23, 2022, 3:55 PM

@gblenn

AWESOME-- and Thanks.

So I have a new system that I want to move the existing pfSense to....and it will do IOMMU - the current one will not. The new system has a 4-port NIC card installed as well. I have already setup the Proxmox installed there (no VMs yet) with the IOMMU after changing the GUB files and adding the Filters. It appears to be working as per the guide I was following.

So following your example, if I "do NOT" use the ORBI in Router mode but as an AP/HUB and have another stand-alone (or existing pfSense install) to do the setup. Sounds like I do not need to create a vmbr0 (on the NEW Proxmox on-board NIC)

Here is how I picture it:

OLD PFSENSE (on old Proxmox):
WAN Port on 4-port0 >> To ISP Modem
LAN Port on 4-port3 >> to ORBI WAN (they call it MODEM) (in AP mode)

NEW PFSENSE (on IOMMU box):
MGMT Port using On-Board >> to ORBI (in AP mode) - just until all setup.
WAN Port on 4-port0 (IOMMU) >> Nothing Pugged in
LAN Port on 4-port3 (IOMMU) >> to ORBI (in AP mode)

Do the install on NEW box - set all my IPs that should be STATIC

Once it is all setup and configured....

move the Cable from OLD WAN to NEW (IOMMU) WAN
plug the ORBI MODEM port into the NEW LAN port
(at this point I could basically shut off the OLD pfSense VM)
Configure the NEW pfSense

@gblenn said in pfSense on PROXMOX with HomeAssistant:

It's no more complicated to do this with only two NICs in use. Then you would have to assign the pfsense LAN to the same port you use to access Proxmox UI. They will be recognized by the switches from their different MAC addresses. And you could even put all of your VM's on that same port as well, leaving all 4 NIC's on the PCI card to be failover or loadbalancing WAN ports for pfsense... if you like...

This is where the confusion comes in - you mention ALL 4 NICs on the card are then Free. How would that be possible if one of them is the WAN and one of them is the LAN. (my head hurts) LOL

jimp · Sep 23, 2022, 4:02 PM

I was addressing the "single NIC" case you mentioned, if you have more than one NIC you don't need VLANs. If you only have one NIC you would need VLANs plus a managed switch. Both can get to the same intended result but using a single NIC with only VLANs will perform poorly compared to using separate NICs.

Both ESX and Proxmox can operate on a single NIC or multiple, it's all in how you setup the networking in the Hypervisor as I mentioned.

It's a best practice to have the management isolated on its own NIC but not required. You can attach a VM to vmbr0 like any other vmbr interface. If you put the pfSense LAN on the same Proxmox bridge as the Proxmox management they'd both be on the same network, which is probably what you want there.

(Note, I would hardcode a static address in Proxmox otherwise you get into a bad chicken-and-egg scenario if it wants to pull a DHCP address for Proxmox from the pfSense VM... But that's the same for ESX as well.)

Gblenn · Sep 23, 2022, 5:41 PM

The process you are suggesting looks perfectly fine to me. The only thing I'm wondering about is your static IP's? Does your ISP not provide DHCP for your WAN connection? Pfsense will of course have 192.168.1.1 but that is set from within itself, not from Proxmox. Another question is why ports 0 and 3 on the new one, you are free to change now, so why not two adjacent ports? Makes it easier to remember when you start playing around with other VM's if that is what you will be doing?

And, about running other VM's on the same machine... I was running pfsense on my main server which also hosts Plex, NextCloud and a number of other servers. I was making a lot of changes and experimentation on that server which occasionally had me running into trouble or wanting to reboot. So that led me to dedicating another HW to pfsense and related VM's (PiHole, NtopNG, HAProxy and the likes).

Also, at step 3. "Configure the NEW pfsense", I would use the config from your existing one. Take a backup, load it in the new one, restart and depending on the NW cards you might have to go in an reassign your network interfaces from within the GUI of pfsense. That's all there is to it.

Regarding my comment on "all 4 ports free", I meant available to pfsense... and of course one of them would be used for WAN then. I actually have 3 ports used for pfsense, where one connects to an LTE Router as failover. So LAN, WAN and WAN2...

bearhntr · Sep 23, 2022, 7:51 PM

@gblenn

I will play with this after work (when I do not need Internet -- WFH here).

Then I gotta figure out why I am getting the ICMPv6 errors. I put the same RULE I had before in pfSense. I know Comcast will not do IPv6 Reverse DNS (as a residential customer) and still do not know why the browser is not doing IPv6. This has always worked.

bearhntr · Sep 25, 2022, 2:10 PM

@Gblenn @stephenw10 @Patch @jimp

Thanks again for all of your help....very much appreciated.

OK. I have put pfSense back on a stand-alone box (well the HP T620+ I had put {Proxmox on) and it is running as before. I have made a new backup and will look to put a new Proxmox install on the new HP Z240 I got for Proxmox and using all the notes from here.

Now I cannot figure out why the IPv6 test is failing - I used to get 18/20 on this test (only because COMCAST will not do an IPv6 Reverse DNS record for residential).

I have rules in pfSense for this. Odd that it is not working. But this also shows it was not tested. Maybe it is not me. :-)

stephenw10 · Sep 25, 2022, 2:57 PM

Do you see it blocked in the firewall log? What rule is blocking it?

bearhntr · Sep 25, 2022, 5:16 PM

@stephenw10

I see nothing in the logs which would be blocking it. In face the Firewall logs do not show anything that I can see ICMP.

I see these when I FILTER on ICMP - but they are all IPV4. Which still should be working too.

based on these rules:

bearhntr · Sep 25, 2022, 5:30 PM

This one seems to get a 10/10: https://test-ipv6.com/

Given that the other one shows that ICMP was not tested - I am not going to worry about it.

stephenw10 · Sep 25, 2022, 6:13 PM

ICMP is required for IPv6 but the default firewall rules should pass it:

# IPv6 ICMP is not auxiliary, it is required for operation
# See man icmp6(4)
# 1    unreach         Destination unreachable
# 2    toobig          Packet too big
# 128  echoreq         Echo service request
# 129  echorep         Echo service reply
# 133  routersol       Router solicitation
# 134  routeradv       Router advertisement
# 135  neighbrsol      Neighbor solicitation
# 136  neighbradv      Neighbor advertisement
pass  quick inet6 proto ipv6-icmp from any to any icmp6-type {1,2,135,136} ridentifier 1000000107 keep state

You probably need to filter by ipv6-icmp if it was being blocked by pfSense. However it shouldn't be possible to block the 4 icmp types as the rule is 'quick' and high up in the rules table.

Steve

bearhntr · Sep 26, 2022, 12:37 PM

@stephenw10 said in pfSense on PROXMOX with HomeAssistant:

ipv6-icmp

I get n nothing:

stephenw10 · Sep 26, 2022, 2:11 PM

My mistake it's actually logged as ICMPv6:

 	Sep 26 13:40:20 	WAN 	Default deny rule IPv6 (1000000105) 	[xxxx:yyyy:7282:101:20d:b4ff:fe0c:aed6]		[xxxx:yyyy:7282:101::2000]		ICMPv6
	Sep 26 13:40:21 	WAN 	Default deny rule IPv6 (1000000105) 	[xxxx:yyyy:7282:101:20d:b4ff:fe0c:aed6]		[xxxx:yyyy:7282:101::2000]		ICMPv6
	Sep 26 13:40:22 	WAN 	Default deny rule IPv6 (1000000105) 	[xxxx:yyyy:7282:101:20d:b4ff:fe0c:aed6]		[xxxx:yyyy:7282:101::2000]		ICMPv6

Echorequest/replies are blocked by default like that but the 4 required types should never be.

Steve

bearhntr · Sep 26, 2022, 2:11 PM

@stephenw10

Not getting any hits - unless I just put in ICMP (and the errors show v4, no v6)

stephenw10 · Sep 26, 2022, 2:21 PM

Not blocked in pfSense then. So either it just wasn't tested, as it seems to imply. Or it's blocked upstream.

Steve

bearhntr · Sep 27, 2022, 12:43 PM

@stephenw10 @jimp @Patch @Gblenn

Thanks again everyone for your help.

I am going to run this like this for a while - and take weekly backups. When I am ready to move to Proxmox - I will install fresh and then restore the last backup.

Right now I have another issue... suddenly the 4-port card that I put into the HP Z240 prevents the machine from booting. I can take it out and put into another machine and that one boots just fine. Boots all the way to Windows server 2019, is seen and all 4-ports are there.

Put it in the HP and I get 3 slow-beeps and RED power light, then 2 fast-beeps and white power light. I have a ticket with the folks at HP. It is an HP card 331T card.