pfSense on PROXMOX with HomeAssistant

bearhntr

@gblenn

For example - tried to make a new one. Here is what I see -- Only the bridges.

bearhntr

This is what I used to build my pfSense in Proxmox.

https://forum.netgate.com/topic/162083/pfsense-vm-on-proxmox-qemu-agent-installation

stephenw10

@bearhntr
So did you try adding an IP to vmbr2?

bearhntr

@stephenw10

I am not sure what you mean on this.... 'adding an IP to vmbr2'

@stephenw10 and @Gblenn

So I just found this. https://www.servethehome.com/how-to-pass-through-pcie-nics-with-proxmox-ve-on-intel-and-amd/

I did the IOMMU thing on the existing machine - HP T620+ ThinClient - and do not think that it actually likes it. In the link above it show adding the NICs as PCI Devices - not as Network Adapters.

I would like to get this all setup and working - but problem being - when I shut down the current pfSense to build a new on - - I lose Internet. I could fall back to my old ORBI as the Router and DHCP - but it really mucks up things until I go back around and reboot many things in the house.

I have a new box that has a much more powerful CPU and it appears that the IOMMU settings are working there. Where I was planning on moving the current pfSense - once I figured out this NIC thing. This new box is the same -- has a PCI Card with 4-ports and an onboard NIC.

I do not have a problem setting up the pfSense again - all over - but wanted to know if there was a better way to do it.

stephenw10

You can see that there is an IP address on vmbr0. If you edit vmbr2 you can add an IP address there too. If it's in the pfSense LAN subnet I expect to be able to use that access Proxmox. Though I should say I've never tried that and cannot test it here directly since I only have one NIC.

Edit: I was able to test that and it doesn't work. So something more would be required there.

Edit2: Actually it looks like that will work fine I just need to reboot Proxmox to apply it and can't do that right now.

Gblenn

Ok so you have pfSense up and running now with a working configuration. Do you really need to move it to the new machine? Even if you can utilize IOMMU there, you will probably not notice any difference in throughput. However the WebUI will likely be more snappy if it has a more powerful CPU. Also if you have more memory and cores you can of course boost it in that regard as well (2 GB is a bit low isn't it?).

I suppose there are two ways you could get it working on the new machine...

Make a copy of the VM from within Proxmox and restore it on the new machine: And to do that you can create a VM running Proxmox Backup Server. Add it to the Datacenter on both machines and then you backup and restore (or "move") VM's betweenr machines.
Make a full backup of your current pfSense configuration from within pfSense. Build a new VM on the new machine using the 2.6.0 ISO. Go through the basic setup and then simply do a restore and it should be up and running exactly like the previous one.

Depending on chipset on the Ethernet cards on each respective machine, you may have to go in and rename the Interfaces in pfSense after the restore.

bearhntr

@stephenw10

I could give it an address vmbr2 - it DOES have one...in pfSense that is the LAN port - and pfSense gives it 192.168.10.254. I do not know that it would make any difference.

How do you do pfSense on a machine in Proxmox without only 1 (one) NIC?

I hate to seem dense - but when I was playing at home with VMWare ESXi (the machine I had it on has just ONE NIC) and had no problems with it and with 4 VMs on there. Only thing is - none of them were pfSense (router or anything like that). I had one VM as Server 2019 and it was a Domain Controller and pfSense was on a stand-alone HP ThinClient which handled DNS and DHCP....the DC just pointed to it as the DNS Forwarder. I gave up on the DC - as I could never get IPv6 to do what I wanted...and thus that ESXi box got formatted and turned into a PLEX box.

bearhntr

@gblenn

The reason that I want to move it to the new machine - is because it has better CPU and 64GB RAM (I bought it to be a new Proxmox Host) -- then look into possibly using the HP T620+ (which "was" my pfSense box - before putting on Proxmox) for something else. Maybe some sort of HA configuration.

I want to install HA (which is on an HP T620 ThinClient) as another VM on this box. I was also looking at an OpenWRT Router (for WiFi - and get rid of the ORBI) - but do not really need a Router with pfSense. pfSense does not work well with WiFi - so I have read.

The ultimate goal to get rid of machines and make VMs out of them. Been looking into AgentDVR and some other stuff for Cameras and such too.

Patch

@bearhntr
I suggest you

Install pfsense on both of your hardware devices. That way if you break one you can then use the other one to rapidly restore internet access. This will be a useful backup in the future when you update Proxmox (occasionally IT changes do not go to plan).
Experiment with multiple VM running pfsense (only one running at a time to start with). Again it enables you to easily compare different setup options. I have a VM configured for pass through and another using Proxmox bridges. After you find the configuration you prefer, set it to start automatically on restart. The VMs you don't like as much can be deleted later.

stephenw10

@bearhntr said in pfSense on PROXMOX with HomeAssistant:

How do you do pfSense on a machine in Proxmox without only 1 (one) NIC?

In my case all the pfSense VMs there have a WAN connection to a bridge that has the one real NIC on it. Then they all have other interfaces to other bridges that don't have a NIC, they only exist internally in Proxmox.
I use that for testing pfSense not for routing my real traffic. If I wanted to have connections to two external subnets (wan and lan) I would need to use VLANs.

Steve

bearhntr

@patch

I am considering putting pfSense back on the HP T620+ like it was before I got started in this Proxmox madness

That way I can leave it until I figure out his NIC stuff on the new HP Z240 that I want to be a Proxmox host.

I just took one of my old slow-ass machines with a single on-board NIC and put ESXi 6.7U3 on it. I know this product and want to see how a pfSense on there works with only the ONE physical NIC.

bearhntr

@stephenw10

I know zilch about VLANs - but someone told me I should do something like that with my SmartHome stuff and keep it separate from my other stuff....also said I should have 2x WiFi Networks for that too (not a Guest and Main - which I already have when people visit and want to use my WiFi).

Someday I will be able to get a UniFI system here.

stephenw10

You have multiple NICs so no need VLANs. But, yes, you would use that for an access point with multiple SSIDs. They're not that complicated.

Steve

bearhntr

@stephenw10

Ok -- got the ESXI setup. While I probably did not need it, I followed this and and installed pfSense in there (only the WAN pulled an IP from my DHCP on the other running pfSense) -- no biggie. I basically wanted to see the differences. If I can grasp this part - I will have a better understand on building this in Proxmox.

https://docs.netgate.com/pfsense/en/latest/recipes/virtualize-esxi.html

SO - given that ESXi 6.7 uses Virtual Switches and Port Groups ( why can everyone not use the same names....lol )

Which of these corresponds to what in Proxmox? Again -- the box where I put ESXi - only has ONE physical NIC. It got the 192.168.10.11 from the 'other' pfSense

Gblenn

I'm guessing the Virtual switches tab where you have defined WAN and LAN are the equivalent of Linux Bridge vmbr1/2 in Proxmox. Either way you will see the device name and can assign them during the setup of pfsense. In proxmox you do that via the Console for the VM. Double click a VM, or right click ans select Console. I would imagine there is something similar in ESXi?
You would see enp1s0f0 in pfsense but the UI for ESXi doesn't seem to reveal that info?

bearhntr

@gblenn

Oh Yeah - there is a 'Physical NICs' tab in ESXi -- show you this. (just has another naming format).

jimp

Proxmox and ESXi handle VLANs very differently.

On ESXi you define a vSwitch backed by the physical NIC and set a VLAN ID on that vSwitch so it operates on just that one VLAN. Then in the guests each interface would talk on a different vswitch dedicated to different VLANs.

On Proxmox you don't get that convenience, at least with bridges. You setup a bridge to the one physical NIC and then you pick that same vmbrX interface on the guests but you manually set the VLAN ID for each network in the guest NIC configuration.

For example in my lab Proxmox setup it only has one upstream connection to my switch, and the switch is tagging all VLANs on that port:

When I set a guest VM to use different VLANs, I set the ID in its NICs:

Note how the two "external" interface here both use vmbr0 with different tag values.

For example:

I haven't messed with openvswitch (OVS) but I've read it works differently and may be closer to ESXi, but it's not as simple to work with.

bearhntr

@jimp

Thanks for the reply. I am not using VLANS at all -- I would not know where to even begin with those. Apologies for the length.

What I am trying to learn/do -- as I can do it in ESXi (but I want to use Proxmox) - is on a machine with ONE NIC (for example) -- How do I setup pfSense. I realize that the line from the cable modem has to go some place.

In my old configuration (a stand-alone pfSense box with 5 NICs - one of the NICs on the 4-port card was my LAN port, and the on-board NIC was the WAN (cable from ISP was there)). This worked great as the LAN cable went into my ORBI (which was set to AP mode) - and it handled the WiFi only The DNS/DHCP/IPv4/IPv6 was all done in pfSense.

So when I went to install Proxmox - I had put the ORBI into Router mode - until Proxmox was loaded and then pfSense on there. I had modem plugged into WAN port on ORBI and then a cable from one of its ports plugged into the on-board port on the new Proxmox box. The ORBI had a DHCP RSVP for this port to give it the IP of 192.168.10.252 (what I wanted the Proxmox to use). I did the install and during the install I chose the enp2s0 (the on-board NIC - the others were called enp1s0f0, 1, 2, 3 (the 4-port card) -- during the install nothing plugged into those).

Proxmox installed - no problems - from a WiFi computer I was able to access the GUI (https://192.168.10.252:8006). Then part of the Proxmox setup/install guide advised me to create a vmbr0 and point it to the port chosen during setup. I did this.

I then installed pfSense using the instructions I found on Netgate. I had me create 2 vmbr# for WAN and LAN. Done - installed...setup the ports in pfSense to use vtnet0 and vtnet1 (they were the vmbr1 and vmbr2 I had created. I then changed the LAN port from 192.168.1.1 (the default) to 192.168.10.254 (what I have used for pfSense since it was on its own box).

I then shut down the modem, put the ORBI back into AP mode - and moved the modem cable into the port on the 4-port that I created the vmbr1 for, then a cable from the vmbr2 LAN into the WAN port on the ORBI (again in AP mode) and rebooted it.

Modem powered on & pfSense was restarted and the WAN got an address from the ISP (both IPv4 and v6). I then configured the LAN to Track Interface >> WAN and it too got an IPv6 (in another range 2601: -- the WAN was 2001: ). There was still a cable from another port in the ORBI to the on-board NIC -- and since ORBI no longer doing DHCP - I had to set it in /etc/network/interfaces for the Proxmox.

This is the way it is setup now -- and it IS WORKING - but I am trying to figure out "WHY" I have to use 3 NIC ports to do this. When I can do the same thing in ESXi with only 2 physical NICs.

I know it has to be something stupid-simple, that I am simply not grasping (due to the way that ESXi names things and probably handles the vSwitches and Port Groups).

At some point, I may do VLANs, but right now I just doing a single IPv4 (192.168.10.xxx) and will let pfSense handle the IPv6 in "Track" WAN mode.

My issue is that when I re-build this or attempt to - I either have to have the ORBI in Router mode - or build another stand-alone pfSense (which is not a problem) -- or I will have no Internet until I get it all set back up - and then do not know what to do with that vmbr0 I created to get started.

Maybe this will help -- I put comments on everything....when I set it all up:

Gblenn

So, here's how I have set it up...

I'm also using a 4 port card like you, with one difference. I use the onboard NIC only for Proxomox (web UI and SSH), nothing else. Why... because I can... and I think it looks cleaner.

I have dedicated two NICs on the PCI-card to pfsense, which I happen to do using IOMMU, but that is not necessary.
The other two ports on the 4 port card I use for any other VM's on Proxmox.

So when I am setting things up, the Proxmox machine is connected to my LAN only with the onboard NIC. And during the setup of pfsense I typically would use two computers. One "master" where I have internet access throughout the process and access the Proxmox UI to create and configure the pfsense VM, assigning ports and running through the initial setup process from the Console window in Proxmox.

Then I have a laptop which I connect to pfsense LAN (vmbr2) in your case, looking at that earlier picture. Once I have gone through the setup on the master pc in the Proxmox Console, it will provide an IP to the laptop and I can access the web UI to finalize the configuration. To do that I usually load a configuration backup that I know is working. After restarting pfsense I would connect also the WAN port (disconnecting whatever other router/fw I happened to be using, in your case ORBI).

It's no more complicated to do this with only two NICs in use. Then you would have to assign the pfsense LAN to the same port you use to access Proxmox UI. They will be recognized by the switches from their different MAC addresses. And you could even put all of your VM's on that same port as well, leaving all 4 NIC's on the PCI card to be failover or loadbalancing WAN ports for pfsense... if you like...

bearhntr

@gblenn

AWESOME-- and Thanks.

So I have a new system that I want to move the existing pfSense to....and it will do IOMMU - the current one will not. The new system has a 4-port NIC card installed as well. I have already setup the Proxmox installed there (no VMs yet) with the IOMMU after changing the GUB files and adding the Filters. It appears to be working as per the guide I was following.

So following your example, if I "do NOT" use the ORBI in Router mode but as an AP/HUB and have another stand-alone (or existing pfSense install) to do the setup. Sounds like I do not need to create a vmbr0 (on the NEW Proxmox on-board NIC)

Here is how I picture it:

OLD PFSENSE (on old Proxmox):
WAN Port on 4-port0 >> To ISP Modem
LAN Port on 4-port3 >> to ORBI WAN (they call it MODEM) (in AP mode)

NEW PFSENSE (on IOMMU box):
MGMT Port using On-Board >> to ORBI (in AP mode) - just until all setup.
WAN Port on 4-port0 (IOMMU) >> Nothing Pugged in
LAN Port on 4-port3 (IOMMU) >> to ORBI (in AP mode)

Do the install on NEW box - set all my IPs that should be STATIC

Once it is all setup and configured....

move the Cable from OLD WAN to NEW (IOMMU) WAN
plug the ORBI MODEM port into the NEW LAN port
(at this point I could basically shut off the OLD pfSense VM)
Configure the NEW pfSense

@gblenn said in pfSense on PROXMOX with HomeAssistant:

It's no more complicated to do this with only two NICs in use. Then you would have to assign the pfsense LAN to the same port you use to access Proxmox UI. They will be recognized by the switches from their different MAC addresses. And you could even put all of your VM's on that same port as well, leaving all 4 NIC's on the PCI card to be failover or loadbalancing WAN ports for pfsense... if you like...

This is where the confusion comes in - you mention ALL 4 NICs on the card are then Free. How would that be possible if one of them is the WAN and one of them is the LAN. (my head hurts) LOL