@adamk11 How many and what type of NICs are you using? Are we to assume that pfsense LAN, and all VLAN's share the same physical port or do you have them separated? If they are using the same physical port, I'm thinking traffic will go. From VM1 to switch, back into pfsense VLAN1, out again to the switch on VLAN2, and then return back on the same interface to VM2. That would create a tromboning effect that may limit your throughput, likely to something less than 5, or? Generally speaking though, my experience from Proxmox using X520 NICs and VirtIO, I don't see any limitations in that regard. Running iperf between two FW's (over WAN interfaces) I get 8+ Gbps. I do have NIC's passed thru (IOMMU) to firewalls, but not to VM's. Just now tested between two VM's in different VLAN's reaching 9.33GBit/s without any problems whatsoever, both VirtIO.

The problem was the bridges. When creating the bridge for lan it should be tied to the physical network port and have a static ip address. All fixed.

For your information, the two 1:1 NATs (one NAT per IP) are configured on the same interface.

Hard to see how it could be anything else.

Just an update for my posted problem. It turns out that it caused by unstable SDN of Proxmox. People in Proxmox forum suggest me to use the more robust "Network Bridge". Now, the connection between the VMs seem to be more stable. In other words, pfSense has no problem at all. Thanks to those who've read and replied to my post.

@seyed said in Best Practice for Connecting Physical Machines to Proxmox LAN Managed by pfSense: Network Configuration: vmbr0 – Proxmox management bridge (Public IP) vmbr1 – pfSense WAN interface (Public IP) vmbr2 – pfSense LAN interface for internal VMs Goal: I have two physical machines, each with public IP addresses assigned to their primary NICs. I would like to route these machines through pfSense by connecting their secondary NICs to the Proxmox LAN (vmbr2), effectively placing them behind the pfSense firewall. What do you mean with Public IPs, especially wrt vmbr0 and your 2 physical machines? Does your ISP provide multiple IP's and are these machines not behind some firewall (other than perhaps the built in one in Proxmox)? Proposed Solution: The Proxmox host has two unused NICs. I am considering connecting the secondary NICs of the physical machines to the unused NICs on the Proxmox server. These unused NICs would be bridged to vmbr2, allowing the physical machines to communicate with pfSense and other internal resources. This sounds like you would connect one interface to the internet and the other to your LAN, and only having the "machine" in between? Do you trust that solution? What is your intent with pfsense here? To connect anything to the LAN side of pfsense, I'd use a physical switch rather than trying to use the switching in Proxmox. It will work but may suffer performance wise and it sure makes life more complicated...

@s0ulf3re So what you are showing are the Proxmox interfaces you have set up, right? vmbr0 with 192.168.1.234 is your Proxmox interface, isn't it?? If you only have one port on Proxmox, I'm thinking you need to use VLANs to be able to separate pfsense LAN away from your main LAN. Otherwise you will have a DHCP and subnet conflict. Perhaps if you can show your pfsense HW setup in Proxmox? You have to attach vmbr0 first (as this will be WAN) and then vmbr2. For vmbr2 you add a VLAN tag in Proxmox, and then all your VM's need to have the same ID on their interfaces. Also assuming you have a VLAN capable switch attached where the same VLAN tag i TAGGED on the port.

Indeed there is no aarch64 ISO installer. You are ending up at the UEFI shell because it's failing to boot anything else. Do you see it trying and failing to boot the ISO image? You might have to choose to boot it.

@triks for anyone in the same boat, it ended up being NTOPNG that was filling the RAMDISK. Followed many posts but couldn't get it to save to SSD so removed it and that resolved the issue.

@SteveITS said in pfSense+ licensing on Proxmox HA cluster: @Gblenn Yes it calculates the NDI based on detected hardware. I haven’t tried but you might add a few extra NICs just in case for future use. I guess the way @griffincash should do it is to wait with registration until decided on a good config. Also you’ll need two Plus licenses for two routers. Agree, since they are both active in a HA config. But I don't see that he should need more licenses when virtualizing vs the alternative of running two 6100s...?

@Gblenn I too run pfsense virtualised (proxmox) on old hardware (i7-3770S) alongside containers. I don't see any performance issues with pfsense either, in the webUI or otherwise. I don't use pfBlocker's DNSBL just the IP blocking. Whilst I understand downloading and updating DNSBL may be CPU intensive, why would that impact performance on every visit to the dashboard? Is the pfblocker widget CPU intensive with respect to building DNSBL stats counters?

With my attempts to build small images with VB, I found the easiest way to get the image to be small is to do the install on as small of a virtual disk as possible. In your case try installing pfSense on a 1.3 or 1.4GB disk. I do not know what the minimum install disk size is, but experimentation will get you there. Once there you can always restore the config you have if needed. There are VB options to reduce the disk, but in this case I suspect it will be easier to just re-install.

@jprez1980 Well., this hardware is 12 years old and considered obsolete. So what do you mean by lower priced? If you consider power consumption, this equipment is a money burner. Gut feeling it probably tops at 3Gbit routing but you should check it with iperf and two pc's connected at 10g speeds with pf in between to be sure. Of courseif we add packet filtering ids/ips etc this will be much less.