@FInnishFlash Doesn't work even with bridged interfaces, and virtio

@Techniker_ctr

Regarding your installations is it UFS or ZFS, could you check top for an ARC entry (e.g., "ARC: 2048B Total, 2048B Header")?

I have three UFS pfSense instances (one bare metal, two VMs), and only one VM exhibits this issue. It displays an ARC entry in top and has zfs.ko loaded, despite being UFS.

Unloading zfs.ko temporarily removes the ARC entry, but it reappears shortly after. This behavior also occurred during OPNsense testing.

I've unloaded and renamed zfs.ko to observe the outcome.

HyperV CPU hvevent goes to 100%
Should a UFS machine have an ARC entry in top?
looking for the reason for sudden very high CPU utilization on network/zfs related process

"However this is not what is observed. What was observed is consistent with busy waiting. The storagenode process was constantly in the “uwait” process state. As if instead of pulling back, moderating bandwidth, sleeping more, storagenode instead loops at a very fast rate on system calls that fail or return a retry error, or just polls on some event or mutex, thereby consuming high CPU while effectively doing nothing (which is also observed: lower performance, lower overall IO, more missed uploads and downloads)."

Sound familiar?

/edit
Quick recap: zfs.ko loads due to a trigger in kernel like ZFS storage access. Is your storage dynamic or fixed? This instance experiences the most reads/writes, just a guess.

@viragomann
I made a mistake in my previous message, sorry about that. Unfortunately, the traffic never reaches the LAN. If it did, the VM would already be accessible.

@stephenw10 Hi, we developed a Azure custom image with Opnsense including the current Azure Agent. With that Azure backup works normally. We stay with this solution, migration from pfSense was bit of work but worthwhile! You can close this task, thank you for your effort.

@adamk11 How many and what type of NICs are you using? Are we to assume that pfsense LAN, and all VLAN's share the same physical port or do you have them separated?
If they are using the same physical port, I'm thinking traffic will go.

From VM1 to switch, back into pfsense VLAN1, out again to the switch on VLAN2, and then return back on the same interface to VM2. That would create a tromboning effect that may limit your throughput, likely to something less than 5, or?

Generally speaking though, my experience from Proxmox using X520 NICs and VirtIO, I don't see any limitations in that regard. Running iperf between two FW's (over WAN interfaces) I get 8+ Gbps. I do have NIC's passed thru (IOMMU) to firewalls, but not to VM's.

Just now tested between two VM's in different VLAN's reaching 9.33GBit/s without any problems whatsoever, both VirtIO.

The problem was the bridges. When creating the bridge for lan it should be tied to the physical network port and have a static ip address. All fixed.

For your information, the two 1:1 NATs (one NAT per IP) are configured on the same interface.

Hard to see how it could be anything else. 😞

@ethanthekiwi said in SCSI error on VM:

For me this issue was caused by thin provisioning on the virtual hard drive. I followed VMware's instructions to "inflate" the disk to thick provisioning and I stopped getting these errors.

Reply

Yeap same solution (https://techdocs.broadcom.com/us/en/vmware-cis/vsphere/vsphere/8-0/vsphere-storage-8-0/working-with-datastores-in-vsphere-storage-environment/using-datastore-browser-in-vsphere-environment.html#GUID-C371B88F-C407-4A69-8F3B-FA877D6955F8-en) worked for myself as well. :)

Just an update for my posted problem.
It turns out that it caused by unstable SDN of Proxmox.

People in Proxmox forum suggest me to use the more robust "Network Bridge".
Now, the connection between the VMs seem to be more stable.
In other words, pfSense has no problem at all.

Thanks to those who've read and replied to my post.

@seyed said in Best Practice for Connecting Physical Machines to Proxmox LAN Managed by pfSense:

Network Configuration:
vmbr0 – Proxmox management bridge (Public IP)
vmbr1 – pfSense WAN interface (Public IP)
vmbr2 – pfSense LAN interface for internal VMs

Goal:
I have two physical machines, each with public IP addresses assigned to their primary NICs. I would like to route these machines through pfSense by connecting their secondary NICs to the Proxmox LAN (vmbr2), effectively placing them behind the pfSense firewall.

What do you mean with Public IPs, especially wrt vmbr0 and your 2 physical machines? Does your ISP provide multiple IP's and are these machines not behind some firewall (other than perhaps the built in one in Proxmox)?

Proposed Solution:

The Proxmox host has two unused NICs.
I am considering connecting the secondary NICs of the physical machines to the unused NICs on the Proxmox server.
These unused NICs would be bridged to vmbr2, allowing the physical machines to communicate with pfSense and other internal resources.

This sounds like you would connect one interface to the internet and the other to your LAN, and only having the "machine" in between? Do you trust that solution? What is your intent with pfsense here?
To connect anything to the LAN side of pfsense, I'd use a physical switch rather than trying to use the switching in Proxmox. It will work but may suffer performance wise and it sure makes life more complicated...

@s0ulf3re So what you are showing are the Proxmox interfaces you have set up, right?
vmbr0 with 192.168.1.234 is your Proxmox interface, isn't it??

If you only have one port on Proxmox, I'm thinking you need to use VLANs to be able to separate pfsense LAN away from your main LAN. Otherwise you will have a DHCP and subnet conflict.

Perhaps if you can show your pfsense HW setup in Proxmox?
You have to attach vmbr0 first (as this will be WAN) and then vmbr2. For vmbr2 you add a VLAN tag in Proxmox, and then all your VM's need to have the same ID on their interfaces. Also assuming you have a VLAN capable switch attached where the same VLAN tag i TAGGED on the port.

Indeed there is no aarch64 ISO installer.

You are ending up at the UEFI shell because it's failing to boot anything else.

Do you see it trying and failing to boot the ISO image? You might have to choose to boot it.

@triks for anyone in the same boat, it ended up being NTOPNG that was filling the RAMDISK. Followed many posts but couldn't get it to save to SSD so removed it and that resolved the issue.

@SteveITS said in pfSense+ licensing on Proxmox HA cluster:

@Gblenn Yes it calculates the NDI based on detected hardware.

I haven’t tried but you might add a few extra NICs just in case for future use.

I guess the way @griffincash should do it is to wait with registration until decided on a good config.

Also you’ll need two Plus licenses for two routers.

Agree, since they are both active in a HA config. But I don't see that he should need more licenses when virtualizing vs the alternative of running two 6100s...?