@epimpin that would be awesome. as per my knowledge, it is possible to adjust the firmware of the connectx-2 cards to enable SR-IOV but this was never intended by mellanox. so I did that and switched on sr-iov in esxi but a message to reboot appeared, so i rebooted the server and after reboot, i had the exact same msg there. in case you figure out a way to enable it, it might be the solution to 10gbit HW conf mellanox connectx-2 MNPH-29D-XTR, FW: 2.9.1200 .ini file ;; Generated automatically by iniprep tool on Mon May 07 15:39:40 IDT 2012 from ./b0_hawk_gen2_464.prs ;; PRS FILE FOR Hawk ;; $Id: b0_hawk_gen2_464.prs,v 1.7.2.3 2012-04-24 12:43:10 ofirm Exp $ [PS_INFO] Name = 81Y9992 Description = Mellanox ConnectX-2 EN Dual-port 10GbE PCI-E 2.0 Adapter [ADAPTER] PSID = IBM0FC0000010 pcie_gen2_speed_supported = true silicon_rev=0xb0 adapter_dev_id = 0x6750 ;;;;; {gpio_mode1, gpio_mode0} {DataOut=0, DataOut=1} ;;;;; 0 = Input PAD ;;;;; 1 = {0,1} Normal Output PAD ;;;;; 2 = {0,Z} 0-pull down the PAD, 1-float ;;;;; 3 = {Z,1} 0-float, 1-pull up the pad ;;;;; Under [ADAPTER] section ;;;;; Integer parameter. Values range : 0x0 - 0xffffffff. gpio_mode1 = 0x80010 gpio_mode0 = 0x0b160bef gpio_default_val = 0x000e031f receiver_detect_time = 0x1e [HCA] hca_header_device_id = 0x6750 hca_header_subsystem_id = 0x0019 eth_xfi_en = true mdio_en_port1 = 0 num_pfs = 1 total_vfs = 64 sriov_en = true [IB] gen_guids_from_mac = true port1_802_3ap_kx4_ability = false port2_802_3ap_kx4_ability = false phy_type_port1 = XFI phy_type_port2 = XFI new_gpio_scheme_en = true read_cable_params_port1_en = true read_cable_params_port2_en = true eth_tx_lane_polarity_port1 = 0x0 eth_rx_lane_polarity_port1 = 0x0 eth_tx_lane_polarity_port2 = 0x0 eth_rx_lane_polarity_port2 = 0x0 eth_tx_lane_reversal_port1 = off eth_tx_lane_reversal_port2 = off eth_rx_lane_reversal_port1 = off eth_rx_lane_reversal_port2 = off ;;;;; SerDes static parameters for FixedLinkSpeed ;;;;; Under [IB] section port1_sd0_muxmain_qdr = 0x1f port2_sd0_muxmain_qdr = 0x1f port1_sd1_muxmain_qdr = 0x1f port2_sd1_muxmain_qdr = 0x1f port1_sd2_muxmain_qdr = 0x1f port2_sd2_muxmain_qdr = 0x1f port1_sd3_muxmain_qdr = 0x1f port2_sd3_muxmain_qdr = 0x1f port1_sd0_ob_preemp_pre_qdr = 0x0 port2_sd0_ob_preemp_pre_qdr = 0x0 port1_sd1_ob_preemp_pre_qdr = 0x0 port2_sd1_ob_preemp_pre_qdr = 0x0 port1_sd2_ob_preemp_pre_qdr = 0x0 port2_sd2_ob_preemp_pre_qdr = 0x0 port1_sd3_ob_preemp_pre_qdr = 0x0 port2_sd3_ob_preemp_pre_qdr = 0x0 port1_sd0_ob_preemp_post_qdr = 0x2 port2_sd0_ob_preemp_post_qdr = 0x2 port1_sd1_ob_preemp_post_qdr = 0x2 port2_sd1_ob_preemp_post_qdr = 0x2 port1_sd2_ob_preemp_post_qdr = 0x2 port2_sd2_ob_preemp_post_qdr = 0x2 port1_sd3_ob_preemp_post_qdr = 0x2 port2_sd3_ob_preemp_post_qdr = 0x2 port1_sd0_ob_preemp_main_qdr = 0x10 port2_sd0_ob_preemp_main_qdr = 0x10 port1_sd1_ob_preemp_main_qdr = 0x10 port2_sd1_ob_preemp_main_qdr = 0x10 port1_sd2_ob_preemp_main_qdr = 0x10 port2_sd2_ob_preemp_main_qdr = 0x10 port1_sd3_ob_preemp_main_qdr = 0x10 port2_sd3_ob_preemp_main_qdr = 0x10 port1_sd0_ob_preemp_msb_qdr = 0x0 port2_sd0_ob_preemp_msb_qdr = 0x0 port1_sd1_ob_preemp_msb_qdr = 0x0 port2_sd1_ob_preemp_msb_qdr = 0x0 port1_sd2_ob_preemp_msb_qdr = 0x0 port2_sd2_ob_preemp_msb_qdr = 0x0 port1_sd3_ob_preemp_msb_qdr = 0x0 port2_sd3_ob_preemp_msb_qdr = 0x0 center_mix90phase = true ext_phy_board_port1 = HAWK3 ext_phy_board_port2 = HAWK3 ;;;;; External Phy: ignore mellanox OUI checking. ;;;;; Under [IB] section ;;;;; Integer parameter. Values range : 0x0 - 0x1. ignore_mellanox_oui = 0x1 ;;;;; External Phy check GPIOs values for the 4 configurable GPIOs per port. ;;;;; every GPIO has 2 bits that can get the values "00", "01", "11" - dont check. ;;;;; Under [IB] section ;;;;; Integer parameter. Values range : 0x0 - 0xff. ext_phy_check_value_port1 = 0xff ext_phy_check_value_port2 = 0xff [PLL] lbist_en = 0 lbist_shift_freq = 3 pll_stabilize = 0x13 flash_div = 0x3 lbist_array_bypass = 1 lbist_pat_cnt_lsb = 0x2 core_f = 44 core_r = 27 ddr_6_db_preemp_pre = 0x4 ddr_6_db_preemp_main = 0x7 ddr_6_db_preemp_post = 0x0 ddr_3_dot_5_db_preemp_pre = 0x2 ddr_3_dot_5_db_preemp_main = 0x7 ddr_3_dot_5_db_preemp_post = 0x0 [FW] server spec: Xeon E5-1620v4 3,5GHz 2011-3 Supermicro X10SRA-F 4x 16GB Samsung DDR4-2133 reg. ECC Ram ESXi: 6.7.0 Update 3 (Build 19997733) I flashed ESXi from pre U1 up to latest patch after enabled SR-IOV (but not working) to see if something has changed. Nothing changed, from pre U1 to post U3 SR-IOV seems not supported, as described above. [image: 1662478310405-3ef4594c-f8c6-4c21-957c-617ebf239f78-grafik-resized.png] cannot select the mellanox card [image: 1662478392459-91cd4b53-2cdf-47cc-b12f-1b5b49990cfe-grafik.png]

Are you passing the Broadcom NICs through to pfSense? It looks like you're probably not but if you are there was an issue we've seen with that driver that required the NIC to be in promiscuous mode. That may be getting reset when pfSense is rebooted. Steve

@loser8491 Okay so there are some virtualization caveats on windows 11. Virtualization on Intel cpu's older than 6th gen will be disabled out of the box due to security concerns with previous generations. There is a workaround but requires some hackery. Check Microsoft forums on the issue. Furthermore, if you have a 6th gen Intel or later or a xen architecture amd cpu or later, check the bios settings. You have to enable virtualization in bios and also directed io if possible. (VT-x and VT-d) respectively. Then reinstall virtual box. If you have an older cpu you need to first verify it is capable of virtualization.

@patch said in pfsense on Proxmox - Help with config: @natharas An over view of your network architecture would help Is the Asus DSL-AC68U configured as a modem only (bridge mode) or a router Why is the Asus DSL-AC68U connected to the TP-Link SG2210P rather than directly to the WAN NIC on your Dell PowerEdge T420. Your Proxmox should be managed via your LAN not WAN port as should pfsense Alternatively your could configure both DELL nic in LAG and use VLANs on your L@ managed switch to separate WAN and LAN but I would not recommend starting with that Thanks for the reply It is currently configured as modem only, I do believe it can be bridged though. I've attempted connecting to the WAN NIC on the Poweredge T420 but had no luck getting a WAN IP via DHCP.

@keyser So I just setup a VM running pfSense on my Synology NAS along side my UniFi network. I'm really glad I spun up an isolated VM this way (with one of my extra public IPs) without having to eff with my production network. I just configured a VLAN-only network on my UDM and assigned it to some switch ports to test with. Man, I am absolutely loving pfSense so far! What a great product. I might end up having to buy a 6100 after all. Everything just works on the first try (DDNS, PIA, pfBlockerNG, Suricata, and ntopng). Thanks again for sharing your experiences and opinions.

@sabrielandoj If you ping from the pfSense GUI using the Ping tool with the default source (WAN IP) all devices on the 192.168.7.x network should response. If they don't, but do if you ping from other machines in this network, there must be something wrong in the network settings on one involved device. If you change the source to LAN it's expected that pings won't work, as long as the devices have no route for 192.168.6.124/25 pointing to pfSense. Responses will be sent to the default gateway.

OK, the solution was to install in a local Virtualbox on my computer with a similarly sized HDD. I then shut down the Virtualbox VM before it rebooted into pfSense after install and cloned the disk to the OVH VPS. Works like a charm! Also much faster.

@blcktape That was already in place. This issues was different than the 100% slow down all the time those settings fix.

@viragomann https://www.youtube.com/watch?v=hdoBQNI_Ab8 this video did not elaborate this issue after the installation of pfsense [image: 1659518201358-capture.png] See the comment by one of the viewers as well, we might have similar question

@laszlo said in Proxmox, pfsense, bridge: I'm bridged the second 2 interface. VLAN untagged management network the bridge, and the tagged VLANs on the interfaces. (100, 102 the first, 101 the second) Uhm... In pf you should have 1 physical interface with your 3 VLANs tagged, making 3 interfaces in pfSense. Then your ProxMox should have 1 connection to a switch, which has all three VLANs tagged on that port. Bridges in FreeBSD should be avoided.

@brianmcg Hi Everyone, I've fixed it myself. :-) I had to turn OFF "Enable Secure Boot" in VM's Security Settings. pfsense is now running.

@thimplicity OK, I got impatient with all the reboots and started the journey yesterday. I created a new VM according to the official pfSense instructions and set it up as q35 with UEFI. I decided against just rolling back the config and will configure the stuff from scratch. So far the basics that I have configured work without a hitch, but it has only been 10h. Thanks for the input!

Sure, I'll answer questions as time allows. I agree with what @Patch said previously though. You need to discover how to do a lot of this for yourself in order to learn from it. Steve

@srytryagn Running pfSense virtualized on top of another system naturally adds an additional layer to the whole system, which may offer additional vulnerabilities and possibilities for attacks. These grow up with the number of services you're running on the host. Hence I'd not recommend to run a production pfSense on a PC which you use for working, playing or any other purposes. However, if you virtualize pfSense together with other machines on a dedicated hypervisor system I'd not much concerns due to security. But Windows + Virtualbox is not really well eligible for these purposes in my opinion. For instance, my home pfSense runs virtualized on top of Linux with KVM aside from a web server to achieve better utilization of my hardware and safe cables, energy and hardware costs. The host itself is only connected to the LAN side of pfSense, so the firewall secures my whole system. The pfSense WAN is connected to the ISP modem and establishes the internet connection via PPPoE. Natting the traffic on the ISP router is basically not a security issue as long as the router works reliably. If pfSense can increase security for the VMs depends on your network design. pfSense, whether bare metal or virtualized, can segregate your (virtualized) network, you can drive this up to connect each VM to a separate network interface if you want, so that no VM can access anything without passing the firewall.

Here is the ARP table of the pfSense.[image: 1657244347547-screen-shot-2022-07-07-at-9.35.06-pm-resized.png]