Ah, nice so it probably was a multimode/singlemode mismatch. Good result! Steve

So I managed to solve this with a BIOS update in the end. Part of the update process was clearing the CMOS so I had to change a bunch of settings back. The only setting I know was different that I chose to leave default this time was the ACPI HPET Table option. Previously it was enabled, now it's disabled. I don't really see how this would affect performance to the extent I was seeing, so I suspect it was something in the BIOS updates that solved the issue. Also MSI interrupts are definitely slower than MSI-X, they took me down to 50mbit. For the next sod that goes googling this, the Motherboard was an ASRock N68-S3 UCC. Initial BIOS version was 1.4, updated to 1.6. I'm now running all defaults except that I've restricted my queues to 1 per NIC with the following in /boot/loader.conf.local hw.igb.num_queues=1 I'm doing this because I have a dual core CPU and 4 NICs, so I'm trying to reduce the amount of context switching. It may work fine as a default, but after the nightmare of getting it to this point I'm just going to leave it be.

@johnminaa said in 4g modem info: I will also recommend Huawei E398 LTE USB Modem. I never owned, used or recommended it. Personally I would recommend Huawei E3372H as a replacement. Fortunately some people already posted here some receipts on using it in the different modes.

@mats thanks for the reply and please forgive the long delay in responding. Thanks for the advice regarding the electrical side. Have a APC UPS powering the cable modem. The coax feeding the cable modem is run thru a surge suppressor . The pfSense box is powered by a separate APC UPS. Anything else I should consider?

Thank you gentlemen. I appreciate your time and feedback.

Good decision!

@stephenw10 around 24 x 10G ports

Ah, interesting. I have seen that with an incompatible SFP module but never with just the connection like that. But yes in that case simply removing the module and rebooting was not sufficient to clear the state in the NIC. It had to be completely powered down to see the correct module again. Steve

Thanks @stephenw10 for the insight, I really appreciate it. I won't waste any more time on this.

I've recently bought an Intel Pro/1000 too to test pfSense on virtual machine (Proxmox)

Ok, you will need to change it's mode. For example: http://www.draisberghof.de/usb_modeswitch/bb/viewtopic.php?t=1226 It would be better to permanently switch it which might be possible from it's interface in Windows or Mac. Otherwise if you can switch it to device ID 1C05 it should be detected: https://github.com/pfsense/FreeBSD-src/blob/ff7d4801f1b88de656e028209818ff005e8a1353/sys/dev/usb/usbdevs#L2437 Steve

If you boot an image that does not use a serial console by default you won't see anything unless you have the VGA header hooked up. The XTM5 does not have a UEFI bios. It you try to boot an image that requires UEFI it won't boot. Are you trying to boot USB? Do you know the pfSense installed boots USB, you've made the required BIOS changes? Steve

You might have more luck with a Broadcom NIC and patched FreeBSD driver: http://www.dslreports.com/forum/r32230041-Internet-Bypassing-the-HH3K-up-to-2-5Gbps-using-a-BCM57810S-NIC I have no way of testing that myself.... Steve

so I fixed this. I plugged the modem into a win box, connected to the serial port then AT+ZCDRUN=E this enters the modem into the download mode disabling the drives altogether. (AT+ZCDRUN=8 did NOT work for me) this is being saved in the nvram. plugged it back in, worked right away ! -- ugen0.2: <ZTE,Incorporated T-Mobile Rocket 4G> at usbus0 u3g0 on uhub0 u3g0: <ZTE,Incorporated T-Mobile Rocket 4G, class 0/0, rev 2.00/0.00, addr 30> on usbus0 u3g0: Found 3 ports.

Thank you both of you for your advices. Actually this server has either broadcom or intel NIC I concluded the study by recommending choosing the one with Intel NIC. I hope there will be no problem during the installation and the configuration.

@stephenw10 100% agree. That is literally the predicament I am in. For now, I VLAN to a PC who has a top of the line CPU in it (overkill really) which would probably do gigabit OpenVPN, but given my internet connection is 100/40, the box above seems like it will be perfect for the cupboard.

look this https://www.pcengines.ch/apu2.htm

That doesn't seem way out. [2.4.5-DEVELOPMENT][root@2220.stevew.lan]/root: sysctl dev.cpu.0.temperature dev.cpu.0.temperature: 57.0C That's at idle pretty much. Steve

Sadly not at least on the 1u version. The only fan was a cpu blower designed to pull from the cpu and motherboard and push the heat out which you can clock down but still noisey. I did buy some 1u silent Gelid fans as supermicro are always generous with connecting fans but still did not work well. The other issue is the very noisy (although labelled as silent) 1u PSU. I've been on the periphery with pfsense for a while now so did not know that "aes-ni must have compatibility" was pushed back. I went out last year and dropped £200 on one of the fanless and silent qotom boxed. ALthough am very happy with it as against running a vm. Cheers

Only thing I found was this: [image: 1549998021994-selection_573.png] But they may have updated it and kept the numbering. Steve