@stephenw10
Thanks for your help and insight!

@pf-beginner said in 2-NIC box with pfSense + L3 switch:

Is there a way that I could configure pfSense to treat Port 2, Port 3, and Port 4 of the switch as separate networks (each containing only one computer) without using VLANs?

No. Or at least not any way I would consider reasonable.

You need 3 VLANs on the link between pfSense and the switch to segregate the 3 networks sharing the same cable.

It's not complex though. The simplest VLAN capable switch could do it.

Steve

@mikefromoz - thanks for the comparison and insight, this is very helpful info! I figured I would follow up as well to share my experiences with these cards. I initially bought a Chelsio T520-SO-CR in 2018, but have upgraded since to a pair of Chelsio T540-SO-CR's. I have been using these cards for about 1 - 2 years now - one of them in my pfSense firewall, the other in a Promox hypervisor. They have been rock solid - I definitely recommend them. From time to time I see them pop up in the Chelsio store - I think it's a pretty good deal for a quad port 10Gbit card:

https://store.chelsio.com/collections/current-adapters/products/t540-lp-cr-4-port-1-10gbe-low-profile-uwire-adapters-with-pci-e-x8-gen3-32kconn-sfp-connector

and to the end:
I have connected two LTE modem (dual WAN): earlyshellcmd and shellcmd - working in not correct.
Corrcect working is only "devd" mechanism!

How many vlans do you have on this lagg?

As mentioned you really have little control over which physical interface will be used. When both vlans are on the same lagg. Its quite possible for you to get a hairpin using the same physical interface for intervlan traffic.

If you want to make sure that is not part of the problem - I would move the vlans you want to test routing throughput on different physical interfaces.

Do you see full bandwidth then?

You don't need to add on your own things for Realtek on 2.5.0. If the stock drivers don't work, then use the one from pkg install realtek-re-kmod

@jeffwestozit
Quick follow-up. I managed to get it to boot once I modified the boot options. I disabled Network boot as an option in the BIOS and it finally booted into windows. I had the network cable plugged into the on-board 1Gb NIC with no cables in either of the ports on the 10Gb NIC. I will try different combinations and confirm.

Morning.
I've swapped the mSATA for a new one, reinstalled pfSense and restored my current config.

All done and seems to be working well :)

Thanks

@stephenw10 oh yea, didn't really think about the re-install case.. that argument alone overrides my OCD, as if i ever have to re-install I want to be to just grab a backup config and be back up and running. Thanks!

Thanks for all the great info everyone.

What a mess.

I think for now I'm just going to focus on the software unless you have any suggestions.

cxgb may support that. If it's this:
https://github.com/pfsense/FreeBSD-src/blob/RELENG_2_4_5/sys/dev/cxgb/cxgb_main.c#L299

There are a few Intel CX4 NICs listed in ix:
https://github.com/pfsense/FreeBSD-src/blob/RELENG_2_4_5/sys/dev/ixgbe/if_ix.c#L63

Steve

Just the normal one. The specs on that may only be suitable as a thin client for Windows but they are fine for pfSense. It's not running any differently to a standard install.

Steve

Deppends what it actually does. Most of those look like they would only block traffic that should never be present but some could certainly cause problems.

Steve

If the NICs appear in the output of pciconf -lv and are listed with a driver attached, sfxge0 and sgxge1, then they shoudl be available.
It sounds like you may have simply not assigned them yet from Interfaces > Assignments.

https://docs.netgate.com/pfsense/en/latest/interfaces/configure.html#interface-configuration

Steve

Ah, nice catch! Weird though.

If nothing is happening then nothing gets logged. A low rate system log is often a sign of good stability not a problem.

Those last logs are all dpinger starting which happens if an interface goes down and then comes back up or you apply a change to it etc.

I see in there that you have an OpenVPN gateway. Do you have the WAN gateway set as default in System > Routing > Gateways? If not it might have switched to a different default route when your WAN went down resulting in no connectivity.

I don't see that logged specifically but those logs are incomplete.

Steve

@stephenw10

The ESXI is dedicated to the setup
ProLiant DL380 G7
12 CPUs x Intel(R) Xeon(R) CPU X5650 @ 2.67GHz
143.99 GB ram
1.64 TB disk

I was hoping in the long run, that i would be able to lift up to about twice the amount of users 16k, exactly same usage profile (mobile only/ usually max 200mb per event max, over 2-3 hours)
The thought was to build two or 3 small pfsens boxes and distribute the load over 3 seperate vlans and then distributing clients across the diffeerent vlans in the WLC, so the only work the box would need to do would be giving dhcp addresses, onbording and gw.

does not go up than ~650 mbps, i have ftth 1000/300.
it works well but it has a lack on eth, doesent matter what i do.
with previous system wich is a pentium G with 2 gigabit Eth i have ~950 mbps in down all the time.
hope that helps someone.

@paint said in What is Best USB NIC 10Gbps that is supported by Pfsense?:

@johnpoz said in What is Best USB NIC 10Gbps that is supported by Pfsense?:

@paint said in What is Best USB NIC 10Gbps that is supported by Pfsense?:

You can get brocade switches pretty cheap

And what do they draw for power? While that is great if a lab and only on X hours a week.. But old enterprise gear is not power friendly ;) Just what I don't want is some switch drawing 150W ;)

ethernet cat 5 cannot deliver good performance anymore

5e can do 5gbps.. And reported even 10ge for short distances.. But when I do go 10g, it will most likely be fiber.. I just want the copper support for 2.5 and 5gig..

At some point would like to see atleast 2.5 on my APs - and sure don't want to have to rerun cable in the attic - which is 5e..

These pull much less than 150w.....

https://forums.servethehome.com/index.php?threads/brocade-icx-series-cheap-powerful-10gbe-40gbe-switching.21107/

My vote to used Brocade with fiber connection: perfect quality and stable working.