Yes, you have to run it on a dedicated machine b ut it can be both the source and sink in testing throughput.

Steve

Where is that latency happening? Try pinging the WAN and LAN locally or from LAN to some local WAN target or to another interface. Can you show it's on any particular interface?

If not then it's probably just a result of running virtualised.

Steve

I would be amazed if pfSense/FreeBSD supported those ASICs. You will probably find you have only a management NIC available and all the routing/filtering will certainly be CPU only.

If you have one already it might be a fun project but that is all. And $500 is a lot for that IMO.

Steve

@phill-0 said in RT-AX89X and repurposing an old computer:

I haven't checked, but I don't think my computer runs on plutonium. lol

You'd also be amazed how long things last when surrounded by 0K space...

The difference probably won't be as large as you think. No where near the difference between having pf enabled with minimal rules and pf disabled.

Steve

Install the lcdproc package.

Steve

Thank you for your replies and suggestions @stephenw10

We did some more testing - I installed one FreeBSD 12.3 and one pfsense dev on few of the spare HP boxes we had lying around. I wanted to be sure what is the problem exactly, the netgate hardware or the OS.

As far as I can tell this is definitely something related to how FreeBSD handles these smart USB devices that have both storage and modem modules.

20220614_082622.jpg 20220614_082146.jpg 20220614_075649.jpg

For virtualized pfsense appliance this works because you can separately mount both modules (storage & modem)

2022-06-14 10_51_12-New USB Device Detected.png

Perhaps netgate hardware team could investigate this as well and and see if this could be patched somehow?

Yup, run pciconf -lv at the command line to see what those cards actually are.

Edit: Yeah they are Realtek RTL8125 so you''ll need the alt driver to use them.

Steve

The fact it shows interrupt load is not a bad thing in itself, that's where the load from pf is shown.

If your required throughput is <500Mbps then you'll probably have no problem.

Steve

@steveits Tank you..:)

You might be able to get that from dmidecode.

Steve

It's failing to negotiate the link speed with the converter for some reason.

Try a different NIC as WAN in pfSense.

Try a different media converter if you can.

You could try setting the interface to 1000baseT full duplex but that should never be required for Gigabit.

Steve

Hmm, well that looks pretty good. It at least proves the NIC is good.

I believe speedtest uses 4 connections though it could vary by server. You might test at 2 and 4 parallel streams in iperf to get a comparable result.

If you are able to test against your own server locally that will also be good test.

Steve

qlxgb is included as a module in 2.6, if should attach if it's compatible.

Do you see any attach errors in the boot log?

Steve

You can set: hw.ix.flow_control=0 as a loader variable and it will apply the default to all ix NICs.

@iptvcld said in benefits to use zfs with disk mirror:

will it cause more issues then what it is worth? As in slowness, mirror issue with data sync, etc..

No. At least not in my experience. I have several setup like that and have never seen any issues. If anything I expect it to be faster.

Steve

Use TRex: https://trex-tgn.cisco.com/

@stephenw10 Thanks for your reply. Yes there are OpenWRT images for this device, will do some testing.

Oh, that's interesting. Never seen that before. 🤔

@mercer2 said in Hardware recommendations for 40gb internet, 100gb lan:

@jamesdwi
Hi James

I found this 100g router, which I received last week

https://youtu.be/7_uLxZYYEpQ

will test to see how it goes.

in regards to saturating a 100g local link and a 40g internet link.

For this kind of speeds better to use Emerson, Nokia, Huawei routers from ISP-grade lines, or a little bit cheaper F5, Extreme, Juniper.
But not SOHO like Ubiquity, Microtik, D-Link, etc...

As a You demonstrate, money, electricity uplinks and rack space - not a big problem for You. Because this equipment are ISP-grade, anyone local distributor would be happy ship, install and give You 1-2 weeks for test, for free.

For some reason this forum won't let me post the URL's of the things I used so I made a txt file with the links I used for these steps:

links.txt

@stephenw10 Thanks, good to know it's somewhat normal. I tried that but unfortunately it didn't seem to have an effect. I guess I can either live with it or turn of BIOS redirection, unless anyone has any particularly clever ideas.

Wanted to update this post. it was the card.
I had a sas LSI pcie card handy stuck in there and did the pciconf -lv and it found it. As a last ditch effort got windows on this machine and even windows could not see it.
All these test were done with the riser installed so my stuff was working.

@phonebuff Hi, see
https://docs.netgate.com/pfsense/en/latest/troubleshooting/upgrades.html#upgrade-not-offered-library-errors

Worst case you can reinstall and restore your backup.

@hansgruber

I'll assume you have a 5000 model.
It looks like a standard intel machine , with industrial specs

b92184e1-fcf8-481e-aae8-8c4f76ecee31-image.png

It doesn't mention much about the network cards (just dual + gigabit)
But there are drivers for realtek network cards on Dell, for that model.
Realtek isn't the "best" networkadapter for pfSense, but with a little driver trickery, it might work perform reasonably well.

On the installed Ubuntu , try to run the commands :

lspci lsusb

And paste the outputs here.

Good info here
https://dl.dell.com/topicspdf/dell-edge-gateway-5000_service-manual_en-us.pdf

I would just make a pfSense USB install stick , and give it a try.

/Bingo

OS Compatibility Chart
It seems that FreeBSD 12 is well supported but with two
hints pointed to the Hardware.

A5 - sSATA (w/o RAID, AHCI mode) B2 - SAS (IR mode)

Actual pfSense 2.6 is based on FreeBSD 12.2 or 12.3
so it should be perhaps support all hardware because
the numbers are talking about FreeBSD 12.

Maybe I am more simple, load pfSense and see what happens. In general pfSense runs on any x64 CPU and will work with a fair number of NICs.

Or pfSense Plus only supports this one from NetGate:

pfSense plus comes with one driver that is supporting many
but not all cards and/or chips. If you get hands on a support
card (chip) it will running out of the box! Coding a driver means you should be also hands on a device or hardware
and such of the Intel QAT cards are often high in price!!!

So if someone is coding that driver, that should be taking
care on all available QAT things on the market he should be sorted with money or hardware for doing this.

Spend some money to the FreeBSD Project and/or support
coders with hardware for getting the maximum out.

Hi together,

Dump Switch

pfSense comes with enough LAN port and you might be
able to insert a dump switch on each, and you don´t need
VLANs, pure routing is here the entire job of the pfSense

Layer2 Switch

pfSense is sorted with one or more VLAN capable switch
and is doing the entire routing between the VLANs on top
of its other work!

Layer3 Switch

pfSense is sorted with one or more Layer3 Switches and
the switch(es) are routing the entire workload self, this free´s up your pfSense for doing other work, or you may be able to install some more packets without problems.

So this might be the first problem, for VLAN or not VLAN
usage. And the other thing is how many Volt/Watt are all the cameras are needing, so you should be looking for
two different numbers their;

Volt/Watt per port that all cameras will be getting enough per port the entire electric budget must be also covering all port
with "xyz" Watt in total!

So if your power budget total and per port is right and
you have a really let us say powerful pfSense you can
also go with dump or layer2 switches.

NETGEAR GS728TP 28 Port Gigabit Ethernet LAN PoE Switch Smart (Netzwerk Switch Managed mit 24x PoE+ 190W, 4x 1G-SFP for ~450 €

NETGEAR GS324TP PoE Switch 24 Port Gigabit Ethernet LAN Switch Smart (24x PoE+ 190W & 2x 1G-SFP, Managed Switch mit WebGUI, VLAN, IGMP, QoS, PoE Switch 19 Zoll Rack-Montage) for ~350 €

Netgear GS524PP Switch 24 Port Gigabit Ethernet LAN PoE Switch (mit 24x PoE+ 300W for ~400 €

NETGEAR JGS524PE PoE Switch 24 Port / 16 PoE Ports (100W) ports for ~200 €
12 PoE ports

Netgear (GS524UP) unmanaged but PoE++ for ~450 €
16 PoE ports

@stephenw10

looks like the issue is caused by PfblockerNG, it would not allow any WAN traffic.
ran the Pfblocker update function, which seems to correct the issue. ( at least things start work once that is run)

@AndyRH
and yes once i to the USB nic to work it did not go above 80MBs

Tim

NETGATE 1100
$189
compared to ClearFog Base
$ 200 to ~ $ 300

NETGATE 2100
$ 349
compared to ClearFog Pro
$ 200 - ~ $ 330

Netgate appliances comes with pfSense and offers
the plus variant and they are not so far away from
their price range.

The MX65 looks to be an ARM CPU so very unlikely anyone has made that work. It would require significant development work. CE will not install on it.

Steve

To add some more information, here's a MTR directly from pfsense to a VPS

pfsense.local (37.x.x.x) -> 194.x.x.x 2022-05-05T14:34:46+0100 Packets Pings Host Loss% Snt Last Avg Best Wrst StDev 1. AS??? 10.208.128.1 94.5% 1000 2.4 138.3 1.0 1405. 352.1 2. AS8657 telepac16-hsi.cprm.net 87.6% 1000 3.1 110.2 2.6 1873. 293.7 3. AS8657 tva-cr1-bu10-200.cprm.net 0.9% 1000 4.6 136.0 3.6 2727. 324.7 4. AS8657 lon1-cr1-be2.cprm.net 0.8% 1000 36.8 190.3 33.6 2627. 371.3 5. AS1299 ldn-b7-link.ip.twelve99.net 0.5% 1000 35.6 196.3 35.2 3310. 384.6 6. AS1299 ldn-bb4-link.ip.twelve99.net 33.7% 1000 35.9 230.6 35.3 3214. 404.2 7. AS1299 adm-bb4-link.ip.twelve99.net 0.7% 1000 39.8 198.2 38.7 3112. 373.2 8. AS1299 ddf-b3-link.ip.twelve99.net 1.0% 1000 49.9 197.4 42.2 3059. 375.0 9. AS1299 contabo-svc072466-ic359931.ip.twelve99-cust.net 0.9% 1000 43.1 189.0 42.9 2958. 349.6 10. AS51167 x.contaboserver.net 0.6% 1000 43.7 191.0 43.0 2859. 352.1

And here's the same but connected directly to the modem

Desktop (37.x.x.x) -> 194.x.x.x 2022-05-05T14:51:17+0100 Packets Pings Host Loss% Snt Last Avg Best Wrst StDev 1. AS??? 10.208.128.1 87.6% 1000 2.2 4.1 1.1 53.3 5.7 2. (waiting for reply) 3. AS8657 tva-cr1-bu10-200.cprm.net 0.0% 1000 4.3 5.0 3.3 50.8 3.8 4. AS8657 lon1-cr1-be2.cprm.net 0.0% 1000 34.3 34.1 33.5 81.0 2.9 5. AS1299 ldn-b7-link.ip.twelve99.net 0.0% 1000 35.7 36.3 35.2 84.1 4.6 6. AS1299 ldn-bb4-link.ip.twelve99.net 0.5% 1000 36.1 45.8 35.4 105.2 11.8 7. AS1299 adm-bb4-link.ip.twelve99.net 0.0% 1000 39.2 41.8 38.6 85.1 3.6 8. AS1299 ddf-b3-link.ip.twelve99.net 0.0% 1000 43.6 44.3 42.2 103.0 3.8 9. AS1299 contabo-svc072466-ic359931.ip.twelve99-cust.net 0.0% 1000 43.8 44.2 42.8 97.0 3.3 10. AS51167 x.contaboserver.net 0.0% 1000 44.0 45.2 43.6 96.5 3.0

@johnpoz said in Diagnosing latency spikes:

While I see how those could lead you to your conclusion.. But I take it when your device is directly connected to your modem you have a different IP with your isp. Or when you have the other edge router as well common for this IP to be different.

They get an IP in the same subnet, and the hops are the same.

I understand that it's not ideal to test this to the internet. But as I only have issues when going out to my WAN and the only variable that I changed was my firewall/router it leads me to believe there's something wrong with my hardware choices.

I'll run the tests during the weekend as to not affect my network so much during the weekdays

@stephenw10
Thank you very much for the fast answer then i will buy it and test it.

Done. Continue in the other thread.

Thanks. I've already got one though. Someone else might.
I'm not sure there's much on it we can use. What there is there is detailed in this thread, mostly here.
It would be nice to know where the LEDs are connected. I could never find them in the usual places.

Steve

Yup, those things ^ 😉

But most important is the expected throughput. So what is you current/future WAN bandwidth?

Do you need high throughput between internal interfaces? At the same time?

Steve

@stephenw10 Will do, appreciate it. I will get it uploaded and send you the link.

DMZ2 Layer2 Switch ------ Server 2 | | pfSense-------LAN----Layer3 Switch-------VLANs | | DMZ1 Layer2 Switch ----- Server1 pfSense-------LAN----Layer3 Switch-------VLANs | | DMZ Layer2 Switch -------- 2 Servers

For big concerns (large files) and routing much traffic
and on top what installed packets will be there in game too! Do you plan using IDS or IPS (inline mode) and if so
where you are want to use it. Is there one or more radius servers in game too? Is there another ids instance inside
this setup, like OSSec or so?