Inter-Device Connectivity Issues on pfSense
-
Yup seems good.
Something is rejecting the pings on those hosts.
-
@AG23 When you set up a bridge, using an intel NIC, pfSense puts interrupts on the same CPU thread.
If you try some TCP/IP protocols from differing interfaces from the bridge, the interrupt to create the packet and/or state may not work properly.
Try, in shell, to verify this with:
sysctl -a | grep interrupt
Also dmesg | grep msi
There is a lot of stuff going on with a multicore cpu.
Also, with that bridge, an ISP may try and send you spanning tree protocol and with STP, and you could have loop detection errors with switches depending on vendor. Maybe I could advise using netgate's L2 firewall. And a /23 subnet for communicating between two different subnets.
Also, some realtek NICs use DPKG and try to prioritize some UDP packets before others in their chipsets. Good luck.
-
@AG23 also there are packet filters on bridge interfaces and member interfaces and a default deny rule could possibly be blocking your pings.
-
https://man.freebsd.org/cgi/man.cgi?query=if_bridge&sektion=4&apropos=0&manpath=FreeBSD+10.1-RELEASE
https://youtu.be/XoLPGH4awKc?si=zLdYHiHUouDkxScT
https://docs.netgate.com/pfsense/en/latest/bridges/create.html
Careful with creating bridges. You may end up needing way better hardware to manage everything.
at 16:40 in his video he says "we don't want to allow spanning tree to just happen, but it will" and if your ISP starts sending it to you, you may need to call them.
-
@stephenw10: Thanks for the help! I will continue configuring the router. Once I plug it into my network, I can perform more extensive testing.
@HLPPC:
/root: sysctl -a | grep interrupt
igc0: Using MSI-X interrupts with 5 vectors
igc1: Using MSI-X interrupts with 5 vectors
igc2: Using MSI-X interrupts with 5 vectors
igc3: Using MSI-X interrupts with 5 vectors
igc4: Using MSI-X interrupts with 5 vectors
igc5: Using MSI-X interrupts with 5 vectors
atrtc0: Can't map interrupt.
vm.stats.vm.v_interrupt_free_min: 2
hw.bxe.interrupt_mode: 2
hw.ix.max_interrupt_rate: 31250
hw.igc.max_interrupt_rate: 20000
hw.em.max_interrupt_rate: 8000
hw.cxgbe.interrupt_types: 7
dev.igc.5.interrupts.rx_desc_min_thresh: 0
dev.igc.5.interrupts.asserts: 1
dev.igc.4.interrupts.rx_desc_min_thresh: 0
dev.igc.4.interrupts.asserts: 57662
dev.igc.3.interrupts.rx_desc_min_thresh: 0
dev.igc.3.interrupts.asserts: 1
dev.igc.2.interrupts.rx_desc_min_thresh: 0
dev.igc.2.interrupts.asserts: 1
dev.igc.1.interrupts.rx_desc_min_thresh: 0
dev.igc.1.interrupts.asserts: 60463
dev.igc.0.interrupts.rx_desc_min_thresh: 0
dev.igc.0.interrupts.asserts: 1/root: dmesg | grep msi
dmseg: Command not found.
/root: dmesg
dmseg: Command not found.
/root: which dmesg
dmseg: Command not found.I did have packet logging enabled for the GUEST and BRIDGE interfaces, disabled it now with no difference in ping results are still 'Request timed out.'.
My ISP provided me with a 'Huawei OptiXstar HN8010Ts-20' which has 1x SC/APC and 1x 10GB RJ45. My plan is to connect the pfSense device directly to the Huawei device. From my ISP community forum, I seen how to connect my own router directly and i will need to configure VLAN 300 on igc0 (WAN).
I'm somewhat familiar with Spanning Tree Protocol (STP) and the problems that can arise when bridging ports and creating loops but I hadn't considered that my ISP could cause this problem. I will investigate if others have encountered these issues in my ISP community forum and keep this in mind if i run into further problems.
-
-
/root: dmesg | grep MSI
igc0: Using MSI-X interrupts with 5 vectors
igc1: Using MSI-X interrupts with 5 vectors
igc2: Using MSI-X interrupts with 5 vectors
igc3: Using MSI-X interrupts with 5 vectors
igc4: Using MSI-X interrupts with 5 vectors
igc5: Using MSI-X interrupts with 5 vectors
ig4iic0: Using MSII don't have in-depth knowledge about interrupt handling. So I'm not sure how to interpret this or whether it could cause performance issues, either in combination with a bridge interface or generally with a 2.5GB network.
-
@AG23 I suppose I was wrong about searching interrupts although I swear I found them before. Regardless, here is the correct search:
sysctl -ad | grep link_irq
On my bridge the output is:
dev.igc.x.link_irqAnd all of the bridged interfaces are on the same IRQ. Somewhere I could probably find which thread it is.
Had to boot some old equipment for that.
remove the d from -ad to drop descriptions -
I don't have in-depth knowledge about interrupt handling. So I'm not sure how to interpret this or whether it could cause performance issues, either in combination with a bridge interface or generally with a 2.5GB network.
Well, cpu mapping and machdep are kind of crazy sometimes. And NTP servers can slow you down by 20ms while using them or traffic shaping them.
I don't recommend mixing 2500mbps and 1gbps links because duplexes get messed with and autonegotiation can be wild with NBase-T downshifting. And mixing them can cause bufferbloat. Eliciting head of line qos features on various hardware.
Also, sometimes you may want crossover cables. And your ISP may have SD-WAN software trying to manage your bandwidth for and with you.
You may also need perfect cat5e cables or shielded cat6 or cat6a cables with appropriate electrical grounding to minimize crosstalk at those speeds, or keep all of your cables parallel and away from power sources.
With Nbase-T I have ran into issues where an uplink was 2500 mbps and a downlink was 1gbps, and in promiscuous mode or during a TCP dump, the link may even start communicating with a motherboard's built in-NIC if it is from an external vendor, causing nbase-t connections to use a gigabit mobo's drivers for kernel level packet capturing, resulting in gigabit speeds.
Also, sometimes gigabit devices and programs try to scan your network with curl and UDP packets and run into the 2500 mbps linkspeeds, especially at ISP router all in ones, causing them to have issues with POSIX and causing memory issues because the data was schmoving too fast.
Also, DNS and DNSSEC can allow for encryption but links getting slower from encryption may cause a downshift. And encryption suddenly giving up can cause data to suddenly move faster. It is a good idea to use an unmanaged switch with no web interface between your pfSense and ISP router. I found a yuanley 2500mbps unmanaged switch that forces nbase-t to stay that way. They have no weird ethertypes either.
-
https://archive.nbaset.ethernetalliance.org/wp-content/uploads/2017/05/NBASET-Downshift-WP-1217.pdf
