pfSense 2.7.0 stop working now and then
-
I have an new pfSense 2.7.0 FreeBSD 14.0-CURRENT pre-installed on an mini PC with Intel(R) Celeron(R) J4125 CPU and it isnt stable.
Just today, we experienced two issues where the pfSense, while two of us were working from home, halted all incoming and outgoing traffic. In both cases, a simple reboot of the pfSense was necessary to restore the connection.
I'm wondering where I should begin my investigation into the cause behind these interruptions. It's worth noting that the GUI appears to be functioning normally during these occurrences.
BIOS Vendor: American Megatrends Inc.
Version: 5.13
Release Date: Fri Sep 16 2022
Version 2.7.0-RELEASE (amd64)
built on Wed Jun 28 03:53:34 UTC 2023
FreeBSD 14.0-CURRENTThe system is on the latest version.
Version information updated at Mon Aug 28 15:13:11 CEST 2023
CPU Type Intel(R) Celeron(R) J4125 CPU @ 2.00GHz
Current: 2000 MHz, Max: 2001 MHz
4 CPUs: 1 package(s) x 4 core(s)
AES-NI CPU Crypto: Yes (inactive)
QAT Crypto: NoHardware crypto Inactive
Kernel PTI Disabled
MDS Mitigation Inactive -
Anything logged when that happens?
Do you have multiple gateways configured? If so make sure the WAN is set as the IPv4 default in System > Routing > Gateways.
Steve
-
@stephenw10 said in pfSense 2.7.0 stop working now and then:
Anything logged when that happens?
Do you have multiple gateways configured? If so make sure the WAN is set as the IPv4 default in System > Routing > Gateways.
Steve
So from this:
To this:
-
Yes, that will prevent it selecting the LAN gateway as the default if the WAN has a glitch. Which is probably what has been happening.
However you almost certainly should not have gateway on LAN. You would only have that if you have additional downstream routers with subnets behind them pfSense has to route to.
The only other time is for the policy modem IPSec workaround.Steve
-
This post is deleted! -
You have adguard running locally?
Split-DNS is generally the best solution if you can use it. How does it fail in the cases it doesn't work?
-
This post is deleted! -
More likely the client isn't using the Adguard server. I'm not that familiar with Adguard, I'd check any logging it has to be sure it's actually seeing those failing queries.
-
This post is deleted! -
@stephenw10 said in pfSense 2.7.0 stop working now and then:
Yes, that will prevent it selecting the LAN gateway as the default if the WAN has a glitch. Which is probably what has been happening.
However you almost certainly should not have gateway on LAN. You would only have that if you have additional downstream routers with subnets behind them pfSense has to route to.
The only other time is for the policy modem IPSec workaround.Steve
I still have the issue... I found this in the log
Sep 11 15:14:30 rc.gateway_alarm 2071 >>> Gateway alarm: WAN_DHCP (Addr:158.174.25.129 Alarm:1 RTT:.102ms RTTsd:.040ms Loss:21%)
I have to reboot the pfSense to get it working again... any idea and/or solution for this?
-
The WAN stops passing traffic? It never comes back up until you reboot it?
If it happens again try disconnecting and reconnecting the WAN cable; does that also bring it back?
-
I haven't attempted it yet, but I'll give it a try next time. This incident occurred after a gap of 9 days since the last occurrence, so it's not happening frequently, but it's still quite frustrating. Today, both my wife and I were in Teams meetings when the WAN connection suddenly went down.
I see now that I have lots off in-errors... is this the reason? What can explain this?My setup looks like this;
WAN <- 10 Gbit fiber -> SFP (DIGITUS 10 Gigabit mediaconverter) RJ45 <- 2.5 Gbit Ethernet -> RJ45 (pfSense Hunsin RS34g-J4125 FNR-RS34) RJ45 <- 1 Gbps Ethernet -> LANI replaced the cat6 cable for the "2.5 Gbit Ethernet" link inbetween the media converter and the pfsense-box... but I still gets error;
-
Seeing errors like that is not necessarily an issue. It depends what sort of errors those are. There is probably more detail in the MAC stats. And the actual percentage of error packets is low.
However seeing collisions is odd. You should never see collisions on a switched segment.
-
@stephenw10 said in pfSense 2.7.0 stop working now and then:
Seeing errors like that is not necessarily an issue. It depends what sort of errors those are. There is probably more detail in the MAC stats. And the actual percentage of error packets is low.
However seeing collisions is odd. You should never see collisions on a switched segment.
What is the underlying reason for collisions? I only have two devices, the fiber/Ethernet converter and the pfSense. Is it possible that the media converter is causing the collisions, or could the issue originate from the ISP's end?
Tomas
-
As far as I know media converters like that are only media converters. They have no switching. So the segment over which collisions might occur is between the pfSense NIC and whatever is at the remote end of the fibre. It may not be a problem. It's still unexpected though.
Can you swap the WAN to a different NIC in pfSense?
-
@stephenw10 said in pfSense 2.7.0 stop working now and then:
As far as I know media converters like that are only media converters. They have no switching. So the segment over which collisions might occur is between the pfSense NIC and whatever is at the remote end of the fibre. It may not be a problem. It's still unexpected though.
Can you swap the WAN to a different NIC in pfSense?
Certainly, but in that case, I'll receive a new MAC address that will require updating with the ISP... but I'm willing to give it a try if it's worth it.
Should I use a crossover cable between the media converter and the NIC of the pfSense? Currently, I am using a Cat 6 patch cable.
My converter converts 10 Gbps SFP to 2.5 Gbps Ethernet… can the lower speed of the inbound traffic cause the collision we see?
-
You shouldn't need a cross-over cable. It shouldn't matter. Anything using 1G Ethernet or higher should be auto-MDI/X. If it did require it it would be failing completely.
Is the media-converter actually linked at 10G on the fiber side? If it has to buffer the traffic between 10G and 2.5G it could be causing problems.
You can spoof the MAC address on the WAN interface to match whatever the ISP has registered. I would definitely try swapping the WAN NICs though, however you achieve that.
The actual amount of errors and collisions shown is pretty small. I wouldn't expect that alone to cause the WAN to stop passing traffic entirely.
Steve
-
Only on incoming data:
Collision: 0,5%
Errors: 0,45%outgoing data is without error/collision...
-
Is igc0 the same NIC?
You can looks at the type of errors those are in the sysctl macstats:
sysctl dev.igc.0 -
@stephenw10
pfSense - Netgate Device ID: febe9d4d7e04acf75053*** Welcome to pfSense 2.7.0-RELEASE (amd64) on pfSense ***
WAN (wan) -> igc0 -> v4/DHCP4: 158.174.25.147/26
LAN (lan) -> igc1 -> v4: 192.168.1.1/24- Logout (SSH only) 9) pfTop
- Assign Interfaces 10) Filter Logs
- Set interface(s) IP address 11) Restart webConfigurator
- Reset webConfigurator password 12) PHP shell + pfSense tools
- Reset to factory defaults 13) Update from console
- Reboot system 14) Disable Secure Shell (sshd)
- Halt system 15) Restore recent configuration
- Ping host 16) Restart PHP-FPM
- Shell
Enter an option: 8
[2.7.0-RELEASE][admin@pfSense.dmz.se]/root: sysctl dev.igc.0
dev.igc.0.wake: 0
dev.igc.0.interrupts.rx_desc_min_thresh: 0
dev.igc.0.interrupts.asserts: 7607622
dev.igc.0.mac_stats.tso_txd: 0
dev.igc.0.mac_stats.tx_frames_1024_1522: 2502085
dev.igc.0.mac_stats.tx_frames_512_1023: 554961
dev.igc.0.mac_stats.tx_frames_256_511: 467355
dev.igc.0.mac_stats.tx_frames_128_255: 505314
dev.igc.0.mac_stats.tx_frames_65_127: 1137035
dev.igc.0.mac_stats.tx_frames_64: 941462
dev.igc.0.mac_stats.mcast_pkts_txd: 4
dev.igc.0.mac_stats.bcast_pkts_txd: 6
dev.igc.0.mac_stats.good_pkts_txd: 6108212
dev.igc.0.mac_stats.total_pkts_txd: 6108212
dev.igc.0.mac_stats.good_octets_txd: 4432696875
dev.igc.0.mac_stats.good_octets_recvd: 11661212584
dev.igc.0.mac_stats.rx_frames_1024_1522: 7380592
dev.igc.0.mac_stats.rx_frames_512_1023: 755436
dev.igc.0.mac_stats.rx_frames_256_511: 577979
dev.igc.0.mac_stats.rx_frames_128_255: 473883
dev.igc.0.mac_stats.rx_frames_65_127: 762285
dev.igc.0.mac_stats.rx_frames_64: 802790
dev.igc.0.mac_stats.mcast_pkts_recvd: 11816
dev.igc.0.mac_stats.bcast_pkts_recvd: 285482
dev.igc.0.mac_stats.good_pkts_recvd: 10752965
dev.igc.0.mac_stats.total_pkts_recvd: 11196512
dev.igc.0.mac_stats.xoff_txd: 0
dev.igc.0.mac_stats.xoff_recvd: 0
dev.igc.0.mac_stats.xon_txd: 0
dev.igc.0.mac_stats.xon_recvd: 0
dev.igc.0.mac_stats.alignment_errs: 0
dev.igc.0.mac_stats.crc_errs: 52083
dev.igc.0.mac_stats.recv_errs: 0
dev.igc.0.mac_stats.recv_jabber: 175
dev.igc.0.mac_stats.recv_oversize: 0
dev.igc.0.mac_stats.recv_fragmented: 418
dev.igc.0.mac_stats.recv_undersize: 0
dev.igc.0.mac_stats.recv_no_buff: 0
dev.igc.0.mac_stats.missed_packets: 0
dev.igc.0.mac_stats.defer_count: 0
dev.igc.0.mac_stats.sequence_errors: 0
dev.igc.0.mac_stats.symbol_errors: 0
dev.igc.0.mac_stats.collision_count: 52676
dev.igc.0.mac_stats.late_coll: 0
dev.igc.0.mac_stats.multiple_coll: 0
dev.igc.0.mac_stats.single_coll: 0
dev.igc.0.mac_stats.excess_coll: 0
dev.igc.0.queue_rx_3.rx_irq: 0
dev.igc.0.queue_rx_3.rxd_tail: 147
dev.igc.0.queue_rx_3.rxd_head: 148
dev.igc.0.queue_rx_2.rx_irq: 0
dev.igc.0.queue_rx_2.rxd_tail: 47
dev.igc.0.queue_rx_2.rxd_head: 48
dev.igc.0.queue_rx_1.rx_irq: 0
dev.igc.0.queue_rx_1.rxd_tail: 380
dev.igc.0.queue_rx_1.rxd_head: 381
dev.igc.0.queue_rx_0.rx_irq: 0
dev.igc.0.queue_rx_0.rxd_tail: 388
dev.igc.0.queue_rx_0.rxd_head: 389
dev.igc.0.queue_tx_3.tx_irq: 0
dev.igc.0.queue_tx_3.txd_tail: 319
dev.igc.0.queue_tx_3.txd_head: 319
dev.igc.0.queue_tx_2.tx_irq: 0
dev.igc.0.queue_tx_2.txd_tail: 206
dev.igc.0.queue_tx_2.txd_head: 206
dev.igc.0.queue_tx_1.tx_irq: 0
dev.igc.0.queue_tx_1.txd_tail: 192
dev.igc.0.queue_tx_1.txd_head: 192
dev.igc.0.queue_tx_0.tx_irq: 0
dev.igc.0.queue_tx_0.txd_tail: 731
dev.igc.0.queue_tx_0.txd_head: 731
dev.igc.0.fc_low_water: 32752
dev.igc.0.fc_high_water: 32768
dev.igc.0.rx_control: 71335938
dev.igc.0.device_control: 1075578433
dev.igc.0.watchdog_timeouts: 0
dev.igc.0.rx_overruns: 0
dev.igc.0.link_irq: 2
dev.igc.0.dropped: 0
dev.igc.0.eee_control: 1
dev.igc.0.itr: 488
dev.igc.0.tx_abs_int_delay: 66
dev.igc.0.rx_abs_int_delay: 66
dev.igc.0.tx_int_delay: 66
dev.igc.0.rx_int_delay: 0
dev.igc.0.rs_dump: 0
dev.igc.0.reg_dump: General Registers
CTRL 401c0641
STATUS 40780683
CTRL_EXIT 10000040Interrupt Registers
ICR 00000000RX Registers
RCTL 04408002
RDLEN 00004000
RDH 00000185
RDT 00000184
RXDCTL 02040808
RDBAL 058d8000
RDBAH 00000000TX Registers
TCTL a503f0fa
TDBAL 0589a000
TDBAH 00000000
TDLEN 00004000
TDH 000002db
TDT 000002db
TXDCTL 0201011f
TDFH 00000000
TDFT 00000000
TDFHS 00000000
TDFPC 00000000dev.igc.0.fc: 3
dev.igc.0.debug: -1
dev.igc.0.nvm: -1
dev.igc.0.iflib.rxq3.rxq_fl0.buf_size: 2048
dev.igc.0.iflib.rxq3.rxq_fl0.credits: 1023
dev.igc.0.iflib.rxq3.rxq_fl0.cidx: 148
dev.igc.0.iflib.rxq3.rxq_fl0.pidx: 147
dev.igc.0.iflib.rxq3.cpu: 3
dev.igc.0.iflib.rxq2.rxq_fl0.buf_size: 2048
dev.igc.0.iflib.rxq2.rxq_fl0.credits: 1023
dev.igc.0.iflib.rxq2.rxq_fl0.cidx: 48
dev.igc.0.iflib.rxq2.rxq_fl0.pidx: 47
dev.igc.0.iflib.rxq2.cpu: 2
dev.igc.0.iflib.rxq1.rxq_fl0.buf_size: 2048
dev.igc.0.iflib.rxq1.rxq_fl0.credits: 1023
dev.igc.0.iflib.rxq1.rxq_fl0.cidx: 381
dev.igc.0.iflib.rxq1.rxq_fl0.pidx: 380
dev.igc.0.iflib.rxq1.cpu: 1
dev.igc.0.iflib.rxq0.rxq_fl0.buf_size: 2048
dev.igc.0.iflib.rxq0.rxq_fl0.credits: 1023
dev.igc.0.iflib.rxq0.rxq_fl0.cidx: 389
dev.igc.0.iflib.rxq0.rxq_fl0.pidx: 388
dev.igc.0.iflib.rxq0.cpu: 0
dev.igc.0.iflib.txq3.r_abdications: 0
dev.igc.0.iflib.txq3.r_restarts: 0
dev.igc.0.iflib.txq3.r_stalls: 0
dev.igc.0.iflib.txq3.r_starts: 1379149
dev.igc.0.iflib.txq3.r_drops: 0
dev.igc.0.iflib.txq3.r_enqueues: 1379206
dev.igc.0.iflib.txq3.ring_state: pidx_head: 0902 pidx_tail: 0902 cidx: 0902 state: IDLE
dev.igc.0.iflib.txq3.txq_cleaned: 1387798
dev.igc.0.iflib.txq3.txq_processed: 1387838
dev.igc.0.iflib.txq3.txq_in_use: 41
dev.igc.0.iflib.txq3.txq_cidx_processed: 318
dev.igc.0.iflib.txq3.txq_cidx: 278
dev.igc.0.iflib.txq3.txq_pidx: 319
dev.igc.0.iflib.txq3.no_tx_dma_setup: 0
dev.igc.0.iflib.txq3.txd_encap_efbig: 0
dev.igc.0.iflib.txq3.tx_map_failed: 0
dev.igc.0.iflib.txq3.no_desc_avail: 0
dev.igc.0.iflib.txq3.mbuf_defrag_failed: 0
dev.igc.0.iflib.txq3.m_pullups: 0
dev.igc.0.iflib.txq3.mbuf_defrag: 0
dev.igc.0.iflib.txq3.cpu: 3
dev.igc.0.iflib.txq2.r_abdications: 0
dev.igc.0.iflib.txq2.r_restarts: 0
dev.igc.0.iflib.txq2.r_stalls: 0
dev.igc.0.iflib.txq2.r_starts: 1369418
dev.igc.0.iflib.txq2.r_drops: 0
dev.igc.0.iflib.txq2.r_enqueues: 1370001
dev.igc.0.iflib.txq2.ring_state: pidx_head: 1937 pidx_tail: 1937 cidx: 1937 state: IDLE
dev.igc.0.iflib.txq2.txq_cleaned: 1378468
dev.igc.0.iflib.txq2.txq_processed: 1378508
dev.igc.0.iflib.txq2.txq_in_use: 42
dev.igc.0.iflib.txq2.txq_cidx_processed: 204
dev.igc.0.iflib.txq2.txq_cidx: 164
dev.igc.0.iflib.txq2.txq_pidx: 206
dev.igc.0.iflib.txq2.no_tx_dma_setup: 0
dev.igc.0.iflib.txq2.txd_encap_efbig: 0
dev.igc.0.iflib.txq2.tx_map_failed: 0
dev.igc.0.iflib.txq2.no_desc_avail: 0
dev.igc.0.iflib.txq2.mbuf_defrag_failed: 0
dev.igc.0.iflib.txq2.m_pullups: 0
dev.igc.0.iflib.txq2.mbuf_defrag: 0
dev.igc.0.iflib.txq2.cpu: 2
dev.igc.0.iflib.txq1.r_abdications: 0
dev.igc.0.iflib.txq1.r_restarts: 0
dev.igc.0.iflib.txq1.r_stalls: 0
dev.igc.0.iflib.txq1.r_starts: 1982043
dev.igc.0.iflib.txq1.r_drops: 0
dev.igc.0.iflib.txq1.r_enqueues: 1983280
dev.igc.0.iflib.txq1.ring_state: pidx_head: 0816 pidx_tail: 0816 cidx: 0816 state: IDLE
dev.igc.0.iflib.txq1.txq_cleaned: 1989783
dev.igc.0.iflib.txq1.txq_processed: 1989823
dev.igc.0.iflib.txq1.txq_in_use: 41
dev.igc.0.iflib.txq1.txq_cidx_processed: 191
dev.igc.0.iflib.txq1.txq_cidx: 151
dev.igc.0.iflib.txq1.txq_pidx: 192
dev.igc.0.iflib.txq1.no_tx_dma_setup: 0
dev.igc.0.iflib.txq1.txd_encap_efbig: 0
dev.igc.0.iflib.txq1.tx_map_failed: 0
dev.igc.0.iflib.txq1.no_desc_avail: 0
dev.igc.0.iflib.txq1.mbuf_defrag_failed: 0
dev.igc.0.iflib.txq1.m_pullups: 0
dev.igc.0.iflib.txq1.mbuf_defrag: 0
dev.igc.0.iflib.txq1.cpu: 1
dev.igc.0.iflib.txq0.r_abdications: 0
dev.igc.0.iflib.txq0.r_restarts: 0
dev.igc.0.iflib.txq0.r_stalls: 0
dev.igc.0.iflib.txq0.r_starts: 1377367
dev.igc.0.iflib.txq0.r_drops: 0
dev.igc.0.iflib.txq0.r_enqueues: 1377615
dev.igc.0.iflib.txq0.ring_state: pidx_head: 1359 pidx_tail: 1359 cidx: 1359 state: IDLE
dev.igc.0.iflib.txq0.txq_cleaned: 1387186
dev.igc.0.iflib.txq0.txq_processed: 1387226
dev.igc.0.iflib.txq0.txq_in_use: 41
dev.igc.0.iflib.txq0.txq_cidx_processed: 730
dev.igc.0.iflib.txq0.txq_cidx: 690
dev.igc.0.iflib.txq0.txq_pidx: 731
dev.igc.0.iflib.txq0.no_tx_dma_setup: 0
dev.igc.0.iflib.txq0.txd_encap_efbig: 0
dev.igc.0.iflib.txq0.tx_map_failed: 0
dev.igc.0.iflib.txq0.no_desc_avail: 0
dev.igc.0.iflib.txq0.mbuf_defrag_failed: 0
dev.igc.0.iflib.txq0.m_pullups: 0
dev.igc.0.iflib.txq0.mbuf_defrag: 0
dev.igc.0.iflib.txq0.cpu: 0
dev.igc.0.iflib.override_nrxds: 0
dev.igc.0.iflib.override_ntxds: 0
dev.igc.0.iflib.use_logical_cores: 0
dev.igc.0.iflib.separate_txrx: 0
dev.igc.0.iflib.core_offset: 0
dev.igc.0.iflib.tx_abdicate: 0
dev.igc.0.iflib.rx_budget: 0
dev.igc.0.iflib.disable_msix: 0
dev.igc.0.iflib.override_qs_enable: 0
dev.igc.0.iflib.override_nrxqs: 0
dev.igc.0.iflib.override_ntxqs: 0
dev.igc.0.iflib.driver_version: 1
dev.igc.0.%parent: pci1
dev.igc.0.%pnpinfo: vendor=0x8086 device=0x15f3 subvendor=0x8086 subdevice=0x0000 class=0x020000
dev.igc.0.%location: slot=0 function=0 dbsf=pci0:1:0:0 handle=_SB_.PCI0.RP03.PXSX
dev.igc.0.%driver: igc
dev.igc.0.%desc: Intel(R) Ethernet Controller I225-V
[2.7.0-RELEASE][admin@pfSense.dmz.se]/root:
