pfSense 2.7.0 stop working now and then
-
I haven't attempted it yet, but I'll give it a try next time. This incident occurred after a gap of 9 days since the last occurrence, so it's not happening frequently, but it's still quite frustrating. Today, both my wife and I were in Teams meetings when the WAN connection suddenly went down.
I see now that I have lots off in-errors... is this the reason? What can explain this?My setup looks like this;
WAN <- 10 Gbit fiber -> SFP (DIGITUS 10 Gigabit mediaconverter) RJ45 <- 2.5 Gbit Ethernet -> RJ45 (pfSense Hunsin RS34g-J4125 FNR-RS34) RJ45 <- 1 Gbps Ethernet -> LANI replaced the cat6 cable for the "2.5 Gbit Ethernet" link inbetween the media converter and the pfsense-box... but I still gets error;
-
Seeing errors like that is not necessarily an issue. It depends what sort of errors those are. There is probably more detail in the MAC stats. And the actual percentage of error packets is low.
However seeing collisions is odd. You should never see collisions on a switched segment.
-
@stephenw10 said in pfSense 2.7.0 stop working now and then:
Seeing errors like that is not necessarily an issue. It depends what sort of errors those are. There is probably more detail in the MAC stats. And the actual percentage of error packets is low.
However seeing collisions is odd. You should never see collisions on a switched segment.
What is the underlying reason for collisions? I only have two devices, the fiber/Ethernet converter and the pfSense. Is it possible that the media converter is causing the collisions, or could the issue originate from the ISP's end?
Tomas
-
As far as I know media converters like that are only media converters. They have no switching. So the segment over which collisions might occur is between the pfSense NIC and whatever is at the remote end of the fibre. It may not be a problem. It's still unexpected though.
Can you swap the WAN to a different NIC in pfSense?
-
@stephenw10 said in pfSense 2.7.0 stop working now and then:
As far as I know media converters like that are only media converters. They have no switching. So the segment over which collisions might occur is between the pfSense NIC and whatever is at the remote end of the fibre. It may not be a problem. It's still unexpected though.
Can you swap the WAN to a different NIC in pfSense?
Certainly, but in that case, I'll receive a new MAC address that will require updating with the ISP... but I'm willing to give it a try if it's worth it.
Should I use a crossover cable between the media converter and the NIC of the pfSense? Currently, I am using a Cat 6 patch cable.
My converter converts 10 Gbps SFP to 2.5 Gbps Ethernet… can the lower speed of the inbound traffic cause the collision we see?
-
You shouldn't need a cross-over cable. It shouldn't matter. Anything using 1G Ethernet or higher should be auto-MDI/X. If it did require it it would be failing completely.
Is the media-converter actually linked at 10G on the fiber side? If it has to buffer the traffic between 10G and 2.5G it could be causing problems.
You can spoof the MAC address on the WAN interface to match whatever the ISP has registered. I would definitely try swapping the WAN NICs though, however you achieve that.
The actual amount of errors and collisions shown is pretty small. I wouldn't expect that alone to cause the WAN to stop passing traffic entirely.
Steve
-
Only on incoming data:
Collision: 0,5%
Errors: 0,45%outgoing data is without error/collision...
-
Is igc0 the same NIC?
You can looks at the type of errors those are in the sysctl macstats:
sysctl dev.igc.0 -
@stephenw10
pfSense - Netgate Device ID: febe9d4d7e04acf75053*** Welcome to pfSense 2.7.0-RELEASE (amd64) on pfSense ***
WAN (wan) -> igc0 -> v4/DHCP4: 158.174.25.147/26
LAN (lan) -> igc1 -> v4: 192.168.1.1/24- Logout (SSH only) 9) pfTop
- Assign Interfaces 10) Filter Logs
- Set interface(s) IP address 11) Restart webConfigurator
- Reset webConfigurator password 12) PHP shell + pfSense tools
- Reset to factory defaults 13) Update from console
- Reboot system 14) Disable Secure Shell (sshd)
- Halt system 15) Restore recent configuration
- Ping host 16) Restart PHP-FPM
- Shell
Enter an option: 8
[2.7.0-RELEASE][admin@pfSense.dmz.se]/root: sysctl dev.igc.0
dev.igc.0.wake: 0
dev.igc.0.interrupts.rx_desc_min_thresh: 0
dev.igc.0.interrupts.asserts: 7607622
dev.igc.0.mac_stats.tso_txd: 0
dev.igc.0.mac_stats.tx_frames_1024_1522: 2502085
dev.igc.0.mac_stats.tx_frames_512_1023: 554961
dev.igc.0.mac_stats.tx_frames_256_511: 467355
dev.igc.0.mac_stats.tx_frames_128_255: 505314
dev.igc.0.mac_stats.tx_frames_65_127: 1137035
dev.igc.0.mac_stats.tx_frames_64: 941462
dev.igc.0.mac_stats.mcast_pkts_txd: 4
dev.igc.0.mac_stats.bcast_pkts_txd: 6
dev.igc.0.mac_stats.good_pkts_txd: 6108212
dev.igc.0.mac_stats.total_pkts_txd: 6108212
dev.igc.0.mac_stats.good_octets_txd: 4432696875
dev.igc.0.mac_stats.good_octets_recvd: 11661212584
dev.igc.0.mac_stats.rx_frames_1024_1522: 7380592
dev.igc.0.mac_stats.rx_frames_512_1023: 755436
dev.igc.0.mac_stats.rx_frames_256_511: 577979
dev.igc.0.mac_stats.rx_frames_128_255: 473883
dev.igc.0.mac_stats.rx_frames_65_127: 762285
dev.igc.0.mac_stats.rx_frames_64: 802790
dev.igc.0.mac_stats.mcast_pkts_recvd: 11816
dev.igc.0.mac_stats.bcast_pkts_recvd: 285482
dev.igc.0.mac_stats.good_pkts_recvd: 10752965
dev.igc.0.mac_stats.total_pkts_recvd: 11196512
dev.igc.0.mac_stats.xoff_txd: 0
dev.igc.0.mac_stats.xoff_recvd: 0
dev.igc.0.mac_stats.xon_txd: 0
dev.igc.0.mac_stats.xon_recvd: 0
dev.igc.0.mac_stats.alignment_errs: 0
dev.igc.0.mac_stats.crc_errs: 52083
dev.igc.0.mac_stats.recv_errs: 0
dev.igc.0.mac_stats.recv_jabber: 175
dev.igc.0.mac_stats.recv_oversize: 0
dev.igc.0.mac_stats.recv_fragmented: 418
dev.igc.0.mac_stats.recv_undersize: 0
dev.igc.0.mac_stats.recv_no_buff: 0
dev.igc.0.mac_stats.missed_packets: 0
dev.igc.0.mac_stats.defer_count: 0
dev.igc.0.mac_stats.sequence_errors: 0
dev.igc.0.mac_stats.symbol_errors: 0
dev.igc.0.mac_stats.collision_count: 52676
dev.igc.0.mac_stats.late_coll: 0
dev.igc.0.mac_stats.multiple_coll: 0
dev.igc.0.mac_stats.single_coll: 0
dev.igc.0.mac_stats.excess_coll: 0
dev.igc.0.queue_rx_3.rx_irq: 0
dev.igc.0.queue_rx_3.rxd_tail: 147
dev.igc.0.queue_rx_3.rxd_head: 148
dev.igc.0.queue_rx_2.rx_irq: 0
dev.igc.0.queue_rx_2.rxd_tail: 47
dev.igc.0.queue_rx_2.rxd_head: 48
dev.igc.0.queue_rx_1.rx_irq: 0
dev.igc.0.queue_rx_1.rxd_tail: 380
dev.igc.0.queue_rx_1.rxd_head: 381
dev.igc.0.queue_rx_0.rx_irq: 0
dev.igc.0.queue_rx_0.rxd_tail: 388
dev.igc.0.queue_rx_0.rxd_head: 389
dev.igc.0.queue_tx_3.tx_irq: 0
dev.igc.0.queue_tx_3.txd_tail: 319
dev.igc.0.queue_tx_3.txd_head: 319
dev.igc.0.queue_tx_2.tx_irq: 0
dev.igc.0.queue_tx_2.txd_tail: 206
dev.igc.0.queue_tx_2.txd_head: 206
dev.igc.0.queue_tx_1.tx_irq: 0
dev.igc.0.queue_tx_1.txd_tail: 192
dev.igc.0.queue_tx_1.txd_head: 192
dev.igc.0.queue_tx_0.tx_irq: 0
dev.igc.0.queue_tx_0.txd_tail: 731
dev.igc.0.queue_tx_0.txd_head: 731
dev.igc.0.fc_low_water: 32752
dev.igc.0.fc_high_water: 32768
dev.igc.0.rx_control: 71335938
dev.igc.0.device_control: 1075578433
dev.igc.0.watchdog_timeouts: 0
dev.igc.0.rx_overruns: 0
dev.igc.0.link_irq: 2
dev.igc.0.dropped: 0
dev.igc.0.eee_control: 1
dev.igc.0.itr: 488
dev.igc.0.tx_abs_int_delay: 66
dev.igc.0.rx_abs_int_delay: 66
dev.igc.0.tx_int_delay: 66
dev.igc.0.rx_int_delay: 0
dev.igc.0.rs_dump: 0
dev.igc.0.reg_dump: General Registers
CTRL 401c0641
STATUS 40780683
CTRL_EXIT 10000040Interrupt Registers
ICR 00000000RX Registers
RCTL 04408002
RDLEN 00004000
RDH 00000185
RDT 00000184
RXDCTL 02040808
RDBAL 058d8000
RDBAH 00000000TX Registers
TCTL a503f0fa
TDBAL 0589a000
TDBAH 00000000
TDLEN 00004000
TDH 000002db
TDT 000002db
TXDCTL 0201011f
TDFH 00000000
TDFT 00000000
TDFHS 00000000
TDFPC 00000000dev.igc.0.fc: 3
dev.igc.0.debug: -1
dev.igc.0.nvm: -1
dev.igc.0.iflib.rxq3.rxq_fl0.buf_size: 2048
dev.igc.0.iflib.rxq3.rxq_fl0.credits: 1023
dev.igc.0.iflib.rxq3.rxq_fl0.cidx: 148
dev.igc.0.iflib.rxq3.rxq_fl0.pidx: 147
dev.igc.0.iflib.rxq3.cpu: 3
dev.igc.0.iflib.rxq2.rxq_fl0.buf_size: 2048
dev.igc.0.iflib.rxq2.rxq_fl0.credits: 1023
dev.igc.0.iflib.rxq2.rxq_fl0.cidx: 48
dev.igc.0.iflib.rxq2.rxq_fl0.pidx: 47
dev.igc.0.iflib.rxq2.cpu: 2
dev.igc.0.iflib.rxq1.rxq_fl0.buf_size: 2048
dev.igc.0.iflib.rxq1.rxq_fl0.credits: 1023
dev.igc.0.iflib.rxq1.rxq_fl0.cidx: 381
dev.igc.0.iflib.rxq1.rxq_fl0.pidx: 380
dev.igc.0.iflib.rxq1.cpu: 1
dev.igc.0.iflib.rxq0.rxq_fl0.buf_size: 2048
dev.igc.0.iflib.rxq0.rxq_fl0.credits: 1023
dev.igc.0.iflib.rxq0.rxq_fl0.cidx: 389
dev.igc.0.iflib.rxq0.rxq_fl0.pidx: 388
dev.igc.0.iflib.rxq0.cpu: 0
dev.igc.0.iflib.txq3.r_abdications: 0
dev.igc.0.iflib.txq3.r_restarts: 0
dev.igc.0.iflib.txq3.r_stalls: 0
dev.igc.0.iflib.txq3.r_starts: 1379149
dev.igc.0.iflib.txq3.r_drops: 0
dev.igc.0.iflib.txq3.r_enqueues: 1379206
dev.igc.0.iflib.txq3.ring_state: pidx_head: 0902 pidx_tail: 0902 cidx: 0902 state: IDLE
dev.igc.0.iflib.txq3.txq_cleaned: 1387798
dev.igc.0.iflib.txq3.txq_processed: 1387838
dev.igc.0.iflib.txq3.txq_in_use: 41
dev.igc.0.iflib.txq3.txq_cidx_processed: 318
dev.igc.0.iflib.txq3.txq_cidx: 278
dev.igc.0.iflib.txq3.txq_pidx: 319
dev.igc.0.iflib.txq3.no_tx_dma_setup: 0
dev.igc.0.iflib.txq3.txd_encap_efbig: 0
dev.igc.0.iflib.txq3.tx_map_failed: 0
dev.igc.0.iflib.txq3.no_desc_avail: 0
dev.igc.0.iflib.txq3.mbuf_defrag_failed: 0
dev.igc.0.iflib.txq3.m_pullups: 0
dev.igc.0.iflib.txq3.mbuf_defrag: 0
dev.igc.0.iflib.txq3.cpu: 3
dev.igc.0.iflib.txq2.r_abdications: 0
dev.igc.0.iflib.txq2.r_restarts: 0
dev.igc.0.iflib.txq2.r_stalls: 0
dev.igc.0.iflib.txq2.r_starts: 1369418
dev.igc.0.iflib.txq2.r_drops: 0
dev.igc.0.iflib.txq2.r_enqueues: 1370001
dev.igc.0.iflib.txq2.ring_state: pidx_head: 1937 pidx_tail: 1937 cidx: 1937 state: IDLE
dev.igc.0.iflib.txq2.txq_cleaned: 1378468
dev.igc.0.iflib.txq2.txq_processed: 1378508
dev.igc.0.iflib.txq2.txq_in_use: 42
dev.igc.0.iflib.txq2.txq_cidx_processed: 204
dev.igc.0.iflib.txq2.txq_cidx: 164
dev.igc.0.iflib.txq2.txq_pidx: 206
dev.igc.0.iflib.txq2.no_tx_dma_setup: 0
dev.igc.0.iflib.txq2.txd_encap_efbig: 0
dev.igc.0.iflib.txq2.tx_map_failed: 0
dev.igc.0.iflib.txq2.no_desc_avail: 0
dev.igc.0.iflib.txq2.mbuf_defrag_failed: 0
dev.igc.0.iflib.txq2.m_pullups: 0
dev.igc.0.iflib.txq2.mbuf_defrag: 0
dev.igc.0.iflib.txq2.cpu: 2
dev.igc.0.iflib.txq1.r_abdications: 0
dev.igc.0.iflib.txq1.r_restarts: 0
dev.igc.0.iflib.txq1.r_stalls: 0
dev.igc.0.iflib.txq1.r_starts: 1982043
dev.igc.0.iflib.txq1.r_drops: 0
dev.igc.0.iflib.txq1.r_enqueues: 1983280
dev.igc.0.iflib.txq1.ring_state: pidx_head: 0816 pidx_tail: 0816 cidx: 0816 state: IDLE
dev.igc.0.iflib.txq1.txq_cleaned: 1989783
dev.igc.0.iflib.txq1.txq_processed: 1989823
dev.igc.0.iflib.txq1.txq_in_use: 41
dev.igc.0.iflib.txq1.txq_cidx_processed: 191
dev.igc.0.iflib.txq1.txq_cidx: 151
dev.igc.0.iflib.txq1.txq_pidx: 192
dev.igc.0.iflib.txq1.no_tx_dma_setup: 0
dev.igc.0.iflib.txq1.txd_encap_efbig: 0
dev.igc.0.iflib.txq1.tx_map_failed: 0
dev.igc.0.iflib.txq1.no_desc_avail: 0
dev.igc.0.iflib.txq1.mbuf_defrag_failed: 0
dev.igc.0.iflib.txq1.m_pullups: 0
dev.igc.0.iflib.txq1.mbuf_defrag: 0
dev.igc.0.iflib.txq1.cpu: 1
dev.igc.0.iflib.txq0.r_abdications: 0
dev.igc.0.iflib.txq0.r_restarts: 0
dev.igc.0.iflib.txq0.r_stalls: 0
dev.igc.0.iflib.txq0.r_starts: 1377367
dev.igc.0.iflib.txq0.r_drops: 0
dev.igc.0.iflib.txq0.r_enqueues: 1377615
dev.igc.0.iflib.txq0.ring_state: pidx_head: 1359 pidx_tail: 1359 cidx: 1359 state: IDLE
dev.igc.0.iflib.txq0.txq_cleaned: 1387186
dev.igc.0.iflib.txq0.txq_processed: 1387226
dev.igc.0.iflib.txq0.txq_in_use: 41
dev.igc.0.iflib.txq0.txq_cidx_processed: 730
dev.igc.0.iflib.txq0.txq_cidx: 690
dev.igc.0.iflib.txq0.txq_pidx: 731
dev.igc.0.iflib.txq0.no_tx_dma_setup: 0
dev.igc.0.iflib.txq0.txd_encap_efbig: 0
dev.igc.0.iflib.txq0.tx_map_failed: 0
dev.igc.0.iflib.txq0.no_desc_avail: 0
dev.igc.0.iflib.txq0.mbuf_defrag_failed: 0
dev.igc.0.iflib.txq0.m_pullups: 0
dev.igc.0.iflib.txq0.mbuf_defrag: 0
dev.igc.0.iflib.txq0.cpu: 0
dev.igc.0.iflib.override_nrxds: 0
dev.igc.0.iflib.override_ntxds: 0
dev.igc.0.iflib.use_logical_cores: 0
dev.igc.0.iflib.separate_txrx: 0
dev.igc.0.iflib.core_offset: 0
dev.igc.0.iflib.tx_abdicate: 0
dev.igc.0.iflib.rx_budget: 0
dev.igc.0.iflib.disable_msix: 0
dev.igc.0.iflib.override_qs_enable: 0
dev.igc.0.iflib.override_nrxqs: 0
dev.igc.0.iflib.override_ntxqs: 0
dev.igc.0.iflib.driver_version: 1
dev.igc.0.%parent: pci1
dev.igc.0.%pnpinfo: vendor=0x8086 device=0x15f3 subvendor=0x8086 subdevice=0x0000 class=0x020000
dev.igc.0.%location: slot=0 function=0 dbsf=pci0:1:0:0 handle=_SB_.PCI0.RP03.PXSX
dev.igc.0.%driver: igc
dev.igc.0.%desc: Intel(R) Ethernet Controller I225-V
[2.7.0-RELEASE][admin@pfSense.dmz.se]/root: -
So it looks like CRC errors:
dev.igc.0.mac_stats.crc_errs: 52083 dev.igc.0.mac_stats.recv_errs: 0 dev.igc.0.mac_stats.recv_jabber: 175 dev.igc.0.mac_stats.recv_oversize: 0 dev.igc.0.mac_stats.recv_fragmented: 418 dev.igc.0.mac_stats.recv_undersize: 0 dev.igc.0.mac_stats.recv_no_buff: 0 dev.igc.0.mac_stats.missed_packets: 0 dev.igc.0.mac_stats.defer_count: 0 dev.igc.0.mac_stats.sequence_errors: 0 dev.igc.0.mac_stats.symbol_errors: 0 dev.igc.0.mac_stats.collision_count: 52676Try disabling checksum hardware offloading in System > Advanced > Networking. You will need to reboot to apply that.
-
-
Looks promising!
no...
still errors and collisions... should I enable h/w CRC again?
My current settings;
-
Since it made no difference it likely doesn't matter. I would try swapping the NICs next. If it follows that then it's probably real incoming errors. I'd probably look at swapping out the media converter in that case though that might not be practical.
However I still don't think those errors/collisions would cause the traffic to stop entirely. At most those are symptoms of some other failure.
-
@stephenw10 said in pfSense 2.7.0 stop working now and then:
Since it made no difference it likely doesn't matter. I would try swapping the NICs next. If it follows that then it's probably real incoming errors. I'd probably look at swapping out the media converter in that case though that might not be practical.
However I still don't think those errors/collisions would cause the traffic to stop entirely. At most those are symptoms of some other failure.
I have contacted Digitus and asked them about issue, let see if they have to say about it.
-
Now I have a 2.5 Gbps switch on the LAN side, and there are no errors reported on that side. Therefore, the issue must be on the WAN side, and it appears to be related to the Digitus media converter, the only device connected in that segment. :(
-
Looks like you're still using the same NIC for the WAN though. I would try swapping the NIC assignments to be sure it's not a bad NIC.
-
@stephenw10 said in pfSense 2.7.0 stop working now and then:
Looks like you're still using the same NIC for the WAN though. I would try swapping the NIC assignments to be sure it's not a bad NIC.
Yes, i haven't change the NIC… how can I change NIC and keep the MAC-address?
-
You could spoof the MAC onto the other NIC. That should be fine since it's in a different l2 segment.
-
@stephenw10 said in pfSense 2.7.0 stop working now and then:
You could spoof the MAC onto the other NIC. That should be fine since it's in a different l2 segment.
same with another NIC... so then it must be the media converter... or?
-
Look at the error rate though. It's far far lower on igc3.
Did it still fail to pass traffic when using igc3 as WAN?
