Big issue from 2.5.2 to 2.6. NIC stop working

reynold

@stephenw10 with tcpdump I noted that NIC stop receiving traffic.
It happens on every NIC

stephenw10

Ah so it appears to lock up the NIC/driver somehow.

Do you mean it causes all NICs to stop passing traffic at the same time?

Or just that any NIC you connect to will fail the same way?

reynold

@stephenw10 i connected at two nics with two devices and two ip addresses. When i try access gui everything crash on all nics. All crash at same time. I tried also opnsense. Samd resulta. I enter ip in web i press enter and nic crash. After some time crash everything also console. I need tò reboot

stephenw10

OK so:

It stops responding at all on all NICs. Even to ping.

The console remains responsive.

Can you ping out from the console? To something via the WAN perhaps?

And nothing is logged when that happens?

I really expect to see some error.

reynold

@stephenw10 Yes
At first NIC responds to ping
If i enter 192.168.1.1 in browser and i press enter it stops working an all NICs even PING
Device is able to ping itself
So appliance is able to ping 192.168.1.1. It responds
Console remain responsive
Nothing is logged, but maybe i do not able to find some log.
In shell i can not see anything.

I tried to replace RAM and PSU but the issue persists

stephenw10

Literally nothing is logged or just nothing that appears related?

Does dmesg return anything different?

You could also check the mac stats for the NIC using: sysctl dev.igb.1
See if anything there is throwing a load of errors.

reynold

@stephenw10 no error i think
it says driver version 2.5.19-fbsd and other info but no error are displayed.
Same problem if i try to make ssh access

Gertjan

@reynold said in Big issue from 2.5.2 to 2.6. NIC stop working:

Same problem if i try to make ssh access

For SSH to work, you need a working interface first.
Console access is your friend.
If you know you're way around, go look here : /var/log/
You'll find all the logs files there.
Check what's been log lines were added to the most recent updated files.

reynold

@Gertjan I have different log system filter boot.log...
What kinds of logs should I check?
How can i display log from the most recent updated?

thank you

reynold

@Gertjan system log says transmit failed network is down

stephenw10

Errors in the sysctls might look like:

dev.igb.0.watchdog_timeouts: 0
dev.igb.0.rx_overruns: 0
dev.igb.0.dropped: 0

dev.igb.0.mac_stats.coll_ext_errs: 0
dev.igb.0.mac_stats.alignment_errs: 0
dev.igb.0.mac_stats.crc_errs: 0
dev.igb.0.mac_stats.recv_errs: 0

dev.igb.0.mac_stats.sequence_errors: 0
dev.igb.0.mac_stats.symbol_errors: 0
dev.igb.0.mac_stats.collision_count: 0

Anything bigger than 0 there could be a clue.

Also interesting would be if the stats stop increasing at all when the NIC stop responding.

reynold

@stephenw10 no errors.
I noted only ring state pidx tail 027 idle
no errors
Replace psu mini stata disk and ram.
Issue persists.

reynold

@stephenw10
obviously if i downgrade to 2.5.2 everythings return working again.
I also tried installing different filesystem ufs and zfs but issue persists

stephenw10

@reynold said in Big issue from 2.5.2 to 2.6. NIC stop working:

system log says transmit failed network is down

What's the actual log entry there? And any surrounding it from the same time period?

Gertjan

@reynold

The dmesg.boot file might also contain important info.

reynold

@stephenw10 I found strange log saying
unbound /var/unbound/root.key file is corrput. removing and recreating

Gertjan

@reynold said in Big issue from 2.5.2 to 2.6. NIC stop working:

unbound /var/unbound/root.key file is corrput. removing and recreating

Next time when you reboot, watch this movie first :
How to Run a pfSense Software File System Check (5/2020) and do what is suggested.

And from now on : always power down with the GUI, as ripping out the power will 'destroy' the file system.
I presume unbound couldn't generate the files you've mentioned.
Two major reasons : disk full (which is quiet impossible if the system was just installed) or file system bad, which is quit possible.
Especially as 2.6.0 doesn't have the ZFS system, but the older UFS file system.

Btw : really ... 2.6.0. ? ok ..... and you're lucky, this forum has ten of thousands of posts about the pfSense 2.6.0 version. That's a good thing, as I don't recall any of them ....

stephenw10

A missing unbound key might be expected after hard reboot yes. None of that explains why all the NICs stop passing traffic though. I'm not aware of any issue with igb that would present like that.