NIC issues and advise

Gertjan

A, ok, thanks, that explains the rapid sequential IP attribution.
Your real WAN was/is

...
110.79.143.248 -> 110.79.210.9-four
....

and the attribution of that one looks normal.

As the logs show, when your pppoe time out, and reconnects, many (like a lot) of 'packages == system processes restart.
And because you have not one WAN IP, but 3 (the two VPNs) processes get restarted even more often and faster.

No need to be a specialist to draw a simple conclusion from this :

...
Feb 14 04:10:24 pfSense tail_pfb[83338]: [pfBlockerNG] Firewall Filter Service stopped
Feb 14 04:10:24 pfSense php_pfb[84182]: [pfBlockerNG] filterlog daemon stopped
Feb 14 04:10:24 pfSense php-fpm[11175]: /rc.start_packages: The command '/usr/local/etc/rc.d/pfb_filter.sh stop' returned exit code '1', the output was 'kill: 15788: No such process'
Feb 14 04:10:24 pfSense php-fpm[11175]: [pfBlockerNG] Restarting firewall filter daemon
Feb 14 04:10:25 pfSense tail_pfb[89730]: [pfBlockerNG] Firewall Filter Service stopped
Feb 14 04:10:25 pfSense php_pfb[93021]: [pfBlockerNG] filterlog daemon stopped
Feb 14 04:10:25 pfSense lighttpd_pfb[97157]: [pfBlockerNG] DNSBL Webserver stopped
Feb 14 04:10:25 pfSense tail_pfb[97178]: [pfBlockerNG] Firewall Filter Service stopped
Feb 14 04:10:25 pfSense php_pfb[99329]: [pfBlockerNG] filterlog daemon stopped
Feb 14 04:10:25 pfSense tail_pfb[22]: [pfBlockerNG] Firewall Filter Service started
Feb 14 04:10:25 pfSense lighttpd_pfb[3769]: [pfBlockerNG] DNSBL Webserver started
Feb 14 04:10:25 pfSense tail_pfb[6878]: [pfBlockerNG] Firewall Filter Service started
Feb 14 04:10:25 pfSense php_pfb[2261]: [pfBlockerNG] filterlog daemon started
Feb 14 04:10:25 pfSense php[6957]: [pfBlockerNG] DNSBL parser daemon started
Feb 14 04:10:25 pfSense check_reload_status[425]: Rewriting resolv.conf
Feb 14 04:10:25 pfSense php_pfb[7747]: [pfBlockerNG] filterlog daemon started
...

Note that pfSense couldn't even follow the pace : a first instance of "pfb_filter.sh" (a PHP stand alone process) was already gone before it could have been signaled to stop ...

I know, I'm not very helpful here.

Your logs by themselves show me a successful reconnect.
I advise you to check if all processes that are restarted are actually 'up and running' :
Example :

dig @127.0.0.1 google.fr +short
dig @192.168.1.1 google.fr +short

If 192.168.1.1 is your pfSense LAN interface.
Etc.

stephenw10

Make sure you have set the default IPv4 gateway to PPPOE_GW in System > Routing > Gateways. Otherwise it may be switching to one of the VPNs as default which might not work.

swansense

@Gertjan said in NIC issues and advise:

@swansense

A, ok, thanks, that explains the rapid sequential IP attribution.
Your real WAN was/is

...
110.79.143.248 -> 110.79.210.9-four
....

and the attribution of that one looks normal.

As the logs show, when your pppoe time out, and reconnects, many (like a lot) of 'packages == system processes restart.
And because you have not one WAN IP, but 3 (the two VPNs) processes get restarted even more often and faster.

No need to be a specialist to draw a simple conclusion from this :

...
Feb 14 04:10:24 pfSense tail_pfb[83338]: [pfBlockerNG] Firewall Filter Service stopped
Feb 14 04:10:24 pfSense php_pfb[84182]: [pfBlockerNG] filterlog daemon stopped
Feb 14 04:10:24 pfSense php-fpm[11175]: /rc.start_packages: The command '/usr/local/etc/rc.d/pfb_filter.sh stop' returned exit code '1', the output was 'kill: 15788: No such process'
Feb 14 04:10:24 pfSense php-fpm[11175]: [pfBlockerNG] Restarting firewall filter daemon
Feb 14 04:10:25 pfSense tail_pfb[89730]: [pfBlockerNG] Firewall Filter Service stopped
Feb 14 04:10:25 pfSense php_pfb[93021]: [pfBlockerNG] filterlog daemon stopped
Feb 14 04:10:25 pfSense lighttpd_pfb[97157]: [pfBlockerNG] DNSBL Webserver stopped
Feb 14 04:10:25 pfSense tail_pfb[97178]: [pfBlockerNG] Firewall Filter Service stopped
Feb 14 04:10:25 pfSense php_pfb[99329]: [pfBlockerNG] filterlog daemon stopped
Feb 14 04:10:25 pfSense tail_pfb[22]: [pfBlockerNG] Firewall Filter Service started
Feb 14 04:10:25 pfSense lighttpd_pfb[3769]: [pfBlockerNG] DNSBL Webserver started
Feb 14 04:10:25 pfSense tail_pfb[6878]: [pfBlockerNG] Firewall Filter Service started
Feb 14 04:10:25 pfSense php_pfb[2261]: [pfBlockerNG] filterlog daemon started
Feb 14 04:10:25 pfSense php[6957]: [pfBlockerNG] DNSBL parser daemon started
Feb 14 04:10:25 pfSense check_reload_status[425]: Rewriting resolv.conf
Feb 14 04:10:25 pfSense php_pfb[7747]: [pfBlockerNG] filterlog daemon started
...

Note that pfSense couldn't even follow the pace : a first instance of "pfb_filter.sh" (a PHP stand alone process) was already gone before it could have been signaled to stop ...

I know, I'm not very helpful here.

Your logs by themselves show me a successful reconnect.
I advise you to check if all processes that are restarted are actually 'up and running' :
Example :
dig @127.0.0.1 google.fr +short
dig @192.168.1.1 google.fr +short
If 192.168.1.1 is your pfSense LAN interface.
Etc.

Thanks with the intel nic the last 2 nights the connection dropped out at 4am. My ISP must be resetting connections at this time or something.

From the little time i had tested all is good with connection from pfsense and its only lan hosts that can not reach the internet(no ICMP, http etc..).

I changed the wan to the realtek nic(not connected) and change it back again and it usually works but not this morning for some reason until i change System > Routing > Gateways to automatic

On previous versions of Pfsense i use to have a lot of issues with unbound if the connection was down for more than 30 seconds. restarting unbound would usually resolve the issue but it was a pain in the backside as at the time my connection was from a bridge mobile broadband router that couldnt hold a connection for more than 24 hours at a time.

@stephenw10 said in NIC issues and advise:

Make sure you have set the default IPv4 gateway to PPPOE_GW in System > Routing > Gateways. Otherwise it may be switching to one of the VPNs as default which might not work.

Thanks for the suggestion but it was always PPPOE_GW. I changed it to automatic this morning and it seemed to bring the connection back up. Ill leave it at automatic and see if the same issue happens again tonight.

There is possible to run a script or something when the PPPOE connection drops like this so services will restart after the connection comes back up?

stephenw10

Just saving that page would likely have brought it back by re-applying the default gateway. It should be set to PPPoE_GW though to avoid it trying to default to one of the VPNs.

swansense

@stephenw10 said in NIC issues and advise:

Just saving that page would likely have brought it back by re-applying the default gateway. It should be set to PPPoE_GW though to avoid it trying to default to one of the VPNs.

Same thing happened again last night.

your right just saving the routing page brought the connection back up.

Im going to have to figure out some way of automatically saving the routing page for me a minute or so after the PPPOE connection goes down hopefully that will save me doing it manually.

stephenw10

OK, then firstly check Diag > Routes when this next happens. Is there a valid default route shown?

If not check the system and routing logs for what happened when the PPPoE went down (and when it came back up).

swansense

@stephenw10 said in NIC issues and advise:

OK, then firstly check Diag > Routes when this next happens. Is there a valid default route shown?

If not check the system and routing logs for what happened when the PPPoE went down (and when it came back up).

Thank you. You were correct so when the connection went down the other night i checked Diag > Routes like you mentioned and it looked like there was a default route on the lan interface for some reason. by re-applying the default gateway the connection was brought back up.

Diag > Routes when No lan internet connectivity before re-applying the default gateway

Diag > Routes when everything working as expected.

Looking at the IP it was my wan address before and i had a script run at boot to set that IP as Pfsense would not allow me use a /32 as a wan address. The script was disabled long ago so im not sure why that address is popping up as a route.

Here is the script with the route commented out. I can remember how i ran this at boot to be honest so just commented out the route in the script when i changed ISPs a few years back.

cat configure-route.sh
#
#
#

#route add -net 62.210.109.1/32 -IFACE em1
#route add default 62.210.109.1
#

I can actually see that route listed but it is disabled and has been since i changed ISP. I just deleted it there now so hopefully that will resolve the issue unless you have any other suggestions?

stephenw10

Ah, some custom script!

Well I would try to find where that is run from and remove it if you can. There is no reason to have that present still.

swansense

@stephenw10

I checked /usr/local/etc/rc.d/ and its not in there.

I must of deleted it so it no longer runs at boot. I deleted the route instead of just disabled it from system default gateway so hopefully that will do the trick. Also the script had all the commands commented out so even if it does run it doesnt actually add the route.

Hopefully this will resolve it but ill find out soon enough i suppose.

swansense

it happened again and the 62.210 route was in dia > status even though it was deleted from system > routing.

i am greping through the file system to see where i can find the script that runs adding the route but i am certain that script only exists in one location and the route commands are commented out. im also nearly certain this script only runs at boot so not sure why it pops up after connection goes down.

other than /usr/local/etc/rc.d/ can you think of anywhere else a script/command can be ran at boot?

stephenw10

It could be in a shellcmd:
https://docs.netgate.com/pfsense/en/latest/development/boot-commands.html?highlight=shellcmd#shellcmd-option

swansense

@stephenw10 said in NIC issues and advise:

It could be in a shellcmd:
https://docs.netgate.com/pfsense/en/latest/development/boot-commands.html?highlight=shellcmd#shellcmd-option

thanks i managed to find it and remove it.

issue has not happened in a week or so now.

Next step is to figure out the realtek nic issue.