WAN periodically Rebooting

diyhouse

@diyhouse Looking in General=> Gateways...
I get the following errors...
I don't know if this is a 'red herring',.. but here it is

Aug 13 13:52:24	dpinger	19013	1_WAN_PPPOE 172.16.12.102: sendto error: 65
Aug 13 13:52:24	dpinger	19013	1_WAN_PPPOE 172.16.12.102: sendto error: 65
Aug 13 13:52:23	dpinger	19013	1_WAN_PPPOE 172.16.12.102: sendto error: 65
Aug 13 13:52:23	dpinger	19013	1_WAN_PPPOE 172.16.12.102: sendto error: 65
Aug 13 13:52:22	dpinger	19013	1_WAN_PPPOE 172.16.12.102: sendto error: 65
Aug 13 13:52:22	dpinger	19013	1_WAN_PPPOE 172.16.12.102: sendto error: 65
Aug 13 13:22:03	dpinger	19013	1_WAN_PPPOE 172.16.12.102: Alarm latency 0us stddev 0us loss 100%
Aug 13 13:22:01	dpinger	19013	send_interval 500ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 1 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% alarm_hold 10000ms dest_addr 172.16.12.102 bind_addr 109.147.50.193 identifier "1_WAN_PPPOE "
Aug 13 13:22:01	dpinger	69150	exiting on signal 15
Aug 13 13:21:55	dpinger	69150	1_WAN_PPPOE 172.16.12.102: sendto error: 65
Aug 13 13:21:54	dpinger	69150	1_WAN_PPPOE 172.16.12.102: sendto error: 65
Aug 13 13:21:54	dpinger	69150	1_WAN_PPPOE 172.16.12.102: sendto error: 65
Aug 13 13:21:53	dpinger	69150	1_WAN_PPPOE 172.16.12.102: sendto error: 65
Aug 13 13:21:53	dpinger	69150	1_WAN_PPPOE 172.16.12.102: sendto error: 65
Aug 13 13:21:52	dpinger	69150	1_WAN_PPPOE 172.16.12.102: sendto error: 65
Aug 13 13:21:52	dpinger	69150	1_WAN_PPPOE 172.16.12.102: sendto error: 65
Aug 13 13:21:51	dpinger	69150	1_WAN_PPPOE 172.16.12.102: sendto error: 65
Aug 13 13:21:50	dpinger	69150	1_WAN_PPPOE 172.16.12.102: sendto error: 65
Aug 13 13:21:50	dpinger	69150	1_WAN_PPPOE 172.16.12.102: sendto error: 65
Aug 13 13:21:49	dpinger	69150	1_WAN_PPPOE 172.16.12.102: sendto error: 65
Aug 13 13:21:49	dpinger	69150	1_WAN_PPPOE 172.16.12.102: sendto error: 65
Aug 13 13:21:48	dpinger	69150	1_WAN_PPPOE 172.16.12.102: sendto error: 65
Aug 13 13:21:48	dpinger	69150	1_WAN_PPPOE 172.16.12.102: sendto error: 65
Aug 13 13:21:47	dpinger	69150	1_WAN_PPPOE 172.16.12.102: sendto error: 65
Aug 13 13:21:47	dpinger	69150	1_WAN_PPPOE 172.16.12.102: sendto error: 65
Aug 13 13:21:46	dpinger	69150	1_WAN_PPPOE 172.16.12.102: sendto error: 65
Aug 13 13:21:46	dpinger	69150	1_WAN_PPPOE 172.16.12.102: sendto error: 65
Aug 13 13:21:45	dpinger	69150	1_WAN_PPPOE 172.16.12.102: sendto error: 65
Aug 13 13:21:45	dpinger	69150	1_WAN_PPPOE 172.16.12.102: sendto error: 65
Aug 13 13:21:44	dpinger	69150	1_WAN_PPPOE 172.16.12.102: sendto error: 65
Aug 13 13:21:44	dpinger	69150	1_WAN_PPPOE 172.16.12.102: sendto error: 65
Aug 13 13:21:43	dpinger	69150	1_WAN_PPPOE 172.16.12.102: sendto error: 65
Aug 13 13:21:43	dpinger	69150	1_WAN_PPPOE 172.16.12.102: sendto error: 65
Aug 13 13:21:42	dpinger	69150	1_WAN_PPPOE 172.16.12.102: sendto error: 65

stephenw10

Hmm, all that looks like it's caused by the PPPoE connection closing. Really feels like the server is closing it. Might need to turn up the logging to see it which is not something I've ever tried for PPP. Let me see...

diyhouse

@stephenw10
Stephen,... In desperation,... I have just tried to setup an 'original BT Modem',.. and found the credentials I was using to access the broadband servers,.. were incorrect.!!
The credentials I have been using for maybe 6yrs+.... are old,..
Rang BT support,.. and they supplied a completely different set of credentials, which are ( as they say ) in line with my updated line,

Needless to say I am running with the new line config and credentials on pfsense,.. lets see how this pans out
upload/ download is now at 60/10,. which is in line with what I recall were the speeds when i 1st went pfsense FTTC....

So lets see how this pans out,... hopeful,.. but not holding my breath..

diyhouse

@diyhouse
Since use of new credentials..1hr 7mins . none of the following error: 65's...in Gateways logs... and PPP is also silent..
fingers crossed..

Aug 13 13:52:29	dpinger	19013	1_WAN_PPPOE 172.16.12.102: sendto error: 65
Aug 13 13:52:28	dpinger	19013	1_WAN_PPPOE 172.16.12.102: sendto error: 65
Aug 13 13:52:28	dpinger	19013	1_WAN_PPPOE 172.16.12.102: sendto error: 65

stephenw10

Hmm, well that would be interesting!

As far as I know BT don't care what the login is, they use the physical link to authenticate you. Maybe that varies. I use bthomehub@btbroadband.com/1234 and it works fine.

I guess we'll see.

diyhouse

@stephenw10
Yes indeed,. I transitioned to BT back in 2013,.. ( I've just discovered ),..
at this transition point they gave me an account:-

green-light@service.btclick.com/Passwd to play with,..

which I have used ever since.

today they have said I need to use something similiar to :-N025134@hg70.btclick.com/Passwd

But as you say we shall see...

diyhouse

@diyhouse Update:..
Although runnning 'smoother',.. I only seem to be able to get run times on the WAN of around 15hrs...
Looking in system=>general log.. filtered by layer down...I have the following... is my issue related to suricata, they seem to occur and each layer down,.. but appreciate could just be a consequence of the WAN failure...

Aug 13 05:28:03	ppp	71373	[wan] IPCP: LayerDown
Aug 13 00:24:36	kernel		igb1: link state changed to DOWN
Aug 13 00:24:30	kernel		igb1: link state changed to DOWN
Aug 13 00:23:46	suricata	38573	[100852] <Error> -- error parsing signature "alert tcp $EXTERNAL_NET $HTTP_PORTS -> $HOME_NET any (msg:"MALWARE-OTHER Win.Trojan.Zeus Spam 2013 dated zip/exe HTTP Response - potential malware download"; flow:to_client,established; content:"-2013.zip|0D 0A|"; fast_pattern:only; content:"-2013.zip|0D 0A|"; http_header; content:"-"; within:1; distance:-14; http_header; file_data; content:"-2013.exe"; content:"-"; within:1; distance:-14; metadata:impact_flag red, policy balanced-ips drop, policy max-detect-ips drop, policy security-ips drop, ruleset community, service http; reference:url,www.virustotal.com/en/file/2eff3ee6ac7f5bf85e4ebcbe51974d0708cef666581ef1385c628233614b22c0/analysis/; classtype:trojan-activity; sid:26470; rev:2;)" from file /usr/local/etc/suricata/suricata_41945_igb1/rules/suricata.rules at line 162
Aug 13 00:23:26	php-cgi	21038	[Suricata] ERROR: Snort VRT rules md5 download failed...
Aug 13 00:23:25	php-cgi	21038	[Suricata] Emerging Threats Open rules file update downloaded successfully.
Aug 13 00:23:24	php-cgi	21038	[Suricata] There is a new set of Emerging Threats Open rules posted. Downloading emerging.rules.tar.gz...
Aug 12 19:23:14	ppp	71373	[wan_link0] LCP: LayerDown
Aug 12 19:23:14	ppp	71373	[wan] IPCP: Down event
Aug 12 19:23:14	ppp	71373	[wan] IFACE: Down event
Aug 12 19:23:13	ppp	71373	[wan] IPCP: LayerDown
Aug 12 19:23:13	ppp	71373	[wan_link0] LCP: Down event
Aug 12 19:23:13	ppp	71373	[wan_link0] Link: DOWN event
Aug 12 18:42:07	kernel		igb1: link state changed to DOWN
Aug 12 18:42:01	kernel		igb1: link state changed to DOWN
Aug 12 18:41:14	suricata	67299	[100333] <Error> -- error parsing signature "alert tcp $EXTERNAL_NET $HTTP_PORTS -> $HOME_NET any (msg:"MALWARE-OTHER Win.Trojan.Zeus Spam 2013 dated zip/exe HTTP Response - potential malware download"; flow:to_client,established; content:"-2013.zip|0D 0A|"; fast_pattern:only; content:"-2013.zip|0D 0A|"; http_header; content:"-"; within:1; distance:-14; http_header; file_data; content:"-2013.exe"; content:"-"; within:1; distance:-14; metadata:impact_flag red, policy balanced-ips drop, policy max-detect-ips drop, policy security-ips drop, ruleset community, service http; reference:url,www.virustotal.com/en/file/2eff3ee6ac7f5bf85e4ebcbe51974d0708cef666581ef1385c628233614b22c0/analysis/; classtype:trojan-activity; sid:26470; rev:2;)" from file /usr/local/etc/suricata/suricata_41945_igb1/rules/suricata.rules at line 162
Aug 12 18:40:54	kernel		igb1: link state changed to DOWN
Aug 12 18:37:28	kernel		igb1: link state changed to DOWN
Aug 12 18:37:22	kernel		igb1: link state changed to DOWN
Aug 12 18:36:35	suricata	85777	[100429] <Error> -- error parsing signature "alert tcp $EXTERNAL_NET $HTTP_PORTS -> $HOME_NET any (msg:"MALWARE-OTHER Win.Trojan.Zeus Spam 2013 dated zip/exe HTTP Response - potential malware download"; flow:to_client,established; content:"-2013.zip|0D 0A|"; fast_pattern:only; content:"-2013.zip|0D 0A|"; http_header; content:"-"; within:1; distance:-14; http_header; file_data; content:"-2013.exe"; content:"-"; within:1; distance:-14; metadata:impact_flag red, policy balanced-ips drop, policy max-detect-ips drop, policy security-ips drop, ruleset community, service http; reference:url,www.virustotal.com/en/file/2eff3ee6ac7f5bf85e4ebcbe51974d0708cef666581ef1385c628233614b22c0/analysis/; classtype:trojan-activity; sid:26470; rev:2;)" from file /usr/local/etc/suricata/suricata_41945_igb1/rules/suricata.rules at line 162
Aug 12 18:36:16	kernel		igb1: link state changed to DOWN
Aug 12 18:34:32	shutdown	18862	power-down by root:
Aug 12 18:13:02	ppp	19091	[wan_link0] LCP: Down event
Aug 12 18:13:02	ppp	19091	[wan_link0] Link: DOWN event

stephenw10

That seems more likely to be a symptom rather than a cause. I assume you have Suricata running on LAN there and it's igb1? If it's running in in-line mode that will cause the re-link on igb1 when Suricata reloads. Suricata will be reloaded when the WAN reconnects. It still looks like the WAN disconnecting is the first thing that happens.

diyhouse

@stephenw10
Yes,.. that's what I wondered,. but was not sure,...
Thinking about connectivity,.. and stability,... A year or so ago,.. my system would run for months,.. even during power cuts,.. so 100+ day of run time was no issue,..
the WAN on the other hand would run for a week or so,.. maybe 3,.. then reboot,... which I took/assumed was BT enforcing their ( you cannot keep the same IP ),..
But now I wonder if this was a symptom,.. of the line performance degrading,... as it is now happening more and more,..
I was hoping that a 'weekly' WAN reboot would keep 'BT happy',.. and stop them forcing reboots an inopportune moments,.. during the day... but no.
The frequency of WAN reboots has over the last several months increased,.. and ( like things that change slowly ) you don't seem to notice the trend..
So looking at the big picture as they say,.. I think I need to contact BT support for some 'line performance help'... as things seem to point to line degradation,.. over the last 6+ months... or do you think i'm barking up the wrong tree....
pls feel free to disagree

stephenw10

Well I've never had that issue. BT only disconnects their end when there's a problem. So I wouldn't expect you to have to reconnect at all.

So, yes, it could be some upstream issue. It's odd though that it doesn't just time out. It would be nice to get some logs showing whatever triggers it. Somehow.

Ah OK edit the mpd custom conf file you created and uncomment the 'log' line there. If you change it to log +all you will see a lot of logging. Possibly too much!
But you should be able enable or disable the options there to get reasonable logging and see what's happening.

stephenw10

And by a lot I mean it logs every second:

Aug 15 22:27:27 	ppp 	4348 	EVENT: Processing timer "BundBm" BundBmTimeout()
Aug 15 22:27:27 	ppp 	4348 	EVENT: Processing event EVENT_TIMEOUT TimerExpires()
Aug 15 22:27:26 	ppp 	4348 	EVENT: Processing event EVENT_TIMEOUT TimerExpires() done
Aug 15 22:27:26 	ppp 	4348 	EVENT: Processing timer "BundBm" BundBmTimeout() done
Aug 15 22:27:26 	ppp 	4348 	EVENT: Registering event EVENT_TIMEOUT TimerExpires() done at timer.c:50
Aug 15 22:27:26 	ppp 	4348 	EVENT: Registering event EVENT_TIMEOUT TimerExpires() at timer.c:50
Aug 15 22:27:26 	ppp 	4348 	EVENT: Starting timer "BundBm" BundBmTimeout() for 1000 ms at bund.c:1678
Aug 15 22:27:26 	ppp 	4348 	[opt4] OUT util: total 6% 0% 1% 0% 1% 36% 1%
Aug 15 22:27:26 	ppp 	4348 	[opt4] IN util: total 18% 0% 1% 0% 1% 103% 3%
Aug 15 22:27:26 	ppp 	4348 	[opt4] 1 1 1 1 1 1
Aug 15 22:27:26 	ppp 	4348 	EVENT: Processing timer "BundBm" BundBmTimeout()
Aug 15 22:27:26 	ppp 	4348 	EVENT: Processing event EVENT_TIMEOUT TimerExpires()
Aug 15 22:27:25 	ppp 	4348 	EVENT: Processing event EVENT_TIMEOUT TimerExpires() done
Aug 15 22:27:25 	ppp 	4348 	EVENT: Processing timer "BundBm" BundBmTimeout() done
Aug 15 22:27:25 	ppp 	4348 	EVENT: Registering event EVENT_TIMEOUT TimerExpires() done at timer.c:50
Aug 15 22:27:25 	ppp 	4348 	EVENT: Registering event EVENT_TIMEOUT TimerExpires() at timer.c:50
Aug 15 22:27:25 	ppp 	4348 	EVENT: Starting timer "BundBm" BundBmTimeout() for 1000 ms at bund.c:1678
Aug 15 22:27:25 	ppp 	4348 	[opt4] OUT util: total 6% 1% 0% 1% 0% 1% 36%
Aug 15 22:27:25 	ppp 	4348 	[opt4] IN util: total 18% 1% 0% 1% 0% 1% 103%
Aug 15 22:27:25 	ppp 	4348 	[opt4] 1 1 1 1 1 1

diyhouse

@stephenw10
Tx Stephen,.. File edited,.. and logging enabled... WAN re-started 8:30am


Aug 16 09:17:24	ppp	9394	EVENT: Processing event EVENT_TIMEOUT TimerExpires() done
Aug 16 09:17:24	ppp	9394	EVENT: Processing timer "BundBm" BundBmTimeout() done
Aug 16 09:17:24	ppp	9394	EVENT: Registering event EVENT_TIMEOUT TimerExpires() done at timer.c:50
Aug 16 09:17:24	ppp	9394	EVENT: Registering event EVENT_TIMEOUT TimerExpires() at timer.c:50
Aug 16 09:17:24	ppp	9394	EVENT: Starting timer "BundBm" BundBmTimeout() for 1000 ms at bund.c:1678
Aug 16 09:17:24	ppp	9394	[wan] OUT util: total 32% 30% 15% 60% 33% 3% 50%
Aug 16 09:17:24	ppp	9394	[wan] IN util: total 207% 11% 5% 1119% 21% 2% 84%
Aug 16 09:17:24	ppp	9394	[wan] 1 1 1 1 1 1
Aug 16 09:17:24	ppp	9394	EVENT: Processing timer "BundBm" BundBmTimeout()
Aug 16 09:17:24	ppp	9394	EVENT: Processing event EVENT_TIMEOUT TimerExpires()
Aug 16 09:17:23	ppp	9394	EVENT: Processing event EVENT_TIMEOUT TimerExpires() done
Aug 16 09:17:23	ppp	9394	EVENT: Processing timer "BundBm" BundBmTimeout() done
Aug 16 09:17:23	ppp	9394	EVENT: Registering event EVENT_TIMEOUT TimerExpires() done at timer.c:50
Aug 16 09:17:23	ppp	9394	EVENT: Registering event EVENT_TIMEOUT TimerExpires() at timer.c:50
Aug 16 09:17:23	ppp	9394	EVENT: Starting timer "BundBm" BundBmTimeout() for 1000 ms at bund.c:1678
Aug 16 09:17:23	ppp	9394	[wan] OUT util: total 25% 6% 30% 15% 60% 33% 3%
Aug 16 09:17:23	ppp	9394	[wan] IN util: total 196% 21% 11% 5% 1119% 21% 2%
Aug 16 09:17:23	ppp	9394	[wan] 1 1 1 1 1 1

stephenw10

Nothing else shown? Nothing at 8.30? I imagine there are a massive amount of logs!

diyhouse

@stephenw10
Yes I had an event at 4:05pm this afternoon,.. unfortunately I was out,.. just got back and extened log length. to 20000 lines,.. and this still only takes me back 15 mins,..
So will have to wait a little longer... dam

diyhouse

@diyhouse
This is a typical log throughput ATM,...

Aug 16 18:19:03	ppp	9394	[wan] 1 1 1 1 1 1
Aug 16 18:19:03	ppp	9394	EVENT: Processing timer "BundBm" BundBmTimeout()
Aug 16 18:19:03	ppp	9394	EVENT: Processing event EVENT_TIMEOUT TimerExpires()
Aug 16 18:19:02	ppp	9394	EVENT: Processing event EVENT_TIMEOUT TimerExpires() done
Aug 16 18:19:02	ppp	9394	EVENT: Processing timer "BundBm" BundBmTimeout() done
Aug 16 18:19:02	ppp	9394	EVENT: Registering event EVENT_TIMEOUT TimerExpires() done at timer.c:50
Aug 16 18:19:02	ppp	9394	EVENT: Registering event EVENT_TIMEOUT TimerExpires() at timer.c:50
Aug 16 18:19:02	ppp	9394	EVENT: Starting timer "BundBm" BundBmTimeout() for 1000 ms at bund.c:1678
Aug 16 18:19:02	ppp	9394	[wan] OUT util: total 132% 32% 5% 350% 41% 27% 337%
Aug 16 18:19:02	ppp	9394	[wan] IN util: total 8052% 821% 6% 30182% 108% 6% 17191%
Aug 16 18:19:02	ppp	9394	[wan] 1 1 1 1 1 1
Aug 16 18:19:02	ppp	9394	EVENT: Processing timer "BundBm" BundBmTimeout()
Aug 16 18:19:02	ppp	9394	EVENT: Processing event EVENT_TIMEOUT TimerExpires()
Aug 16 18:19:01	ppp	9394	EVENT: Processing event EVENT_TIMEOUT TimerExpires() done
Aug 16 18:19:01	ppp	9394	EVENT: Processing timer "BundBm" BundBmTimeout() done
Aug 16 18:19:01	ppp	9394	EVENT: Registering event EVENT_TIMEOUT TimerExpires() done at timer.c:50
Aug 16 18:19:01	ppp	9394	EVENT: Registering event EVENT_TIMEOUT TimerExpires() at timer.c:50
Aug 16 18:19:01	ppp	9394	EVENT: Starting timer "BundBm" BundBmTimeout() for 1000 ms at bund.c:1678
Aug 16 18:19:01	ppp	9394	[wan] OUT util: total 82% 37% 32% 5% 350% 41% 27%
Aug 16 18:19:01	ppp	9394	[wan] IN util: total 5191% 26% 821% 6% 30182% 108% 6%
Aug 16 18:19:01	ppp	9394	[wan] 1 1 1 1 1 1

stephenw10

Yeah +all is probably more than you need. You should be able to enable those things individually. Though I'm not sure which one you would need.

Gertjan

@diyhouse said in WAN periodically Rebooting:

and this still only takes me back 15 mins

IF you have some disk space left, you could goto Status > System Logs > Settings and make the log files (way) bigger :

diyhouse

@Gertjan Yes many tx,.. I wondered if there was a parameter to limit log file size.
Although I had created at listing limit of 100000lines... it didn't seem that big.
One thing I might add the the wonderful pfsense dev. boys/girls,... is it would be nice to set the size limit in some more appro. units,.. eg meg bytes, or gig bytes,... pls
thankyou
BTW gone as big as system lets me,.. 2000000000bytes,.. or 2gigs as I recokon.. I have 99G free,.. but guess system is considering all log files,..

Gertjan

@diyhouse said in WAN periodically Rebooting:

Although I had created at listing limit of 100000lines...

That a GUI setting.
100000 lines in a GUI is ... 'bad' as PHP is very low with 'thoudands' and things go downhill after hundreds of thousands. Until PHP breaks trying to show it.
Big log files are meant to be show 'on the command line'. Or use SSH+FTP = SFTP, which supports stuf like this in the blink of an eye.

@diyhouse said in WAN periodically Rebooting:

some more appro. units,.. eg meg bytes, or gig bytes,... pls
thankyou

Would be nice for the ones who are asking ... but never looked / found the log files.
The ones who 'know', don't bother, they go straight to the source, as things down there are the same dor the last several decades already ^^

Btw : huge log files are 'nice', because it gives you a big overview of what heppens.
But then pfSense has to rotate them, and that takes a lot of resource (CPU power etc) and not every pfSense has big drives, and endless write cycles, etc.
Running out of disk space is a known issue also.

diyhouse

@Gertjan Tx for the comments,..
I will drop the GUI max lines down though,...
BTW,.. Log files,.. where are they stored in the file structure?
this is my current file system

Filesystem                          Size    Used   Avail Capacity  Mounted on
zroot/ROOT/default                  100G    1.2G     99G     1%    /
devfs                               1.0K      0B    1.0K     0%    /dev
zroot/tmp                            99G    356K     99G     0%    /tmp
zroot/var                            99G    413M     99G     0%    /var
zroot                                99G     88K     99G     0%    /zroot
zroot/reservation                   110G     96K    110G     0%    /zroot/reservation
zroot/ROOT/default/var_cache_pkg     99G    177M     99G     0%    /var/cache/pkg
zroot/ROOT/default/var_db_pkg        99G    5.4M     99G     0%    /var/db/pkg
tmpfs                               4.0M    152K    3.9M     4%    /var/run