V123 - working fine for long time, now disconnects and high CPU



  • Hello. Have some things going on in last couple days, need some guidance please.

    First, system is older Dell 1.5ghz pentium 4 with 512mb pc133 ram. Hdd is 20gb IBM ata66. 2 Intel 82550 Pro cards.

    Config is very simple. DSL modem to pfsense, PPPOE. Lan NIC is connected to switches, serving my house with 6 computers and a few laptops intermittently. Nothing special really, except I do have a synology NAS that I allowed remote connections to.

    Have had system up for maybe 6 months or more, no problems. I periodically look at logs and note CPU useage would be high at 30%, normally around 5% (RRD graphs). Do some gaming, a little streaming and occassional file download, but normally inactive while I am at work. The family browses/facebooks a bit, but not much throughput.

    Connection is 10m x 768. Have had connection for many years. Modem is newish.

    I did install squid and squidguard. I configured shallas blacklist and am using captive portal. The use is minimal, as I am only filtering traffic for 3 computers (kids). Simply allowing some access while blocking all access to sites they are not old enough to view. Bypassed for all other MAC addresses.

    I also use DHCP, but statically map IP to MAC.

    Like I mentioned, I have had no problems, everything been working great. Couple days ago, no internet, but could ping pfsense box. Restarted modem, then it worked. Worked fine all that night. Yesterday, intermittent disconnect of WAN IP. I thought it was modem, but seems to be related to pfsense box.

    When looking at RRD graphs, there is an unusual high CPU spike at the same time the WAN drops. The pfsense box will, after a couple minutes, reload things and get to working again. Then it works fine until the CPU spike hits again. This is new behaviour.

    I am a pretty advanced user for windows OS, but have minimal experience with unix flavors, so am unsure really how to proceed. I found a couple commands I can run from the console when I get home (I think it is top or something) that will show me a little data. I know I have the fxp NICs, and I already tried the "offload" setting both ways, with no difference. I have read it might be squid or squidguard as well, but haven't traveled that option yet.

    Here is what the system log shows, and it sort of repeats this sequence of events at every WAN disconnect. I would like to see more of the log, but don't know where to do that at or how to do that.

    I am seeking help in tracking this issue down. I have another box (slower) that I first tried pfsense on, I was thinking of trying that, to determine if the error is in the current box. I could also hook back up to my router (dlink) and see if the problem goes away as well.

    Thanks for any help.

    Nov 18 08:42:35	check_reload_status: reloading filter
    Nov 18 08:42:32	php[489]: /pkg_edit.php: Reloading Squid for configuration sync
    Nov 18 07:23:40	php: : phpDynDNS: (Success) IP Address Changed Successfully! (72.160.11.189)
    Nov 18 07:23:40	php: : phpDynDNS: updating cache file /cf/conf/dyndns.cache: 72.160.11.189
    Nov 18 07:23:40	php: : DynDns: Current Service: dyndns
    Nov 18 07:23:40	php: : DynDns: DynDns _checkStatus() starting.
    Nov 18 07:23:39	php: : DynDns: DynDns _update() starting. Dynamic
    Nov 18 07:23:39	php: : DynDns: DynDns _update() starting.
    Nov 18 07:23:39	php: : DynDns: cacheIP != wan_ip. Updating.
    Nov 18 07:23:39	php: : DynDns: Cached IP: 72.160.15.227
    Nov 18 07:23:39	php: : DynDns: Current WAN IP: 72.160.11.189
    Nov 18 07:23:39	php: : DynDns: _detectChange() starting.
    Nov 18 07:23:39	php: : DynDns: updatedns() starting
    Nov 18 07:23:39	php: : DynDns: Running updatedns()
    Nov 18 07:23:36	check_reload_status: updating dyndns
    Nov 18 07:23:35	check_reload_status: reloading filter
    Nov 18 07:23:34	php: : Configuring slbd
    Nov 18 07:23:34	php: : pfSense package system has detected an ip change 72.160.15.227 -> 72.160.11.189 ... Restarting packages.
    Nov 18 07:23:34	last message repeated 29 times
    Nov 18 07:23:34	php: : The File Manager package is missing required dependencies and must be reinstalled.
    Nov 18 07:23:32	php: : Reloading Squid for configuration sync
    Nov 18 07:23:29	squid[29211]: Squid Parent: child process 29831 started
    Nov 18 07:23:28	php: : Reloading Squid for configuration sync
    Nov 18 07:23:26	squid[29211]: Squid Parent: child process 29214 exited due to signal 6
    Nov 18 07:23:26	kernel: pid 29214 (squid), uid 62: exited on signal 6
    Nov 18 07:23:26	squid[29214]: The url_rewriter helpers are crashing too rapidly, need help!
    Nov 18 07:23:25	last message repeated 3 times
    Nov 18 07:23:20	php: : Reloading Squid for configuration sync
    Nov 18 07:23:17	Squid_Alarm[29278]: Squid has resumed. Reconfiguring filter.
    Nov 18 07:23:17	Squid_Alarm[29269]: Squid has resumed. Reconfiguring filter.
    Nov 18 07:23:16	squid[29211]: Squid Parent: child process 29214 started
    Nov 18 07:23:16	php: : Starting Squid
    Nov 18 07:23:05	php: : SQUID is installed but not started. Not installing "filter" rules.
    Nov 18 07:23:05	php: : SQUID is installed but not started. Not installing "nat" rules.
    Nov 18 07:23:04	php: : Reloading Squid for configuration sync
    Nov 18 07:23:04	squid[28942]: Squid Parent: child process 28945 exited due to signal 9
    Nov 18 07:23:03	Squid_Alarm[29021]: Reconfiguring filter...
    Nov 18 07:23:02	php: : Reloading Squid for configuration sync
    Nov 18 07:23:00	squid[28942]: Squid Parent: child process 28945 started
    Nov 18 07:23:00	Squid_Alarm[28935]: Attempting restart...
    Nov 18 07:23:00	Squid_Alarm[28933]: Squid has exited. Reconfiguring filter.
    Nov 18 07:22:59	squid[54609]: Exiting due to unexpected forced shutdown
    Nov 18 07:22:59	squid[54609]: Squid Parent: child process 54823 exited due to signal 15
    Nov 18 07:22:59	php: : Resyncing configuration for all packages.
    Nov 18 07:22:57	php: : Creating rrd update script
    Nov 18 07:22:57	php: : Informational: DHClient spawned /etc/rc.newwanip and the new ip is wan - 72.160.11.189.
    Nov 18 07:22:52	php: : rc.newwanip working with (IP address: 72.160.11.189) (interface: wan) (interface real: ng0).
    Nov 18 07:22:52	php: : Informational: rc.newwanip is starting wan.
    Nov 18 07:22:51	check_reload_status: rc.newwanip starting
    Nov 18 07:22:48	mpd: 72.160.11.189 -> 72.160.11.1
    Nov 18 07:22:48	mpd: IPADDR 72.160.11.1
    Nov 18 07:22:48	mpd: 72.160.11.1 is OK
    Nov 18 07:22:48	mpd: IPADDR 72.160.11.1
    Nov 18 07:22:48	mpd: SECDNS 209.206.160.253
    Nov 18 07:22:48	mpd: PRIDNS 209.206.184.249
    Nov 18 07:22:48	mpd: IPADDR 72.160.11.189
    Nov 18 07:22:48	mpd: SECDNS 209.206.160.253
    Nov 18 07:22:48	mpd: PRIDNS 209.206.184.249
    Nov 18 07:22:48	mpd: IPADDR 72.160.11.189
    Nov 18 07:22:48	mpd: SECDNS 209.206.160.253
    Nov 18 07:22:48	mpd: PRIDNS 209.206.184.249
    Nov 18 07:22:48	mpd: 72.160.11.189 is OK
    Nov 18 07:22:48	mpd: IPADDR 72.160.11.189
    Nov 18 07:22:48	mpd: SECDNS 0.0.0.0
    Nov 18 07:22:48	mpd: PRIDNS 0.0.0.0
    Nov 18 07:22:48	mpd: IPADDR 0.0.0.0
    Nov 18 07:22:48	mpd: COMPPROTO VJCOMP, 16 comp. channels, no comp-cid
    Nov 18 07:22:48	mpd: SECDNS 0.0.0.0
    Nov 18 07:22:48	mpd: PRIDNS 0.0.0.0
    Nov 18 07:22:48	mpd: COMPPROTO VJCOMP, 16 comp. channels, no comp-cid
    Nov 18 07:22:48	mpd: IPADDR 0.0.0.0
    Nov 18 07:22:46	mpd: SECDNS 0.0.0.0
    Nov 18 07:22:46	mpd: PRIDNS 0.0.0.0
    Nov 18 07:22:46	mpd: COMPPROTO VJCOMP, 16 comp. channels, no comp-cid
    Nov 18 07:22:46	mpd: IPADDR 0.0.0.0
    Nov 18 07:22:44	mpd: SECDNS 0.0.0.0
    Nov 18 07:22:44	mpd: PRIDNS 0.0.0.0
    Nov 18 07:22:44	mpd: COMPPROTO VJCOMP, 16 comp. channels, no comp-cid
    Nov 18 07:22:44	mpd: IPADDR 0.0.0.0
    Nov 18 07:22:42	mpd: SECDNS 0.0.0.0
    Nov 18 07:22:42	mpd: PRIDNS 0.0.0.0
    Nov 18 07:22:42	mpd: COMPPROTO VJCOMP, 16 comp. channels, no comp-cid
    Nov 18 07:22:42	mpd: IPADDR 0.0.0.0
    Nov 18 07:22:41	mpd: MAGICNUM 337e3806
    Nov 18 07:22:41	mpd: MRU 1492
    Nov 18 07:22:41	mpd: MAGICNUM 5b64e971
    Nov 18 07:22:41	mpd: AUTHPROTO PAP
    Nov 18 07:22:41	mpd: MRU 1492
    Nov 18 07:22:41	mpd: MAGICNUM 5b64e971
    Nov 18 07:22:41	mpd: AUTHPROTO PAP
    Nov 18 07:22:41	mpd: MRU 1492
    Nov 18 07:22:41	mpd: MAGICNUM 337e3806
    Nov 18 07:22:41	mpd: MRU 1492
    Nov 18 07:22:35	mpd: SECDNS 209.206.160.253
    Nov 18 07:22:35	mpd: PRIDNS 209.206.184.249
    Nov 18 07:22:35	mpd: IPADDR 72.160.15.69
    Nov 18 07:22:33	mpd: SECDNS 209.206.160.253
    Nov 18 07:22:33	mpd: PRIDNS 209.206.184.249
    Nov 18 07:22:33	mpd: IPADDR 72.160.15.69
    Nov 18 07:22:31	mpd: SECDNS 209.206.160.253
    


  • @sully:

    When looking at RRD graphs, there is an unusual high CPU spike at the same time the WAN drops. The pfsense box will, after a couple minutes, reload things and get to working again. Then it works fine until the CPU spike hits again. This is new behaviour.

    The CPU spike is probably a consequence of the WAN link coming on again - some big processes re-initialising.

    @sully:

    Here is what the system log shows, and it sort of repeats this sequence of events at every WAN disconnect. I would like to see more of the log, but don't know where to do that at or how to do that.

    pfSense shell command clog /var/log/system.log will show the full log. You can execute a shell command through the web GUI by going to Diagnostics -> Command Prompt and typing the command in the Command box. There are other log files in /var/log (e.g. firewall log) that you can display using the same technique. I would look in the system log for a hint of the wan link going down, the post says the 10 lines before that and the 10 lines after it.

    @sully:

    I am seeking help in tracking this issue down. I have another box (slower) that I first tried pfsense on, I was thinking of trying that, to determine if the error is in the current box. I could also hook back up to my router (dlink) and see if the problem goes away as well.

    Using the two other systems for a while could provide useful comparison information.

    What sort of NICs are you using? I'm interested in the FreeBSD device names, e.g. sk0, re1 etc. It may be that the WAN nic in the box is "locking up", the driver resets it and this causes the wan link to go down the recover and the recovery causes the CPU spike.



  • 2 Intel 82550 Pro cards

    I know I have the fxp NICs

    Do you need more than that? I can pull them, but thought those two pieces of data were what would be needed.

    The cpu spike on wan linking sounds logical, had not thought about that.

    I looked at the log. It is chock full of entries, some I can see and understand, other stuff I have no idea. I can see the even happening, both in RRD and now in this log. Here is the beginning of it, and multiple times it happens it appears very much the same, except of course IP is different due to dhcp

    Nov 18 15:57:56 pfsense mpd: [pppoe] LCP: no reply to 1 echo request(s)
    Nov 18 15:58:06 pfsense mpd: [pppoe] LCP: no reply to 2 echo request(s)
    Nov 18 15:58:16 pfsense mpd: [pppoe] LCP: no reply to 3 echo request(s)
    Nov 18 15:58:26 pfsense mpd: [pppoe] LCP: no reply to 4 echo request(s)
    Nov 18 15:58:36 pfsense mpd: [pppoe] LCP: no reply to 5 echo request(s)
    Nov 18 15:58:36 pfsense mpd: [pppoe] LCP: peer not responding to echo requests
    Nov 18 15:58:36 pfsense mpd: [pppoe] LCP: LayerFinish
    Nov 18 15:58:36 pfsense mpd: [pppoe] LCP: LayerStart
    Nov 18 15:58:36 pfsense mpd: [pppoe] LCP: state change Opened --> Starting
    Nov 18 15:58:36 pfsense mpd: [pppoe] LCP: phase shift NETWORK --> DEAD
    Nov 18 15:58:36 pfsense mpd: [pppoe] setting interface ng0 MTU to 1500 bytes
    Nov 18 15:58:36 pfsense mpd: [pppoe] up: 0 links, total bandwidth 9600 bps
    Nov 18 15:58:36 pfsense mpd: [pppoe] IPCP: Down event
    Nov 18 15:58:36 pfsense mpd: [pppoe] IPCP: state change Opened --> Starting
    Nov 18 15:58:36 pfsense mpd: [pppoe] IPCP: LayerDown
    Nov 18 15:58:36 pfsense mpd: [pppoe] IFACE: Down event
    Nov 18 15:58:36 pfsense mpd: [pppoe] exec: /sbin/route delete 0.0.0.0 72.160.11.1
    Nov 18 15:58:36 pfsense mpd: [pppoe] exec: /sbin/route delete 72.160.11.189 -iface lo0
    Nov 18 15:58:36 pfsense mpd: [pppoe] exec: /sbin/ifconfig ng0 down delete -link0
    Nov 18 15:58:36 pfsense mpd: [pppoe] LCP: LayerDown
    Nov 18 15:58:36 pfsense mpd: [pppoe] device: CLOSE event in state UP
    Nov 18 15:58:36 pfsense mpd: [pppoe] device is now in state CLOSING
    Nov 18 15:58:36 pfsense mpd: [pppoe] device: OPEN event in state CLOSING
    Nov 18 15:58:36 pfsense mpd: [pppoe] device is now in state CLOSING
    Nov 18 15:58:36 pfsense mpd: [pppoe] device: DOWN event in state CLOSING
    Nov 18 15:58:36 pfsense mpd: [pppoe] device is now in state DOWN
    Nov 18 15:58:36 pfsense mpd: [pppoe] link: DOWN event
    Nov 18 15:58:36 pfsense mpd: [pppoe] LCP: Down event
    

    The last lines, device closing opening link down up etc, is repeated many times until this segment near the end of the event

    Nov 18 16:00:56 pfsense mpd: [pppoe] LCP: Up event
    Nov 18 16:00:56 pfsense mpd: [pppoe] LCP: state change Starting --> Req-Sent
    Nov 18 16:00:56 pfsense mpd: [pppoe] LCP: phase shift DEAD --> ESTABLISH
    Nov 18 16:00:56 pfsense mpd: [pppoe] LCP: SendConfigReq #29
    Nov 18 16:00:56 pfsense mpd:  MRU 1492
    Nov 18 16:00:56 pfsense mpd:  MAGICNUM 27ae8972
    Nov 18 16:00:56 pfsense mpd: [pppoe] LCP: rec'd Configure Request #156 link 0 (Req-Sent)
    Nov 18 16:00:56 pfsense mpd:  MRU 1492
    Nov 18 16:00:56 pfsense mpd:  AUTHPROTO PAP
    Nov 18 16:00:56 pfsense mpd:  MAGICNUM 3c2f9fec
    Nov 18 16:00:56 pfsense mpd: [pppoe] LCP: SendConfigAck #156
    Nov 18 16:00:56 pfsense mpd:  MRU 1492
    Nov 18 16:00:56 pfsense mpd:  AUTHPROTO PAP
    Nov 18 16:00:56 pfsense mpd:  MAGICNUM 3c2f9fec
    Nov 18 16:00:56 pfsense mpd: [pppoe] LCP: state change Req-Sent --> Ack-Sent
    Nov 18 16:00:58 pfsense mpd: [pppoe] LCP: SendConfigReq #30
    Nov 18 16:00:58 pfsense mpd:  MRU 1492
    Nov 18 16:00:58 pfsense mpd:  MAGICNUM 27ae8972
    Nov 18 16:01:00 pfsense mpd: [pppoe] LCP: SendConfigReq #31
    Nov 18 16:01:00 pfsense mpd:  MRU 1492
    Nov 18 16:01:00 pfsense mpd:  MAGICNUM 27ae8972
    Nov 18 16:01:02 pfsense mpd: [pppoe] LCP: SendConfigReq #32
    Nov 18 16:01:02 pfsense mpd:  MRU 1492
    Nov 18 16:01:02 pfsense mpd:  MAGICNUM 27ae8972
    Nov 18 16:01:02 pfsense mpd: [pppoe] PPPoE connection closed
    Nov 18 16:01:02 pfsense mpd: [pppoe] device: DOWN event in state UP
    Nov 18 16:01:02 pfsense mpd: [pppoe] device is now in state DOWN
    Nov 18 16:01:02 pfsense mpd: [pppoe] link: DOWN event
    Nov 18 16:01:02 pfsense mpd: [pppoe] LCP: Down event
    Nov 18 16:01:02 pfsense mpd: [pppoe] LCP: state change Ack-Sent --> Starting
    Nov 18 16:01:02 pfsense mpd: [pppoe] LCP: phase shift ESTABLISH --> DEAD
    Nov 18 16:01:02 pfsense mpd: [pppoe] device: OPEN event in state DOWN
    Nov 18 16:01:02 pfsense mpd: [pppoe] pausing 4 seconds before open
    Nov 18 16:01:02 pfsense mpd: [pppoe] device is now in state DOWN
    Nov 18 16:01:06 pfsense mpd: [pppoe] device: OPEN event in state DOWN
    Nov 18 16:01:06 pfsense mpd: [pppoe] pausing 1 seconds before open
    Nov 18 16:01:06 pfsense mpd: [pppoe] device is now in state DOWN
    Nov 18 16:01:07 pfsense mpd: [pppoe] device: OPEN event in state DOWN
    Nov 18 16:01:07 pfsense mpd: [pppoe] device is now in state OPENING
    Nov 18 16:01:07 pfsense mpd: [pppoe] rec'd ACNAME "rb-bras.ka-0A014030500677"
    Nov 18 16:01:07 pfsense mpd: [pppoe] PPPoE connection successful
    Nov 18 16:01:07 pfsense mpd: [pppoe] device: UP event in state OPENING
    Nov 18 16:01:07 pfsense mpd: [pppoe] device is now in state UP
    Nov 18 16:01:07 pfsense mpd: [pppoe] link: UP event
    Nov 18 16:01:07 pfsense mpd: [pppoe] link: origination is local
    Nov 18 16:01:07 pfsense mpd: [pppoe] LCP: Up event
    Nov 18 16:01:07 pfsense mpd: [pppoe] LCP: state change Starting --> Req-Sent
    Nov 18 16:01:07 pfsense mpd: [pppoe] LCP: phase shift DEAD --> ESTABLISH
    Nov 18 16:01:07 pfsense mpd: [pppoe] LCP: SendConfigReq #33
    Nov 18 16:01:07 pfsense mpd:  MRU 1492
    Nov 18 16:01:07 pfsense mpd:  MAGICNUM 33b63756
    Nov 18 16:01:07 pfsense mpd: [pppoe] LCP: rec'd Configure Request #7 link 0 (Req-Sent)
    Nov 18 16:01:07 pfsense mpd:  MRU 1492
    Nov 18 16:01:07 pfsense mpd:  AUTHPROTO PAP
    Nov 18 16:01:07 pfsense mpd:  MAGICNUM 2936b0f0
    Nov 18 16:01:07 pfsense mpd: [pppoe] LCP: SendConfigAck #7
    Nov 18 16:01:07 pfsense mpd:  MRU 1492
    Nov 18 16:01:07 pfsense mpd:  AUTHPROTO PAP
    Nov 18 16:01:07 pfsense mpd:  MAGICNUM 2936b0f0
    Nov 18 16:01:07 pfsense mpd: [pppoe] LCP: state change Req-Sent --> Ack-Sent
    Nov 18 16:01:07 pfsense mpd: [pppoe] LCP: rec'd Configure Ack #33 link 0 (Ack-Sent)
    Nov 18 16:01:07 pfsense mpd:  MRU 1492
    Nov 18 16:01:07 pfsense mpd:  MAGICNUM 33b63756
    Nov 18 16:01:07 pfsense mpd: [pppoe] LCP: state change Ack-Sent --> Opened
    Nov 18 16:01:07 pfsense mpd: [pppoe] LCP: phase shift ESTABLISH --> AUTHENTICATE
    Nov 18 16:01:07 pfsense mpd: [pppoe] LCP: auth: peer wants PAP, I want nothing
    Nov 18 16:01:07 pfsense mpd: [pppoe] PAP: using authname "<snipped>"
    Nov 18 16:01:07 pfsense mpd: [pppoe] PAP: sending REQUEST
    Nov 18 16:01:07 pfsense mpd: [pppoe] LCP: LayerUp
    Nov 18 16:01:07 pfsense mpd: [pppoe] PAP: rec'd ACK #1
    Nov 18 16:01:07 pfsense mpd: [pppoe] LCP: authorization successful
    Nov 18 16:01:07 pfsense mpd: [pppoe] LCP: phase shift AUTHENTICATE --> NETWORK
    Nov 18 16:01:07 pfsense mpd: [pppoe] setting interface ng0 MTU to 1492 bytes
    Nov 18 16:01:07 pfsense mpd: [pppoe] up: 1 link, total bandwidth 64000 bps
    Nov 18 16:01:07 pfsense mpd: [pppoe] IPCP: Up event
    Nov 18 16:01:07 pfsense mpd: [pppoe] IPCP: state change Starting --> Req-Sent
    Nov 18 16:01:07 pfsense mpd: [pppoe] IPCP: SendConfigReq #51
    Nov 18 16:01:07 pfsense mpd:  IPADDR 0.0.0.0
    Nov 18 16:01:07 pfsense mpd:  COMPPROTO VJCOMP, 16 comp. channels, no comp-cid
    Nov 18 16:01:07 pfsense mpd:  PRIDNS 0.0.0.0
    Nov 18 16:01:07 pfsense mpd:  SECDNS 0.0.0.0
    Nov 18 16:01:07 pfsense mpd: [pppoe] error writing len 32 frame to bypass: Network is down
    Nov 18 16:01:09 pfsense mpd: [pppoe] IPCP: SendConfigReq #52
    Nov 18 16:01:09 pfsense mpd:  IPADDR 0.0.0.0
    Nov 18 16:01:09 pfsense mpd:  COMPPROTO VJCOMP, 16 comp. channels, no comp-cid
    Nov 18 16:01:09 pfsense mpd:  PRIDNS 0.0.0.0
    Nov 18 16:01:09 pfsense mpd:  SECDNS 0.0.0.0
    Nov 18 16:01:09 pfsense mpd: [pppoe] IPCP: rec'd Configure Reject #52 link 0 (Req-Sent)
    Nov 18 16:01:09 pfsense mpd:  COMPPROTO VJCOMP, 16 comp. channels, no comp-cid
    Nov 18 16:01:09 pfsense mpd: [pppoe] IPCP: SendConfigReq #53
    Nov 18 16:01:09 pfsense mpd:  IPADDR 0.0.0.0
    Nov 18 16:01:09 pfsense mpd:  PRIDNS 0.0.0.0
    Nov 18 16:01:09 pfsense mpd:  SECDNS 0.0.0.0
    Nov 18 16:01:09 pfsense mpd: [pppoe] IPCP: rec'd Configure Nak #53 link 0 (Req-Sent)
    Nov 18 16:01:09 pfsense mpd:  IPADDR 72.160.19.160
    Nov 18 16:01:09 pfsense mpd:    72.160.19.160 is OK
    Nov 18 16:01:09 pfsense mpd:  PRIDNS 209.206.184.249
    Nov 18 16:01:09 pfsense mpd:  SECDNS 209.206.160.253
    Nov 18 16:01:09 pfsense mpd: [pppoe] IPCP: SendConfigReq #54
    Nov 18 16:01:09 pfsense mpd:  IPADDR 72.160.19.160
    Nov 18 16:01:09 pfsense mpd:  PRIDNS 209.206.184.249
    Nov 18 16:01:09 pfsense mpd:  SECDNS 209.206.160.253
    Nov 18 16:01:09 pfsense mpd: [pppoe] IPCP: rec'd Configure Ack #54 link 0 (Req-Sent)
    Nov 18 16:01:09 pfsense mpd:  IPADDR 72.160.19.160
    Nov 18 16:01:09 pfsense mpd:  PRIDNS 209.206.184.249
    Nov 18 16:01:09 pfsense mpd:  SECDNS 209.206.160.253
    Nov 18 16:01:09 pfsense mpd: [pppoe] IPCP: state change Req-Sent --> Ack-Rcvd
    Nov 18 16:01:10 pfsense mpd: [pppoe] IPCP: rec'd Configure Request #47 link 0 (Ack-Rcvd)
    Nov 18 16:01:10 pfsense mpd:  IPADDR 72.160.19.1
    Nov 18 16:01:10 pfsense mpd:    72.160.19.1 is OK
    Nov 18 16:01:10 pfsense mpd: [pppoe] IPCP: SendConfigAck #47
    Nov 18 16:01:10 pfsense mpd:  IPADDR 72.160.19.1
    Nov 18 16:01:10 pfsense mpd: [pppoe] IPCP: state change Ack-Rcvd --> Opened
    Nov 18 16:01:10 pfsense mpd: [pppoe] IPCP: LayerUp
    Nov 18 16:01:10 pfsense mpd:   72.160.19.160 -> 72.160.19.1
    Nov 18 16:01:10 pfsense mpd: [pppoe] IFACE: Up event
    Nov 18 16:01:10 pfsense mpd: [pppoe] setting interface ng0 MTU to 1492 bytes
    Nov 18 16:01:10 pfsense mpd: [pppoe] exec: /sbin/ifconfig ng0 72.160.19.160 72.160.19.1 netmask 0xffffffff -link0
    Nov 18 16:01:10 pfsense mpd: [pppoe] exec: /sbin/route add 72.160.19.160 -iface lo0
    Nov 18 16:01:10 pfsense mpd: [pppoe] exec: /sbin/route add 0.0.0.0 72.160.19.1
    Nov 18 16:01:10 pfsense mpd: [pppoe] exec: /usr/local/sbin/ppp-linkup ng0 inet 72.160.19.160 72.160.19.1 <snipped> dns1 209.206.184.249 dns2 209.206.160.253
    Nov 18 16:01:10 pfsense mpd: [pppoe] IFACE: Up event
    Nov 18 16:01:11 pfsense check_reload_status: rc.newwanip starting</snipped></snipped>
    

    Then some data remains, which looks to be restarting of services such as squid, and delcarations that things were forced to stop and are now restarting, etc. This box seems to work fine other than this new problem. This weekend I will try other hardware like another machine, a couple routers and also use livecd for pfsense v2, and hopefully have more info.

    Thanks for taking the time to reply. Much appreciated.

    Sully.



  • @sully:

    2 Intel 82550 Pro cards

    I know I have the fxp NICs

    Do you need more than that? I can pull them, but thought those two pieces of data were what would be needed.

    Sorry, I missed that. If you have another NIC you can put in the box (you could purchase one for "small change") you could try using that for the WAN connection and see if you get different behaviour. Even just swap the WAN and LAN cables over (and swap the assignments in pfSense) and see if the behaviour moves.

    Regarding your log, pfSense WAN link has lost contact with the ppp server and for a while. As previously speculated this could be due to a driver or hardware "lockup" in pfSense. Some NICs have more of a reputation for that than the fxps do. Perhaps there is a new intermittent problem in your modem. Perhaps some changes in telephone circuit wiring has affected DSL performance (that has happened to me a couple of times when the phone company made some adjustments on behalf of other subscribers). Perhaps someone has installed some new equipment which intermittently squirts a lot of noise over a telephone circuit. I cam across a puzzling problem with my internet connection a while ago. After an extended power outage I and my family noticed our home internet was very sluggish. After some investigation I discovered that pings between pfSense and ADSL modem were very lossy (up to 60% loss on 50 pings). Modem problem? Since I didn't have a replacement modem on hand I thought I would try a difference interface to the modem and used a USB to Ethernet device I had. This completely cured the problem: no losses in many tries of 50 pings. I am puzzled what went wrong. The modem was previously connected to a VLAN capable switch and pfSense was using the switch as a port multiplier in that a single pfSense physical port was carrying VLAN tagged traffic for both the modem and a server. Pings to the server didn't show any loss while concurrent pings to the modem were showing significant loss. The modem has extremely limited reporting of errors on its LAN interface. (Sometimes it is worth paying a bit more!) I'm not trying to hijack this thread looking for suggestions for my puzzle. I am recounting this experience to illustrate some of the strange problems that can be found in network troubleshooting.



  • This problem still persists but has morphed some.

    I stopped using the pfsense box to troubleshoot.

    I used a dlink dir-655 and original gamers lounge routers.

    I used 3 different dsl modems.

    I isolated switches and machines.

    It boils down to something the telco cannot find I guess.

    If I hook dsl modem up, set pppoe in it (modem routers) and connect, then pull network cable from modem so that no inside influence is available, within 24 hours the pppoe connection has been lost and a new ip assigned. I talked to a lean technician at telco, and they said the lease for IP should be indefinate until connection is interuppted, which is how it has been historically.

    Telco has sent service call to my house to check NID 2 times, and both times they say it clear, good signal strength and very low noise.

    I have a very hard time believing 3 modems and 2 routers + pfsense box could be the problem. 2 of the modems were pulled from active service, where they were not exhibiting this problem.

    Logs of modems, routers and pfsense box show nothing, only ppp lost and then reconnect a minute later.

    Now, in the past week, using pfsense again, the IP has not dropped the ppp, but every so often I lose all ability to send packes. If I am on teamspeak, I get a timeout. Happens with iether pfsense or one of the dlink routers. The ability of the router to "ping" is gone, 100% packet loss. Then, a minute or two later, all is normal, the IP does NOT change, speeds are up to par, ping times are normal.

    A number of hours later (variable), same thing happens, could be 1 hour or 12 hours later. Modem does not lose link, IP not changed, just an intermittent "pause" in all incoming/outgoing packets.

    Very strange, and it appears beyond what the telco has knowledge of (not suprising). However, they don't stay hooked up long enough to see the issue, as you cannot reproduce it at will.

    I don't suppose there is anyone who has help for this issue, so I am just posting it so anyone searching for such things in the future can maybe benefit from it.



  • Perhaps your modem is restarting (rebooting). Some modems report an uptime. It could be worth checking the uptime after an "outage".

    On hot summer afternoons my pfSense box which has been located near a west facing window sometimes shuts down (I suspect due to overheating which is probably due in part to poorly performing fans).



  • @wallabybob:

    Perhaps your modem is restarting (rebooting). Some modems report an uptime. It could be worth checking the uptime after an "outage".

    On hot summer afternoons my pfSense box which has been located near a west facing window sometimes shuts down (I suspect due to overheating which is probably due in part to poorly performing fans).

    I thought to do that. The modem, if you set it to do the pppoe, is accesible and does contain logs. However, the logs in the modem are pretty much like the logs of the router - it only says the connection is down, then tells of getting a new IP. Never any internal errors like of the modem itself shutting down. So, uptime I am afraid is going to go nowhere that I can tell.

    My pfsense box doesn't appear to overheat at all. If I install a windows OS on it, and use speedfan, all temps are low to medium at best, even running apps that would use more cpu than pfsense does.

    I am at a loss. I don't think it is related to the hardware myself, but getting the telco to fess up is proving difficult.


  • Netgate Administrator

    It seems like you have done everything you possibly could to rule out a problem at your end. Multiplt routers, multiple modems etc.
    I would bet good money on it being a faulty line card at your exchange but without a friendly telco engineer you are going to have a hard time getting anything done about it.  :(
    Once enough people start having a problem (there are something like 20 lines per card) they will change it.  ::)

    Steve


Log in to reply