Bellsouth/AT&T PPPoE broken



  • I've been using pfSense off and on for quite a while.  A couple of weeks back, I switched over from Monowall with the intention of being a permanent change.  I configured my DSL modem for Bridged mode and configured pfSense for PPPoE with the username and password for my DSL service.  It's been working fine since (and it worked fine the previous times I've used pfSense).  But tonight, at about 8:03 PM EST, it stopped working.  Until it had stopped working today, it has been days since I changed any setting in pfSense (or looked at the webGUI, even).

    I went through the normal procedures of rebooting the DSL modem and rebooting pfSense, but it would not work.  I switch the DSL modem back to PPPoE mode and changed pfSense back to DHCP, and it works.  Switch things back, and it doesn't.  The log looks like this:

    Dec 24 00:03:56 mpd: 74.237.250.30 is OK
    Dec 24 00:03:56 mpd: IPADDR 74.237.250.30
    Dec 24 00:03:56 mpd: 74.237.250.30 is OK
    Dec 24 00:03:56 mpd: IPADDR 74.237.250.30
    Dec 24 00:03:56 mpd: 74.237.250.30 is OK
    Dec 24 00:03:56 mpd: IPADDR 74.237.250.30
    Dec 24 00:03:56 mpd: 74.237.250.30 is OK
    Dec 24 00:03:56 mpd: IPADDR 74.237.250.30
    Dec 24 00:03:56 mpd: 74.237.250.30 is OK
    Dec 24 00:03:56 mpd: IPADDR 74.237.250.30
    Dec 24 00:03:56 mpd: 74.237.250.30 is OK
    Dec 24 00:03:56 mpd: IPADDR 74.237.250.30
    Dec 24 00:03:56 mpd: 74.237.250.30 is OK
    Dec 24 00:03:56 mpd: IPADDR 74.237.250.30
    Dec 24 00:03:56 mpd: 74.237.250.30 is OK
    Dec 24 00:03:56 mpd: IPADDR 74.237.250.30
    Dec 24 00:03:56 mpd: 74.237.250.30 is OK
    Dec 24 00:03:56 mpd: IPADDR 74.237.250.30
    Dec 24 00:03:56 mpd: IPADDR 0.0.0.0
    Dec 24 00:03:56 mpd: SECDNS 0.0.0.0
    Dec 24 00:03:56 mpd: COMPPROTO VJCOMP, 16 comp. channels, no comp-cid
    Dec 24 00:03:56 mpd: IPADDR 68.152.180.7
    Dec 24 00:03:56 mpd: 68.152.180.7 is OK
    Dec 24 00:03:56 mpd: IPADDR 68.152.180.7
    Dec 24 00:03:56 mpd: SECDNS 0.0.0.0
    Dec 24 00:03:56 mpd: COMPPROTO VJCOMP, 16 comp. channels, no comp-cid
    Dec 24 00:03:56 mpd: IPADDR 0.0.0.0
    Dec 24 00:03:56 mpd: Using authname "SanitizedUserName@bellsouth.net"
    Dec 24 00:03:56 mpd: Name: "jcvlflcl77w"
    Dec 24 00:03:56 mpd: MAGICNUM f6eac5aa
    Dec 24 00:03:56 mpd: MRU 1492
    Dec 24 00:03:56 mpd: AUTHPROTO CHAP MD5
    Dec 24 00:03:56 mpd: MRU 1500
    Dec 24 00:03:56 mpd: MAGICNUM 04c40ab9
    Dec 24 00:03:56 mpd: AUTHPROTO CHAP MD5
    Dec 24 00:03:56 mpd: MRU 1500
    Dec 24 00:03:56 mpd: MAGICNUM 04c40ab9
    Dec 24 00:03:56 mpd: MAGICNUM f6eac5aa
    Dec 24 00:03:56 mpd: MRU 1492
    Dec 24 00:03:50 mpd: 74.237.250.30 is OK
    Dec 24 00:03:50 mpd: IPADDR 74.237.250.30
    Dec 24 00:03:50 mpd: 74.237.250.30 is OK
    Dec 24 00:03:50 mpd: IPADDR 74.237.250.30
    Dec 24 00:03:50 mpd: 74.237.250.30 is OK
    Dec 24 00:03:50 mpd: IPADDR 74.237.250.30
    Dec 24 00:03:50 mpd: 74.237.250.30 is OK
    Dec 24 00:03:50 mpd: IPADDR 74.237.250.30
    Dec 24 00:03:50 mpd: 74.237.250.30 is OK
    Dec 24 00:03:50 mpd: IPADDR 74.237.250.30
    Dec 24 00:03:50 mpd: 74.237.250.30 is OK
    Dec 24 00:03:50 mpd: IPADDR 74.237.250.30
    Dec 24 00:03:50 mpd: 74.237.250.30 is OK
    Dec 24 00:03:50 mpd: IPADDR 74.237.250.30
    Dec 24 00:03:50 mpd: 74.237.250.30 is OK
    Dec 24 00:03:50 mpd: IPADDR 74.237.250.30
    Dec 24 00:03:50 mpd: 74.237.250.30 is OK
    Dec 24 00:03:50 mpd: IPADDR 74.237.250.30
    Dec 24 00:03:50 mpd: IPADDR 0.0.0.0
    Dec 24 00:03:50 mpd: SECDNS 0.0.0.0
    Dec 24 00:03:50 mpd: COMPPROTO VJCOMP, 16 comp. channels, no comp-cid
    Dec 24 00:03:50 mpd: IPADDR 68.152.180.7
    Dec 24 00:03:50 mpd: 68.152.180.7 is OK
    Dec 24 00:03:50 mpd: IPADDR 68.152.180.7
    Dec 24 00:03:50 mpd: SECDNS 0.0.0.0
    Dec 24 00:03:50 mpd: COMPPROTO VJCOMP, 16 comp. channels, no comp-cid
    Dec 24 00:03:50 mpd: IPADDR 0.0.0.0
    Dec 24 00:03:50 mpd: Using authname "SanitizedUserName@bellsouth.net"
    Dec 24 00:03:50 mpd: Name: "jcvlflcl77w"
    Dec 24 00:03:50 mpd: MAGICNUM fa04a922
    Dec 24 00:03:50 mpd: MRU 1492
    Dec 24 00:03:50 mpd: AUTHPROTO CHAP MD5

    Thinking that perhaps my ISP had made some sort of change to only let their DSL modem do PPPoE (and not Customer Provided Equipment), I hooked my Mac directly up to my DSL modem while it was set to Bridged mode.  I had never set up PPPoE on my Mac before, but I used the username from pfSense along with the password from there, and it immediately connected and worked just fine.  I tried pfSense afterward in PPPoE mode, and it continued to do the above.  (In fact, those log entries are from the last time that I tried.)

    For the time being, I'm back to the DSL modem performing PPPoE and pfSense running in DHCP mode (with IP Pass-thru set on the DSL modem).

    Could anyone shed any light on this?  Am I the only one to experience this sort of issue?



  • By the way, the effect of using DHCP on my pfSense box in this case means I get a private IP handed to pfSense, so the next hop is the DSL modem.  This pretty much makes my quality graph useless, since very rarely do I expect to ever have problems with the patch cable between my pfSense box and my DSL modem.  :)

    I'd much rather have the next hop past my DSL modem monitored by pfSense.  If there was some way to specify the IP Address that the Quality graph pings?  I've looked around the webGUI but haven't found any way to do this, unless I've just overlooked it.



  • While I am not sure what is causing your PPPOE issue you can change the quality graph IP address by modifying config.xml.  Check out this thread: http://forum.pfsense.org/index.php/topic,3470.msg21418.html#msg21418



  • Thanks Scott, that got me fixed on the graph issue!

    Not sure what's up with my strange PPPoE issue, but I honestly don't care which mode I run in as long as my pfSense box is accessible directly from the outside and the quality graph is showing useful information.    I've had issues off and on with my ISP and this quality graph is very useful to me in being able to see a history of when issues started, etc.

    Thanks again!



  • This is becoming a much larger issue. We have 4 clients in the field 2 Monos and 2 PFSenses. They all have developed the exact same PPPoE issue as you have described. They all went down starting last Friday. We have upgraded to the latest versions of both and the issue remains. It seems that BellSouth (AT&T) has made some change that has affected the PPPoE of these builds. Interestingly enough, it doesn't seem to affect the Linux PPPoE component. We use DD-WRT at 40 locations and none of these is having the issue. I say that not to be divisive, but to help find a solution.

    We are going to try the beta version of Mono since it is on FreeBSD 6 and see if there is anything there. I will post back with any new info.



  • Post the PPPoE logs from one of the Bellsouth nodes.  Why does this not surprise me that the new death star is changing everything.  Heck I got a bill from these &(@#&(%#&@#@( for 50$ for a phone that I had for 2 days before they fixed it to be my old #.  This new death star is ridiculous at every turn!



  • Ah, so I'm not the only one…  Last week I had a client with Bellsouth DSL have their service stop working with m0n0wall. I couldn't get it working again with PPPoE no matter what I did, and Bellsouth's tech support is brain dead. I ended up having to leave the PPPoE on the modem with a private IP on the firewall's WAN, and let the modem NAT everything incoming to the firewall. Nasty solution but it was good enough for that site, and I had wasted enough time on it. Didn't realize it was a widespread issue.

    But I also have other sites on Bellsouth business DSL with m0n0wall that do not have any PPPoE issues.

    If anybody has further info on this please post to this thread. I renamed it to reflect the nature of the problem.



  • I'm having the same issue with Ma Bell.  I can connect to PPPoE from windows XP directly (bypassing m0n0wall), but m0n0wall fails to connect.  I have tried this using m0n0wall 1.22, 1.232, and 1.3b7.  I've noticed that all 3 versions use the same (really old) version of mpd (3.18).

    Here's the log.  See attached for a wireshark trace of the direct-through-XP working scenario

    Dec 27 22:50:33 	mpd: [pppoe] device: OPEN event in state DOWN
    Dec 27 22:50:33 	mpd: [pppoe] device is now in state OPENING
    Dec 27 22:50:33 	mpd: [pppoe] rec'd ACNAME "smyrgama72w"
    Dec 27 22:50:33 	mpd: [pppoe] PPPoE connection successful
    Dec 27 22:50:33 	mpd: [pppoe] device: UP event in state OPENING
    Dec 27 22:50:33 	mpd: [pppoe] device is now in state UP
    Dec 27 22:50:33 	mpd: [pppoe] link: UP event
    Dec 27 22:50:33 	mpd: [pppoe] link: origination is local
    Dec 27 22:50:33 	mpd: [pppoe] LCP: Up event
    Dec 27 22:50:33 	mpd: [pppoe] LCP: state change Starting --> Req-Sent
    Dec 27 22:50:33 	mpd: [pppoe] LCP: phase shift DEAD --> ESTABLISH
    Dec 27 22:50:33 	mpd: [pppoe] LCP: SendConfigReq #81
    Dec 27 22:50:33 	mpd: MRU 1492
    Dec 27 22:50:33 	mpd: MAGICNUM ee1bb960
    Dec 27 22:50:33 	mpd: [pppoe] LCP: rec'd Configure Request #78 link 0 (Req-Sent)
    Dec 27 22:50:33 	mpd: MAGICNUM 719f7c11
    Dec 27 22:50:33 	mpd: MRU 1500
    Dec 27 22:50:33 	mpd: AUTHPROTO CHAP MD5
    Dec 27 22:50:33 	mpd: [pppoe] LCP: SendConfigAck #78
    Dec 27 22:50:33 	mpd: MAGICNUM 719f7c11
    Dec 27 22:50:33 	mpd: MRU 1500
    Dec 27 22:50:33 	mpd: AUTHPROTO CHAP MD5
    Dec 27 22:50:33 	mpd: [pppoe] LCP: state change Req-Sent --> Ack-Sent
    Dec 27 22:50:33 	mpd: [pppoe] LCP: rec'd Configure Ack #81 link 0 (Ack-Sent)
    Dec 27 22:50:33 	mpd: MRU 1492
    Dec 27 22:50:33 	mpd: MAGICNUM ee1bb960
    Dec 27 22:50:33 	mpd: [pppoe] LCP: state change Ack-Sent --> Opened
    Dec 27 22:50:33 	mpd: [pppoe] LCP: phase shift ESTABLISH --> AUTHENTICATE
    Dec 27 22:50:33 	mpd: [pppoe] LCP: auth: peer wants CHAP, I want nothing
    Dec 27 22:50:33 	mpd: [pppoe] LCP: LayerUp
    Dec 27 22:50:33 	mpd: [pppoe] CHAP: rec'd CHALLENGE #1
    Dec 27 22:50:33 	mpd: Name: "smyrgama72w"
    Dec 27 22:50:33 	mpd: Using authname "SanitizedUsername@bellsouth.net"
    Dec 27 22:50:33 	mpd: [pppoe] CHAP: sending RESPONSE
    Dec 27 22:50:33 	mpd: [pppoe] CHAP: rec'd SUCCESS #1
    Dec 27 22:50:33 	mpd: [pppoe] LCP: authorization successful
    Dec 27 22:50:33 	mpd: [pppoe] LCP: phase shift AUTHENTICATE --> NETWORK
    Dec 27 22:50:33 	mpd: [pppoe] setting interface ng0 MTU to 1492 bytes
    Dec 27 22:50:33 	mpd: [pppoe] up: 1 link, total bandwidth 64000 bps
    Dec 27 22:50:33 	mpd: [pppoe] IPCP: Up event
    Dec 27 22:50:33 	mpd: [pppoe] IPCP: state change Starting --> Req-Sent
    Dec 27 22:50:33 	mpd: [pppoe] IPCP: SendConfigReq #145
    Dec 27 22:50:33 	mpd: IPADDR 0.0.0.0
    Dec 27 22:50:33 	mpd: COMPPROTO VJCOMP, 16 comp. channels, no comp-cid
    Dec 27 22:50:33 	mpd: PRIDNS 0.0.0.0
    Dec 27 22:50:33 	mpd: SECDNS 0.0.0.0
    Dec 27 22:50:33 	mpd: [pppoe] IPCP: rec'd Configure Request #63 link 0 (Req-Sent)
    Dec 27 22:50:33 	mpd: IPADDR 65.14.248.6
    Dec 27 22:50:33 	mpd: 65.14.248.6 is OK
    Dec 27 22:50:33 	mpd: [pppoe] IPCP: SendConfigAck #63
    Dec 27 22:50:33 	mpd: IPADDR 65.14.248.6
    Dec 27 22:50:33 	mpd: [pppoe] IPCP: state change Req-Sent --> Ack-Sent
    Dec 27 22:50:33 	mpd: [pppoe] IPCP: rec'd Configure Reject #145 link 0 (Ack-Sent)
    Dec 27 22:50:33 	mpd: COMPPROTO VJCOMP, 16 comp. channels, no comp-cid
    Dec 27 22:50:33 	mpd: SECDNS 0.0.0.0
    Dec 27 22:50:33 	mpd: [pppoe] IPCP: SendConfigReq #146
    Dec 27 22:50:33 	mpd: IPADDR 0.0.0.0
    Dec 27 22:50:33 	mpd: PRIDNS 0.0.0.0
    Dec 27 22:50:33 	mpd: [pppoe] IPCP: rec'd Configure Nak #146 link 0 (Ack-Sent)
    Dec 27 22:50:33 	mpd: IPADDR 74.224.20.229
    Dec 27 22:50:33 	mpd: 74.224.20.229 is OK
    Dec 27 22:50:33 	mpd: PRIDNS 205.152.37.23
    Dec 27 22:50:33 	mpd: [pppoe] IPCP: SendConfigReq #147
    Dec 27 22:50:33 	mpd: PRIDNS 205.152.37.23
    Dec 27 22:50:33 	mpd: [pppoe] IPCP: rec'd Configure Nak #147 link 0 (Ack-Sent)
    Dec 27 22:50:33 	mpd: IPADDR 74.224.20.229
    Dec 27 22:50:33 	mpd: 74.224.20.229 is OK
    Dec 27 22:50:33 	mpd: [pppoe] IPCP: SendConfigReq #148
    Dec 27 22:50:33 	mpd: PRIDNS 205.152.37.23
    Dec 27 22:50:33 	mpd: [pppoe] IPCP: rec'd Configure Nak #148 link 0 (Ack-Sent)
    Dec 27 22:50:33 	mpd: IPADDR 74.224.20.229
    Dec 27 22:50:33 	mpd: 74.224.20.229 is OK
    Dec 27 22:50:33 	mpd: [pppoe] IPCP: SendConfigReq #149
    Dec 27 22:50:33 	mpd: PRIDNS 205.152.37.23
    Dec 27 22:50:33 	mpd: [pppoe] IPCP: rec'd Configure Nak #149 link 0 (Ack-Sent)
    Dec 27 22:50:33 	mpd: IPADDR 74.224.20.229
    Dec 27 22:50:33 	mpd: 74.224.20.229 is OK
    Dec 27 22:50:33 	mpd: [pppoe] IPCP: SendConfigReq #150
    Dec 27 22:50:33 	mpd: PRIDNS 205.152.37.23
    Dec 27 22:50:33 	mpd: [pppoe] IPCP: rec'd Configure Nak #150 link 0 (Ack-Sent)
    Dec 27 22:50:33 	mpd: IPADDR 74.224.20.229
    Dec 27 22:50:33 	mpd: 74.224.20.229 is OK
    Dec 27 22:50:33 	mpd: [pppoe] IPCP: SendConfigReq #151
    Dec 27 22:50:33 	mpd: PRIDNS 205.152.37.23
    Dec 27 22:50:33 	mpd: [pppoe] IPCP: rec'd Configure Nak #151 link 0 (Ack-Sent)
    Dec 27 22:50:33 	mpd: IPADDR 74.224.20.229
    Dec 27 22:50:33 	mpd: 74.224.20.229 is OK
    Dec 27 22:50:33 	mpd: [pppoe] IPCP: SendConfigReq #152
    Dec 27 22:50:33 	mpd: PRIDNS 205.152.37.23
    Dec 27 22:50:34 	mpd: [pppoe] IPCP: rec'd Configure Nak #152 link 0 (Ack-Sent)
    Dec 27 22:50:34 	mpd: IPADDR 74.224.20.229
    Dec 27 22:50:34 	mpd: 74.224.20.229 is OK
    Dec 27 22:50:34 	mpd: [pppoe] IPCP: SendConfigReq #153
    Dec 27 22:50:34 	mpd: PRIDNS 205.152.37.23
    Dec 27 22:50:34 	mpd: [pppoe] IPCP: rec'd Configure Nak #153 link 0 (Ack-Sent)
    Dec 27 22:50:34 	mpd: IPADDR 74.224.20.229
    Dec 27 22:50:34 	mpd: 74.224.20.229 is OK
    Dec 27 22:50:34 	mpd: [pppoe] IPCP: SendConfigReq #154
    Dec 27 22:50:34 	mpd: PRIDNS 205.152.37.23
    Dec 27 22:50:34 	mpd: [pppoe] IPCP: rec'd Configure Nak #154 link 0 (Ack-Sent)
    Dec 27 22:50:34 	mpd: IPADDR 74.224.20.229
    Dec 27 22:50:34 	mpd: 74.224.20.229 is OK
    Dec 27 22:50:34 	mpd: [pppoe] IPCP: not converging
    Dec 27 22:50:34 	mpd: [pppoe] IPCP: parameter negotiation failed
    Dec 27 22:50:34 	mpd: [pppoe] IPCP: state change Ack-Sent --> Stopped
    Dec 27 22:50:34 	mpd: [pppoe] IPCP: LayerFinish
    Dec 27 22:50:34 	mpd: [pppoe] bundle: CLOSE event in state OPENED
    Dec 27 22:50:34 	mpd: [pppoe] closing link "pppoe"...
    Dec 27 22:50:34 	mpd: [pppoe] link: CLOSE event
    Dec 27 22:50:34 	mpd: [pppoe] LCP: Close event
    Dec 27 22:50:34 	mpd: [pppoe] LCP: state change Opened --> Closing
    Dec 27 22:50:34 	mpd: [pppoe] LCP: phase shift NETWORK --> TERMINATE
    Dec 27 22:50:34 	mpd: [pppoe] setting interface ng0 MTU to 1500 bytes
    Dec 27 22:50:34 	mpd: [pppoe] up: 0 links, total bandwidth 9600 bps
    Dec 27 22:50:34 	mpd: [pppoe] IPCP: Down event
    Dec 27 22:50:34 	mpd: [pppoe] IPCP: state change Stopped --> Starting
    Dec 27 22:50:34 	mpd: [pppoe] IPCP: LayerStart
    Dec 27 22:50:34 	mpd: [pppoe] closing link "pppoe"...
    Dec 27 22:50:34 	mpd: [pppoe] LCP: SendTerminateReq #82
    Dec 27 22:50:34 	mpd: [pppoe] LCP: LayerDown
    Dec 27 22:50:34 	mpd: [pppoe] bundle: OPEN event in state CLOSED
    Dec 27 22:50:34 	mpd: [pppoe] opening link "pppoe"...
    Dec 27 22:50:34 	mpd: [pppoe] link: CLOSE event
    Dec 27 22:50:34 	mpd: [pppoe] LCP: Close event
    Dec 27 22:50:34 	mpd: [pppoe] link: OPEN event
    Dec 27 22:50:34 	mpd: [pppoe] LCP: Open event
    Dec 27 22:50:34 	mpd: [pppoe] LCP: state change Closing --> Stopping
    Dec 27 22:50:34 	mpd: [pppoe] LCP: rec'd Terminate Ack #82 link 0 (Stopping)
    Dec 27 22:50:34 	mpd: [pppoe] LCP: state change Stopping --> Stopped
    Dec 27 22:50:34 	mpd: [pppoe] LCP: phase shift TERMINATE --> ESTABLISH
    Dec 27 22:50:34 	mpd: [pppoe] LCP: LayerFinish
    Dec 27 22:50:34 	mpd: [pppoe] device: CLOSE event in state UP
    Dec 27 22:50:34 	mpd: [pppoe] device is now in state CLOSING
    Dec 27 22:50:34 	mpd: [pppoe] device: DOWN event in state CLOSING
    Dec 27 22:50:34 	mpd: [pppoe] device is now in state DOWN
    Dec 27 22:50:34 	mpd: [pppoe] link: DOWN event
    Dec 27 22:50:34 	mpd: [pppoe] LCP: Down event
    Dec 27 22:50:34 	mpd: [pppoe] LCP: state change Stopped --> Starting
    Dec 27 22:50:34 	mpd: [pppoe] LCP: phase shift ESTABLISH --> DEAD
    Dec 27 22:50:34 	mpd: [pppoe] LCP: LayerStart
    Dec 27 22:50:34 	mpd: [pppoe] device: OPEN event in state DOWN
    Dec 27 22:50:34 	mpd: [pppoe] pausing 6 seconds before open
    Dec 27 22:50:34 	mpd: [pppoe] device is now in state DOWN
    Dec 27 22:50:34 	mpd: [pppoe] device: OPEN event in state DOWN
    Dec 27 22:50:34 	mpd: [pppoe] device is now in state DOWN
    

    capture.txt



  • Here is my log -

    Last 50 system log entries
    Dec 27 16:39:06 mpd: IPADDR xxx.xxx.xxx.xxx
    Dec 27 16:39:06 mpd: xxx.xxx.xxx.xxx is OK
    Dec 27 16:39:06 mpd: [pppoe] IPCP: SendConfigReq #126
    Dec 27 16:39:06 mpd: PRIDNS 205.152.37.23
    Dec 27 16:39:06 mpd: [pppoe] IPCP: rec'd Configure Nak #126 link 0 (Ack-Sent)
    Dec 27 16:39:06 mpd: IPADDR xxx.xxx.xxx.xxx
    Dec 27 16:39:06 mpd: xxx.xxx.xxx.xxx is OK
    Dec 27 16:39:06 mpd: [pppoe] IPCP: not converging
    Dec 27 16:39:06 mpd: [pppoe] IPCP: parameter negotiation failed
    Dec 27 16:39:06 mpd: [pppoe] IPCP: state change Ack-Sent –> Stopped
    Dec 27 16:39:06 mpd: [pppoe] IPCP: LayerFinish
    Dec 27 16:39:06 mpd: [pppoe] bundle: CLOSE event in state OPENED
    Dec 27 16:39:06 mpd: [pppoe] closing link "pppoe"…
    Dec 27 16:39:06 mpd: [pppoe] link: CLOSE event
    Dec 27 16:39:06 mpd: [pppoe] LCP: Close event
    Dec 27 16:39:06 mpd: [pppoe] LCP: state change Opened –> Closing
    Dec 27 16:39:06 mpd: [pppoe] LCP: phase shift NETWORK –> TERMINATE
    Dec 27 16:39:06 mpd: [pppoe] setting interface ng0 MTU to 1500 bytes
    Dec 27 16:39:06 mpd: [pppoe] up: 0 links, total bandwidth 9600 bps
    Dec 27 16:39:06 mpd: [pppoe] IPCP: Down event
    Dec 27 16:39:06 mpd: [pppoe] IPCP: state change Stopped –> Starting
    Dec 27 16:39:06 mpd: [pppoe] IPCP: LayerStart
    Dec 27 16:39:06 mpd: [pppoe] closing link "pppoe"…
    Dec 27 16:39:06 mpd: [pppoe] LCP: SendTerminateReq #230
    Dec 27 16:39:06 mpd: [pppoe] LCP: LayerDown
    Dec 27 16:39:06 mpd: [pppoe] bundle: OPEN event in state CLOSED
    Dec 27 16:39:06 mpd: [pppoe] opening link "pppoe"…
    Dec 27 16:39:06 mpd: [pppoe] link: CLOSE event
    Dec 27 16:39:06 mpd: [pppoe] LCP: Close event
    Dec 27 16:39:06 mpd: [pppoe] link: OPEN event
    Dec 27 16:39:06 mpd: [pppoe] LCP: Open event
    Dec 27 16:39:06 mpd: [pppoe] LCP: state change Closing –> Stopping
    Dec 27 16:39:06 mpd: [pppoe] LCP: rec'd Terminate Ack #230 link 0 (Stopping)
    Dec 27 16:39:06 mpd: [pppoe] LCP: state change Stopping –> Stopped
    Dec 27 16:39:06 mpd: [pppoe] LCP: phase shift TERMINATE –> ESTABLISH
    Dec 27 16:39:06 mpd: [pppoe] LCP: LayerFinish
    Dec 27 16:39:06 mpd: [pppoe] device: CLOSE event in state UP
    Dec 27 16:39:06 mpd: [pppoe] device is now in state CLOSING
    Dec 27 16:39:06 mpd: [pppoe] device: DOWN event in state CLOSING
    Dec 27 16:39:06 mpd: [pppoe] device is now in state DOWN
    Dec 27 16:39:06 mpd: [pppoe] link: DOWN event
    Dec 27 16:39:06 mpd: [pppoe] LCP: Down event
    Dec 27 16:39:06 mpd: [pppoe] LCP: state change Stopped –> Starting
    Dec 27 16:39:06 mpd: [pppoe] LCP: phase shift ESTABLISH –> DEAD
    Dec 27 16:39:06 mpd: [pppoe] LCP: LayerStart
    Dec 27 16:39:06 mpd: [pppoe] device: OPEN event in state DOWN
    Dec 27 16:39:06 mpd: [pppoe] pausing 6 seconds before open
    Dec 27 16:39:06 mpd: [pppoe] device is now in state DOWN
    Dec 27 16:39:06 mpd: [pppoe] device: OPEN event in state DOWN
    Dec 27 16:39:06 mpd: [pppoe] device is now in state DOWN



  • My DSL is doing the same.  I get back from Christmas and I got a gift from AT$T.  The gift of aggrivation.  Gee thanks.  If someone knows how to fix this, Please help!!!



  • Can someone get a tcpdump and dump all packets and then send it to us?

    It appears to be failing around:

    Dec 27 16:39:06 mpd: [pppoe] IPCP: not converging
    Dec 27 16:39:06 mpd: [pppoe] IPCP: parameter negotiation failed



  • I am not going to be able to, we have switched out the eight routers that affected with temporary replacements. These are all at live locations.

    Interestingly enough, this seems to be a widespread but not total. I have a pfsense @ home on a BellSouth account and it is still up and running. 10 miles away I have 2 clients that both have a Monowall and both were affected.

    To sum up, we have a total of 8 locations all over Atlanta that were affected. It all started last Friday with some on Monday and one on Wednesday. It seems like they are upgrading something a little at a time.



  • I am happy to give a tcpdump if someone will explain how to do it.  I have 2 dsl accounts.  1 in bridge mode (now not working) and 1 in router mode.  I am happy to help however i can, but i am rather ignorant as to how



  • Has anyone called them to inquire about these changes?



  • I am sure someone there has a clue, but i don't think they let those people talk to customers on the phone.  In other words, yes I have called and they say there is nothing wrong.  the problem must be with our router.  ???



  • @Ryan:

    I am sure someone there has a clue, but i don't think they let those people talk to customers on the phone.  In other words, yes I have called and they say there is nothing wrong.  the problem must be with our router.  ???

    Yes, but what do they say when you tell them it worked FINE until their changes a week ago?

    I would put some heat on them.  Lord knows that if my ISP did this to me, I would become their worst nightmare in no time.



  • @sullrich:

    Yes, but what do they say when you tell them it worked FINE until their changes a week ago?

    I would put some heat on them.  Lord knows that if my ISP did this to me, I would become their worst nightmare in no time.

    Basically what happened in my case:  I told them that everything was working fine until a couple of days ago.  Asked if they changed something on their end.  "My records don't show any changes…" They had me bypass the router completely and connect directly from XP.  When this worked, they told me "it must be a problem with your router.  What brand is it? Linksys? Netgear?" flips page in script  I couldn't get them to pass me along to anyone more clueful.

    I can probably get a trace of the pppoe connection.  However, I'm not sure how to go about it, since I don't have a non-switch hub that I can use to snoop the traffic.  Any suggestions?



  • Yeah, typical huge companies.  That's really bad to hear.



  • This is also happening to our at&t/bellsouth dsl,  north of atlanta.



  • I've spent some time doing some investigation on this issue.  I set up freebsd on an old machine I had lying around to debug the problem.  The first thing I tried was to try to establish a pppoe link using the more recent mpd4 instead of the mpd 3.18 that m0n0wall uses.  Unfortunately, that didn't work; it failed with a similar error.

    Next I tried capturing the network traffic to see what was going on.  Here are the results.  The first link is MPD 3.18 on freebsd, which is showing the problem.  The second is a capture from the pppoe client on XP, which works.
    http://attenuated.org/~greg/mpd/mpd_notworking.txt
    http://attenuated.org/~greg/mpd/xp_working.txt

    After comparing the traffic, it seems that after getting the NAK from the server, MPD is not filling out the ip address from the NAK in the subsequent request.  I am determined to get this working, so I am going to try to debug the problem and hopefully come up with a patch that fixes the problem.



  • @megalodon:

    I've spent some time doing some investigation on this issue.  I set up freebsd on an old machine I had lying around to debug the problem.  The first thing I tried was to try to establish a pppoe link using the more recent mpd4 instead of the mpd 3.18 that m0n0wall uses.  Unfortunately, that didn't work; it failed with a similar error.

    Next I tried capturing the network traffic to see what was going on.  Here are the results.  The first link is MPD 3.18 on freebsd, which is showing the problem.  The second is a capture from the pppoe client on XP, which works.
    http://attenuated.org/~greg/mpd/mpd_notworking.txt
    http://attenuated.org/~greg/mpd/xp_working.txt

    After comparing the traffic, it seems that after getting the NAK from the server, MPD is not filling out the ip address from the NAK in the subsequent request.  I am determined to get this working, so I am going to try to debug the problem and hopefully come up with a patch that fixes the problem.

    Great, thanks!  We surely will include it if you can get it working.



  • You're the man, megalodon.  I'm eagerly watching this thread for a fix.  I'd try to work on it myself, but I honestly don't know where to begin!



  • When I started this thread I suspected an ISP change, and it appears that it was..

    One other piece of info here is that I have a static IP Address.  Not sure if this is only affecting people with statics or what…

    From looking at the TCP dump info, posted above (Sorry if this is redundant - I just posted it because it helped me "think it through", and I thought others might also be interested):

    Both sides get through CHAP and move into configuration negotiations.  Eventually, they both get to the same basic point:  Frame 16 in the MPD file looks essentially the same as frame 22 in the XP file.  In both cases, the next frame AT&T sends is a Nak containing the IP Address and Primary DNS server IP.  MPD replies with a frame containing the Primary DNS server IP.  XP replies with a frame containing the IP Address and Primary DNS server IP.

    In looking at the way this negotiation has been going up to this point, it appears that options have been rejected by the ISP until it gets down to these two - The IP Address and primary DNS server.

    To sum up: AT&T is expecting to send the IP Address and Primary DNS server to the client, and that they expect the client to echo that information back to them (a confirmation, I suppose) before they will give an Ack.

    Next, I downloaded the source of MPD 3.18.  I'm having a bit of a time following it, as I haven't done much C programming in many years, but this is what I've gathered from it:

    ipcp.c appears to be the most closely related file that I could track down.   
    Two constants, TY_PRIMARYDNS and TY_IPADDR, look to be deeply involved.  A function on line 339 is called IpcpBuildConfigReq.  Inside that function, a comment on line 364 states that it will "Add option if we desire it and it hasn't been rejected".  That leads me to believe that MPD mistakenly believes the TY_IPADDR option has been rejected by the ISP, so it isn't offering that anymore.

    Further looking at the code makes me think we might be able to jury-rig it to work by just commenting out line 344, which is an "if" statement.  This would cause line 345 to always be executed, which I think would cause the IP Address to always be in the request.

    Unfortunately, I don't have a pfSense test system to try it out on.  Is anyone with an appropriate system game on trying out this mod to ipcp.c?  That, of course, assumes that pfSense is using this version of MPD.



  • On second thought, this might be a safer "Hack":

    Insert this code in ipcp.c, within the IpcpBuildConfigReq function, just about line 346:

    /* Hack for BellSouth/AT&T PPPoE behavior */
      if (fp->reqid == (FSM_MAXNAK - 1))
    cp = FsmConfValue(cp, TY_IPADDR, 4, &ipcp->want_addr.s_addr);

    If I've read the code correctly, that should wait until the last attempt before it would normally fail, at which time it would insert the IP Address field in the Configuration Request, regardless of if it has been previously rejected.  This would be less likely to break MPD, if this code is used for other purposes.  (I believe FSM_MAXNAK is a valid variable here, but as I indicated above, my C is rusty.)



  • Warning: Object directory not changed from original /usr/ports/net/mpd/work/mpd-3.18/src
    cc -O2 -fno-strict-aliasing -pipe  -DPATH_CONF_DIR="/usr/local/etc/mpd" -DSYSLOG_FACILITY=LOG_DAEMON -DMPD_VERSION='"3.18 (root@freebsd6.geekgod.com 01:13 31-Dec-2007)"' -g -Wall  -Wcast-align  -Wchar-subscripts  -Wformat  -Winline  -Wmissing-declarations  -Wmissing-prototypes  -Wnested-externs  -Wpointer-arith  -Wwrite-strings -DPHYSTYPE_MODEM -DPHYSTYPE_UDP -DPHYSTYPE_NG_SOCKET -DPHYSTYPE_PPTP '-DPPTP_VENDOR_NAME="FreeBSD mpd-3.18"' -DPHYSTYPE_PPPOE -DENCRYPTION_DES -DENCRYPTION_MPPE -DCOMPRESSION_MPPC  -c ipcp.c
    ipcp.c: In function IpcpBuildConfigReq': ipcp.c:352: error:FSM_MAXNAK' undeclared (first use in this function)
    ipcp.c:352: error: (Each undeclared identifier is reported only once
    ipcp.c:352: error: for each function it appears in.)
    *** Error code 1

    Stop in /usr/ports/net/mpd/work/mpd-3.18/src.



  • I must have been tired last night.  In ipcp.c, it includes fsm.h, but that constant is defined on line 23 of fsm.c.

    We could move the Definition of that constant from fsm.c to fsm.h.  If (for some odd reason) that didn't work, we could always hard-wire the if statement that I suggested to 9 instead of (FSM_MAXNAK - 1).. (Yea, ugly, I know)

    Or just comment out the first "if" in that function (line 344 on my copy):
    if (!IPCP_REJECTED(ipcp, TY_IPADDR) || ipcp->want_addr.s_addr == 0)

    so that the next line is always executed:
    cp = FsmConfValue(cp, TY_IPADDR, 4, &ipcp->want_addr.s_addr);



  • I spent some time looking at the code last night, and I think I know why it's failing.  The problem section is definitely this part in ipcp.c:

    
      /* Put in my desired IP address */
      if (!IPCP_REJECTED(ipcp, TY_IPADDR) || ipcp->want_addr.s_addr == 0)
        cp = FsmConfValue(cp, TY_IPADDR, 4, &ipcp->want_addr.s_addr);
    
    

    You can hack it to make it work by removing the if statement so that the FsmConfValue line is always executed.  The if statement is essentially checking to see if the IPADDR option has been rejected by the server.  If not, it should add its desired ipaddr to the request.

    The reason this is failing is because Bellsouth is rejecting the SECONDARYDNS option, which has a value of 131.  MPD stores the rejected options in a bit mask.  Look at the IPCP_REJECTED and IPCP_PEER_REJ macros towards the top of ipcp.c.

    The problem is that when you left shift 131 places (the value of TY_SECONDARYDNS), it's going to wrap the peer_reject variable a bunch of times, with the end result the same as if you had shifted to the left by 3, which just so happens to be the value of TY_IPADDR.

    So, to make a long story short, if the server rejects the SECONDARYDNS option, the MPD code thinks the IPADDR option has been rejected.  The way it keeps track of rejected options is fundamentally broken.  I could probably come up with a patch to redo the IPCP_REJECTED/IPCP_PEER_REJ macros within a couple of days (unless there is an MPD dev reading this, who could probably do it a lot faster, and integrate it into the main MPD codebase)

    The question I have is: is it really so bad to take out that if statement so that the desired ipaddr is always added to the request?  I can't think of a reason why the server would ever reject the IPADDR option.  I don't really know the ins and outs of IPCP though.



  • @ptaylor:

    If I've read the code correctly, that should wait until the last attempt before it would normally fail, at which time it would insert the IP Address field in the Configuration Request, regardless of if it has been previously rejected.  This would be less likely to break MPD, if this code is used for other purposes.  (I believe FSM_MAXNAK is a valid variable here, but as I indicated above, my C is rusty.)

    I'm not sure this is the right approach… it should actually be putting the IP in the first request after the first NAK.  This is what XP's PPPoE client does:

    http://attenuated.org/~greg/mpd/xp_working.txt



  • @megalodon:

    I'm not sure this is the right approach… it should actually be putting the IP in the first request after the first NAK.  This is what XP's PPPoE client does:

    http://attenuated.org/~greg/mpd/xp_working.txt

    I'm not sure it is the right approach either, but thought that it would be less likely to cause issues if this code is used for some purpose other than what we are doing with it here.  I believe that MPD can run in a server mode as well, and perhaps for other purposes.  I do not know if Monowall or pfSense are taking advantage of any of those features, though.

    Your previous post indicates the issue is with the way MPD determines what is rejected.  I think you are spot-on here, as that code segment you posted is the what I homed in on as well.  Fixing the underlying cause would be the best option, but that would be best handled by someone intimately familiar with the MPD code, as I don't think it would be trivial.  If someone can verify that MPD's only use in pfSense is as a PPPoE client, then the suggestion to simply take the "if" statement out would probably take care of everything in short order.



  • Try editing config.xml and in the pppoe section add:

    <dnsnosec>IE: from /etc/inc/interfaces.inc:

    if (!isset($config['pppoe']['dnsnosec'])) {
    $mpdconf .= << <eod<br>set ipcp enable req-sec-dns

    EOD;</eod<br></dnsnosec>



  • Scott, you rock!  This minor config change has me up and running!

    Perhaps this can be put in as a webGui option since this is becoming a common issue.



  • Excellent!  That got me up and running as well.  Thanks for finding that.

    I went ahead and submitted a bug ticket to the MPD project so that they are at least aware of the problem:
    http://sourceforge.net/tracker/index.php?func=detail&aid=1861740&group_id=14145&atid=114145

    I also whipped up a patch to add an option to the webgui.  This patch is against m0n0wall 1.3b7.  It should be fairly easy to adapt to pfsense.  I'm going to post this to the monowall development mailing list.  Hopefully it'll get included in the next release:

    --- interfaces_wan.php.orig     2008-01-01 13:04:15.000000000 -0500
    +++ interfaces_wan.php  2008-01-01 13:35:52.000000000 -0500
    @@ -38,6 +38,7 @@
     $pconfig['username'] = $config['pppoe']['username'];
     $pconfig['password'] = $config['pppoe']['password'];
     $pconfig['provider'] = $config['pppoe']['provider'];
    +$pconfig['pppoe_dnsnosec'] = isset($config['pppoe']['dnsnosec']);
     $pconfig['pppoe_dialondemand'] = isset($config['pppoe']['ondemand']);
     $pconfig['pppoe_idletimeout'] = $config['pppoe']['timeout'];
    
    @@ -179,6 +180,7 @@
                    unset($config['pppoe']['username']);
                    unset($config['pppoe']['password']);
                    unset($config['pppoe']['provider']);
    +               unset($config['pppoe']['dnsnosec']);
                    unset($config['pppoe']['ondemand']);
                    unset($config['pppoe']['timeout']);
                    unset($config['pptp']['username']);
    @@ -208,6 +210,7 @@
                            $config['pppoe']['username'] = $_POST['username'];
                            $config['pppoe']['password'] = $_POST['password'];
                            $config['pppoe']['provider'] = $_POST['provider'];
    +                       $config['pppoe']['dnsnosec'] = $_POST['pppoe_dnsnosec'] ? true : false;
                            $config['pppoe']['ondemand'] = $_POST['pppoe_dialondemand'] ? true : false;
                            $config['pppoe']['timeout'] = $_POST['pppoe_idletimeout'];
                    } else if ($_POST['type'] == "PPTP") {
    @@ -274,6 +277,7 @@
                            document.iform.username.disabled = 1;
                            document.iform.password.disabled = 1;
                            document.iform.provider.disabled = 1;
    +                       document.iform.pppoe_dnsnosec.disabled = 1;
                            document.iform.pppoe_dialondemand.disabled = 1;
                            document.iform.pppoe_idletimeout.disabled = 1;
                            document.iform.ipaddr.disabled = 0;
    @@ -297,6 +301,7 @@
                            document.iform.username.disabled = 1;
                            document.iform.password.disabled = 1;
                            document.iform.provider.disabled = 1;
    +                       document.iform.pppoe_dnsnosec.disabled = 1;
                            document.iform.pppoe_dialondemand.disabled = 1;
                            document.iform.pppoe_idletimeout.disabled = 1;
                            document.iform.ipaddr.disabled = 1;
    @@ -320,6 +325,7 @@
                            document.iform.username.disabled = 0;
                            document.iform.password.disabled = 0;
                            document.iform.provider.disabled = 0;
    +                       document.iform.pppoe_dnsnosec.disabled = 0;
                            document.iform.pppoe_dialondemand.disabled = 0;
                            if (document.iform.pppoe_dialondemand.checked || enable_change) {
                                    document.iform.pppoe_idletimeout.disabled = 0;
    @@ -347,6 +353,7 @@
                            document.iform.username.disabled = 1;
                            document.iform.password.disabled = 1;
                            document.iform.provider.disabled = 1;
    +                       document.iform.pppoe_dnsnosec.disabled = 1;
                            document.iform.pppoe_dialondemand.disabled = 1;
                            document.iform.pppoe_idletimeout.disabled = 1;
                            document.iform.ipaddr.disabled = 1;
    @@ -374,6 +381,7 @@
                            document.iform.username.disabled = 1;
                            document.iform.password.disabled = 1;
                            document.iform.provider.disabled = 1;
    +                       document.iform.pppoe_dnsnosec.disabled = 1;
                            document.iform.pppoe_dialondemand.disabled = 1;
                            document.iform.pppoe_idletimeout.disabled = 1;
                            document.iform.ipaddr.disabled = 1;
    @@ -501,6 +509,13 @@
    
     Hint: this field can usually be left
                         empty
    
    +                
    +                  Secondary DNS
    +                   >
    +                    **Disable secondary DNS request**
    
    +                    This option disables the request for a secondary DNS server in the PPPoE connection.  Workaround for some BellSouth/AT&T DSL users.
    +                  
    +                
    
                       Dial on demand
                        onClick="enable_change(false)" >
    
    


  • Scott Rocks.  Thanks a bunch.  I am honored to be in a forum with such genious.  You are da bomb.  Oh and if you can't tell the fix works great for me.



  • Great, glad it worked out.  We are now exploring what to do about this.  I almost half inclined to ship with this option on by default and if someone needs multiple dns servers they can uncomment this.

    Thoughts?



  • For now, I think having this option enabled by default would probably be fine.

    In my mind, though, the best long term fix would be for the MPD dev's fix this little bug.

    I've never heard of an ISP not giving you more than one DNS server before, which is probably why this problem has remained hidden until now.

    As a side note, my employer uses BellSouth for their Internet service, and since I work in the network department there, I've been involved with issues courtesy of their DNS servers on numerous occasions.  Perhaps they felt that only handing out a single DNS server (splitting up who gets handed which server) would half the traffic to their servers, possibly cutting down on the problems.  This seems like a short-sited fix though, since that introduces more single points of failure for their customers.

    If fixing their DNS issues was the goal of this change, I'd think a better answer would be to get a couple of F5 load balancers and a few more DNS servers.  Those F5's aren't cheap, though…  Hey - They could use pfSense instead!  :)



  • Thanks so much for the help. This one hit me right after new years. Damn Bell South.



  • This issue has been fixed in MPD 5 and Manuel has back-ported the fix to MPD 3.18.  He has a pre-release version that has been verified to work (without the dnsnosec option).

    So, this issue sounds like it will be all taken care of very soon.

    Paul



  • I've just commited a new mpd which contains his changes.  Please check the snapshot server in about an hour or two from now.


Log in to reply