WAN interfaces fail to return after power outage

bfeitell

It is an insidious bug. I triggered the DHCP renewal problems by saving and applying on the WAN with or without changes. Unless triggered by the user, it will lurk until the next DHCP renewal fails, and that may not happen for 30 minutes or more. Looking through recent forum posts, I suspect this bug is in play whenever a user notices arpresolve: can't set llinfo for $GATEWAY on $INTERFACE errors.

peter_richardson

Okay so it went a bit over my head there. Can someone please break it down for me? Where are we up to with this one? Is there a patch? Or a configuration change is needed?

stephenw10

First try just adding the following to the option modifiers field in the advanced section of the dhcp setup on WAN. Check the 'Advanced Configuration' box to see that field if it's not already.
supersede interface-mtu 0

If that works then you can try the patch instead. That would be a helpful test for us.

Steve

peter_richardson

Hey Steve, okay I tried it (see screenshot) but didn't change anything. System log didn't report any errors this time though. I didn't have the WWAN connected this time so that's why it's not showing up. To get the WAN connection going again after power out, I need to either unplug the Ethernet cable and re plug, or disable and re enable the interface, or make some change in the WAN interface and save.

stephenw10

Hmm, no 'arpresolve' errors though?

Did you ever try running ifconfig -av during the working and non-working states to compare them?

Steve

peter_richardson

Did you see the above images? One of them is of the syslog while it was happening after putting in the string into the Option Modifiers field.

Yes I did, but it's on my other computer, I'll paste it here later.

stephenw10

Yes I see those. I don't see any arpresolve errors in there but I thought you may have seen some that aren't in that shot. That only shows 10s worth of logs.

Steve

peter_richardson

yeah that's the last few entries from it booting, then there's nothing else after that.

Here's the output from ifconfig -v but according to www.diffchecker.com there's no difference in the output whether it's working properly or not:

Petes-MBP:~ Peter$ ifconfig -v
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384 index 1
 	eflags=12000000<ECN_DISABLE,SENDLIST>
 	options=1203<RXCSUM,TXCSUM,TXSTATUS,SW_TIMESTAMP>
 	inet 127.0.0.1 netmask 0xff000000 
 	inet6 ::1 prefixlen 128 
 	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1 
 	nd6 options=201<PERFORMNUD,DAD>
 	link quality: 100 (good)
 	state availability: 0 (true)
 	timestamp: disabled
 	qosmarking enabled: no mode: none
gif0: flags=8010<POINTOPOINT,MULTICAST> mtu 1280 index 2
 	eflags=1000000<ECN_ENABLE>
 	state availability: 0 (true)
 	qosmarking enabled: no mode: none
stf0: flags=0<> mtu 1280 index 3
 	eflags=1000000<ECN_ENABLE>
 	state availability: 0 (true)
 	qosmarking enabled: no mode: none
XHC20: flags=0<> mtu 0 index 4
 	eflags=41000000<ECN_ENABLE,FASTLN_ON>
 	state availability: 0 (true)
 	qosmarking enabled: yes mode: none
en0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500 index 5
 	eflags=41200880<TXSTART,ARPLL,NOACKPRI,ECN_ENABLE,FASTLN_ON>
 	ether a0:99:9b:14:37:55 
 	inet6 fe80::42d:fd6:9a6d:c5ec%en0 prefixlen 64 secured scopeid 0x5 
 	inet 10.20.63.133 netmask 0xffffff00 broadcast 10.20.63.255
 	nd6 options=201<PERFORMNUD,DAD>
 	media: autoselect
 	status: active
 	type: Wi-Fi
 	link quality: 100 (good)
 	state availability: 0 (true)
 	scheduler: FQ_CODEL (driver managed)
 	link rate: 53.95 Mbps
 	qosmarking enabled: yes mode: none
p2p0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 2304 index 6
 	eflags=41000080<TXSTART,ECN_ENABLE,FASTLN_ON>
 	ether 02:99:9b:14:37:55 
 	media: autoselect
 	status: inactive
 	type: Wi-Fi
 	state availability: 0 (true)
 	scheduler: FQ_CODEL (driver managed)
 	link rate: 10.00 Mbps
 	qosmarking enabled: yes mode: none
awdl0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1484 index 7
 	eflags=413e0080<TXSTART,LOCALNET_PRIVATE,ND6ALT,RESTRICTED_RECV,AWDL,NOACKPRI,ECN_ENABLE,FASTLN_ON>
 	ether 0a:60:dd:0c:a4:0f 
 	inet6 fe80::860:ddff:fe0c:a40f%awdl0 prefixlen 64 scopeid 0x7 
 	nd6 options=201<PERFORMNUD,DAD>
 	media: autoselect
 	status: active
 	type: Wi-Fi
 	state availability: 0 (true)
 	scheduler: FQ_CODEL (driver managed)
 	link rate: 10.00 Mbps
 	qosmarking enabled: yes mode: none
en1: flags=8963<UP,BROADCAST,SMART,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500 index 8
 	eflags=41000080<TXSTART,ECN_ENABLE,FASTLN_ON>
 	options=60<TSO4,TSO6>
 	ether 6a:00:00:9f:55:50 
 	media: autoselect <full-duplex>
 	status: inactive
 	type: Ethernet
 	state availability: 0 (true)
 	scheduler: FQ_CODEL 
 	qosmarking enabled: yes mode: none
en2: flags=8963<UP,BROADCAST,SMART,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500 index 9
 	eflags=41000080<TXSTART,ECN_ENABLE,FASTLN_ON>
 	options=60<TSO4,TSO6>
 	ether 6a:00:00:9f:55:51 
 	media: autoselect <full-duplex>
 	status: inactive
 	type: Ethernet
 	state availability: 0 (true)
 	scheduler: FQ_CODEL 
 	qosmarking enabled: yes mode: none
bridge0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500 index 10
 	eflags=41000000<ECN_ENABLE,FASTLN_ON>
 	options=63<RXCSUM,TXCSUM,TSO4,TSO6>
 	ether 6a:00:00:9f:55:50 
 	Configuration:
 		id 0:0:0:0:0:0 priority 0 hellotime 0 fwddelay 0
 		maxage 0 holdcnt 0 proto stp maxaddr 100 timeout 1200
 		root id 0:0:0:0:0:0 priority 0 ifcost 0 port 0
 		ipfilter disabled flags 0x2
 	member: en1 flags=3<LEARNING,DISCOVER>
 	        ifmaxaddr 0 port 8 priority 0 path cost 0
 	        hostfilter 0 hw: 0:0:0:0:0:0 ip: 0.0.0.0
 	member: en2 flags=3<LEARNING,DISCOVER>
 	        ifmaxaddr 0 port 9 priority 0 path cost 0
 	        hostfilter 0 hw: 0:0:0:0:0:0 ip: 0.0.0.0
 	media: <unknown type>
 	status: inactive
 	state availability: 0 (true)
 	qosmarking enabled: yes mode: none
utun0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> mtu 2000 index 11
 	eflags=1002080<TXSTART,NOAUTOIPV6LL,ECN_ENABLE>
 	inet6 fe80::aae2:a7ca:ad8a:540%utun0 prefixlen 64 scopeid 0xb 
 	nd6 options=201<PERFORMNUD,DAD>
 	agent domain:ids501 type:clientchannel flags:0xc3 desc:"IDSNexusAgent ids501 : clientchannel"
 	state availability: 0 (true)
 	scheduler: FQ_CODEL 
 	qosmarking enabled: no mode: none
utun1: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> mtu 1380 index 12
 	eflags=1002080<TXSTART,NOAUTOIPV6LL,ECN_ENABLE>
 	inet6 fe80::a4b9:a2b2:95c:b086%utun1 prefixlen 64 scopeid 0xc 
 	inet6 fdb7:98e5:bc83:2490:a4b9:a2b2:95c:b086 prefixlen 64 
 	nd6 options=201<PERFORMNUD,DAD>
 	state availability: 0 (true)
 	scheduler: FQ_CODEL 
 	qosmarking enabled: no mode: none
Petes-MBP:~ Peter$

jimp

@peter_richardson said in WAN interfaces fail to return after power outage:

yeah that's the last few entries from it booting, then there's nothing else after that.

Here's the output from ifconfig -v but according to www.diffchecker.com there's no difference in the output whether it's working properly or not:

That appears to be from your Mac, not pfSense. Try that command on pfSense when it works and when it doesn't.

peter_richardson

@jimp sorry, that was a silly mistake! I'll test again later today and report back.

peter_richardson

@jimp said in WAN interfaces fail to return after power outage:

hat command on pfSense when it works and when it doesn't.

Hey guys, sorry for the late reply, I have finally had a spare minute to do some testing. Please see attached.

EDIT: I CAN'T REPLY BECAUSE I GET THIS ERROR:

ERROR
Post content was flagged as spam by Akismet.com

stephenw10

What are you trying to post? The ifconfig output?

It will probably allow you to post a screenshot of it if it's still blocking you.

Steve

peter_richardson

Yes, 2 of then but that would make 4 images plus the other image, and then you can't copy and paste text for analysis. Can we please fix this issue? Why would it mark my post as spam when I am a registered user, replying to a thread that I created, from the same public ip as always? And why not also tell me why it has been marked as spam so I can try to fix whatever it's issue is?

peter_richardson

for now, click here to view my reply:
https://www.evernote.com/l/ADhKbADk7v9OEqpkJOKxMDlLcgZZPWWDJXs

peter_richardson

anyone?

stephenw10

I up-voted enough of your posts that you should not get hit by Akismet again.

Do you see anything logged showing it getting the 0.0.0.0 address? It looks like the WAN doesn't come back up because it sees that as a valid address. If something giving that to it you can exclude a particular DHCP server as a source.

Steve

peter_richardson

Thanks Stephen!

So you're saying that it appears that pfSense is giving the WWAN interface the address of 0.0.0.0?

stephenw10

No, more likely it's something upstream giving it that before the connection comes up. Cable modems do that with monotonous regularity. But I'm not sure what sort of hand-off you have. I know NBN can be several different things.
If that is the case though you should see something logged in the DHCP logs as the dhclient pulls the bad address.

Steve

peter_richardson

I see. No, the WWAN isn't NBN, the WWAN is 4G with a Telstra SIM card, using a Netgear Nighthawk M1.

So changing the WWAN interface from DHCP to static with an IP address that is in the M1's usual DHCP range should fix this. IE: set the IP to 192.168.0.50 but alas, it just shows on the dashboard as "gateway down". So it looks like this isn't the issue. In fact, once I set the IP to static, it didn't work at all, which makes me wonder what is going on since previously, any changes to the WWAN connection would make the connection come back online, and giving it a static address shouldn't prevent it from working under normal conditions, so long as the address is within the M1's normal DHCP range.

stephenw10

Hmm, well the DHCP logs should still show if the bad address is coming from somewhere upstream. If it's coming from a different DHCP server you can block it as a source and the client will continue to look for other servers. That works well with cable modems that do this.
Anyway check the logs for 0.0.0.0 messages.

Steve