dhclient exiting on WAN

[tomashk]

Thank you very much for suggestions. I'll start analysis after work and check what works as quick fix for me and if I have time in the next few days I'll try to go deeper, but in that case I will try to at least understand why it is broken for me because it might help others. But I guess first I have to read suggested sources and only after that think about fixing anything :)

[tomashk]

1. dhclient problem:

• Timing parameters (backoff, etc.) doesn't work in a few configurations I tried. Probably because dhclient is no able to start. The only line with dhclient in logs is:

rc.bootup: The command '/sbin/dhclient -c /var/etc/dhclient_wan.conf igb0 > /tmp/igb0_output 2> /tmp/igb0_error_output' returned exit code '1', the output was ''

and contents of /tmp/igb0_output is

igb0: no link .............. giving up

• I tried to wait but without my intervention (renew button in interfaces status page) dhclient is never called in my scenario
• Now I remember that I had almost the same problem many years ago but at some point it just started to work. Maybe something changed in pfsense or maybe modem timings changed.

2. "Additional" problem

• Still when I renew WAN dhclient is called and I'm getting WAN IP. But all devices on LAN's side doesn't get Internet access. It is strange because pfsense shows IP for WAN and dpinger (after forcing its start) provide stats for gateway widget
• As suggested I connected with putty to pfsense and tried pinging google. Both IP and domain complained about connection. Especially interesting was pinging of IP address because it returned "ping: sendto: No route to host" error.
• After that I looked to Diagnostics / Routes and it looked properly until I compared it with properly working situation and I have noticed that default route is not there. I mentioned that I have to save for example routing configuration to have working internet connection. The change that happens at that point is that default route is added to routing table and with that internet works.

So that was short update in case somebody sees something interesting. Thanks to previous suggestions now I have some new information to work with.

[stephenw10]

Is WAN_DHCP set as the default gateway in System > Routing?

Do you have more than one IPv4 gateway shown there?

[tomashk]

@stephenw10
Now it is like that:
login-to-view
Previously it has been set as Automatic but I have just switched it to WAN_DHPC and it behaves exactly the same (or at least I don't see any changes). Only after pressing "Save" on System/Routing/Gateways page I got default route.

Also as you can see IPv6 gateway is set to none if it matters but I don't use IPv6 so I don't think that it means anything.

[tomashk]

Also I looked in the logs to see how it behaves when I used renew button for interface (scenario 1) to get IP (when dhclient did not work for me on the start of modem) and how it behaves when I get properly working internet and I just unplug ethernet cable from modem and plug it back (scenario 2).

• scenario 1
  login-to-view
• scenario 2
  login-to-view

As we can see first one doesn't add default route. Unfortunately I have not analysed it enough to know if it is bad or not. I just show my observations

[stephenw10]

Hmm, so in scenario 1 you end up without a default route even though you do have a gateway?

[tomashk]

@stephenw10
Yes. Now with working internet I have situation from the picture below. With scenario 1 I don't have the first line starting with default
login-to-view

I will test a little bit more soon because I want to rule any situation that those problems are caused by my mistakes with configuration. I believe those problems started with ver 2.6/22.01 but I can't be sure - I haven't noticed it previously because I had long uptime and there was no need for turning it off.

[stephenw10]

Mmm, there are a number of threads reporting similar issues. It look to be real IMO but it's not something we can easily replicate. It seems to be timing related.

Steve

[Gertjan]

Scenario 1: from bottom to top :

login-to-view

Scenario 2: from bottom to top :

login-to-view

So, if scenario 2 is what should happen = adding the 'default' route, it is not logged and thus not happening in scenario 1.
Maybe the IP was the same so the "add route default" wasn't needed ?
As far as know, dhclient uses shell scripts to do this. I'll have a look into that this weekend.

[tomashk]

Sorry for late response - I was away for a few days.

@gertjan said in dhclient exiting on WAN:

Maybe the IP was the same so the "add route default" wasn't needed ?

Probably IP was the same. Unfortunately I've lost those logs but currently whenever I renew I get the same WAN IP.

PS. For test I installed pfsense on another device and restored my configuration. Of course it works properly every time. Probably it will take some time to find how it differs when compared with "wrong logs". The main difference I see from the start is that when it works (on new installation) dhclient is able to start properly. So there is nothing like

rc.bootup: The command '/sbin/dhclient -c /var/etc/dhclient_wan.conf igb0 > /tmp/igb0_output 2> /tmp/igb0_error_output' returned exit code '1', the output was ''

in the logs. And later I can see standard DHCPDISCOVER (or others) or dhclient's messages about link state like

igb0 link state up -> down
igb0 link state down -> up

I guess I'll try to find out what might stop dhclient from starting and throwing

igb0: no link .............. giving up

into /tmp/igb0_output. Of course I'll try to find other differences, but this one looks like the main one.

[Gertjan]

@tomashk said in dhclient exiting on WAN:

I guess I'll try to find out what might stop dhclient from starting and throwing

igb0: no link .............. giving up

into /tmp/igb0_output

You've answered your own question.

The dhclient aborts when there is no WAN interface any more.
It stops executing, as the interface WAN isn't listed as a system resource any more.

Your issue isn't dhclient , that does exacly what it is designed to do.

Focus on the WAN interface : make it stopping going down - up - down - up - etc.

The thing is, you know already who brings the interface down : the other device in front of it.
Modems do so very often. It's their way to 'signal' that they created a "soft link" with the upstream ISP.
Or : this scenario : it powers up. The internal web GUI comes up, and the modem brings up it's LAN == the wan of pfSense. When the dhclient starts now, and send a DHCPDISCOVER, it will obtain an RFC1918 from the DHCP server of the modem. This RFC1918 will permit you to connect to the GUI of the modem to change it settings.
Moments later, the modem finished building the soft link with the ISP.
It brings down it's LAN, as it has to as this is the only way to reset the current connection, and brings it again. Now, when the dhclient of pfSense send a DHCPDISCOVER, the modem will be transparent, and the DHCPDISCOVER will reach the ISP. This time, the DHCP server of the ISP will reply, and give you a 'real' WAN IP.
This is how most modems work. For decades now.

Things become even more interesting if initially the modem has a hard time syncing with the ISP. Every time it looses the connection, because, for example, the initial speed was to high, it aborts, (bring its LAN down), and tries a connection with a lower speed.

dhclient, on the pfSense side, looses track.

Your mission, as a modem user, is to find the right timings settings.
You know it('s possible, because most often, when you hook up a PC to your modem and use that pppoe connection setting, your PC will connect.
Your PC also uses a dhcp client like software.

[stephenw10]

@tomashk said in dhclient exiting on WAN:

For test I installed pfsense on another device and restored my configuration. Of course it works properly every time.

What's the difference between those devices?

The is probably a timing issue. If the upstream device resets the link it may only be down for a short time and that must coincide with the dhclient script running in order to fail.
Running on hardware that is significantly faster or slower could well change the timing enough to prevent it happening.

It should not be possible to hit that but it may also be easy to work around by adding a boot delay in pfSense.

Steve

[tomashk]

@stephenw10 The one failing is based on small Asrock motherboard with Intel Celeron J4005 and working one is hp t620 plus thin client. I believe the second one is taking longer to start.

As you said this is probably a timing issue. The idea with boot delay sounds like good quick fix once I'll try to go back to J4005. For now I'll have to leave it because work doesn't leave me much time to experiment. For a moment I have that temporary new device which works. But in a week or two I am going to get back to it and try to understand it better.

By understand I mean for example that long time ago I looked into https://github.com/pfsense/FreeBSD-src/blob/devel-12/sbin/dhclient/dhclient.c (the file is important - I just took random branch from github as example) and I found the code that gives up (line 469)

login-to-view

Due to value 10 in line 468 and sleep from line 472 dhlient will try to wait for WAN interface only for 10 seconds. I was thinking that maybe it would be good to change value 10 to environment variable (with default value 10) so user can set this, but there are two problems with my current poor understanding:

• I'm not able to predict what would be broken in completely different part of the system if dhclient starts too long (maybe nothing but maybe some kind of the butterfly effect :) )
• there might already exist some system script that should be used for such situations

[stephenw10]

Mmm, I agree increasing that initially seems to be a good idea but could easily cause problems on systems with multiple disconnected WANs.

It would be nice to have that value configurable.

As a simple test you could just set the line:

autoboot_delay="30"

In /boot/loader.conf. That's normally 3s. If that works you might put it in loader.conf.local instead to prevent it getting overwritten.

Steve

[tomashk]

Quick update. Because I wanted to test the release candidate for 22.05 I took the old device, that had problems described above. After updating to the RC version it still existed. But I used suggested autoboot_delay and it fixed this timing problem. For me 10 seconds were enough to make it work.

[stephenw10]

Anyone still hitting this who can test there is a patch now available here:
https://redmine.pfsense.org/issues/13671#note-4
We need feedback there to confirm it.

Steve