Always Wan-ip but gateway is 100% packet loss

AcidSleeper

A idea from a swedish forum.

What if there´s a problem when the link-speed is to be negotiated between MC and pfSense?

MC (Inteno) maximum 1GbE and PFsense 2.5GbE

Should I set the Speed and Duplex in Inferface -> Wan?

jrey

Possible, what speed is it showing?

Should I set the Speed and Duplex in Inferface -> Wan?
you can certainly force it to match, if it is not auto detecting the correct value.

this sequence seems "interesting"

Dec 5 09:03:33	dhclient	77794	igc0 link state up -> down
Dec 5 09:03:34	dhclient	77794	connection closed
Dec 5 09:03:34	dhclient	77794	exiting.
Dec 5 09:06:08	dhclient	71781	PREINIT
Dec 5 09:06:08	dhclient	70811	DHCPREQUEST on igc0 to 255.255.255.255 port 67
Dec 5 09:06:09	dhclient	70811	DHCPACK from 192.121.XXX.2
Dec 5 09:06:09	dhclient	72733	REBOOT
Dec 5 09:06:09	dhclient	74082	Starting add_new_address()
Dec 5 09:06:09	dhclient	74466	ifconfig igc0 inet 192.121.XXX.50 netmask 255.255.255.128 broadcast 192.121.XXX.127
Dec 5 09:06:09	dhclient	75182	New IP Address (igc0): 192.121.XXX.50
Dec 5 09:06:09	dhclient	75852	New Subnet Mask (igc0): 255.255.255.128
Dec 5 09:06:09	dhclient	76620	New Broadcast Address (igc0): 192.121.XXX.127
Dec 5 09:06:09	dhclient	77335	New Routers (igc0): 192.121.XXX.1
Dec 5 09:06:09	dhclient	78369	Adding new routes to interface: igc0
Dec 5 09:06:09	dhclient	79802	/sbin/route add -host 192.121.XXX.1 -iface igc0
Dec 5 09:06:09	dhclient	80974	/sbin/route add default 192.121.XXX.1
Dec 5 09:06:09	dhclient	82065	Creating resolv.conf
Dec 5 09:06:09	dhclient	70811	bound to 192.121.XXX.50 -- renewal in 43170 seconds.
Dec 5 09:09:31	dhclient	2416	dhclient already running, pid: 77427.
Dec 5 09:09:31	dhclient	2416	exiting.
Dec 5 09:09:31	dhclient	2775	dhclient already running, pid: 77427.
Dec 5 09:09:31	dhclient	2775	exiting.
Dec 5 09:09:34	dhclient	59957	PREINIT
Dec 5 09:09:34	dhclient	78067	DHCPREQUEST on igc0 to 255.255.255.255 port 67
Dec 5 09:09:34	dhclient	78067	DHCPACK from 192.121.XXX.2
Dec 5 09:09:34	dhclient	61008	REBOOT
Dec 5 09:09:34	dhclient	62192	Starting add_new_address()
Dec 5 09:09:34	dhclient	63168	ifconfig igc0 inet 192.121.XXX.50 netmask 255.255.255.128 broadcast 192.121.XXX.127
Dec 5 09:09:34	dhclient	64280	New IP Address (igc0): 192.121.XXX.50
Dec 5 09:09:34	dhclient	65065	New Subnet Mask (igc0): 255.255.255.128
Dec 5 09:09:34	dhclient	65381	New Broadcast Address (igc0): 192.121.XXX.127
Dec 5 09:09:34	dhclient	66014	New Routers (igc0): 192.121.XXX.1
Dec 5 09:09:34	dhclient	66921	Adding new routes to interface: igc0
Dec 5 09:09:34	dhclient	68062	Creating resolv.conf
Dec 5 09:09:34	dhclient	78067	bound to 192.121.XXX.50 -- renewal in 43170 seconds.

Still have to piece together the other 3 log files to see the entire sequence of events, specifically

Dec 5 09:06:09	dhclient	77335	New Routers (igc0): 192.121.XXX.1
Dec 5 09:06:09	dhclient	78369	Adding new routes to interface: igc0
Dec 5 09:06:09	dhclient	79802	/sbin/route add -host 192.121.XXX.1 -iface igc0
Dec 5 09:06:09	dhclient	80974	/sbin/route add default 192.121.XXX.1
Dec 5 09:06:09	dhclient	82065	Creating resolv.conf

Dec 5 09:09:30	kernel		Uptime: 12m25s
Dec 5 09:09:30	kernel		---<<BOOT>>---

compared to the one 3 minutes later

Dec 5 09:09:34	dhclient	66014	New Routers (igc0): 192.121.XXX.1
Dec 5 09:09:34	dhclient	66921	Adding new routes to interface: igc0
Dec 5 09:09:34	dhclient	68062	Creating resolv.conf
Dec 5 09:09:34	dhclient	78067	bound to 192.121.XXX.50 -- renewal in 43170 seconds.

would be interesting to know what is in /etc/resolv.conf under both conditions
"Online working" and "when not"

also the contents of /var/db/dhclient.leases.igc0 under both conditions.

"Gateway Action" to 'disable gateway monitoring action'

this will turn off the monitoring, and you could try it, however you have cases where it is working. You are not pinging the ISP, you are pinging a remote. ISP shouldn't be blocking that. You can find out, when you are online and it is working... do a trace route to 1.1.1.1

you also have a couple of entries that imply the dhclient is not running

DHCP Client not running on wan (igc0)

start tracking if you can, when you get the IP assigned from their .2 vs their .3
ie when you get the address assigned from .2 does it work and from .3 not work or visa versa.

jrey

@AcidSleeper

Actually after another cup of coffee and more of your log files, it occurs to me that you might want to add one of the DHCP servers responding in

Interfaces -> WAN

put one of the addresses in here (not both) either the 192.121.xxx.2 or .3
it just seems odd that is such a small space /25 they would have 2 servers handing out addresses, unless (read my IM) the XXX is in different segment, which would also be an ISP why question? From the logs I've seen over the past day or so, you seem to get IP from .2 most often, so start by rejecting the .3 in this field,

Let's see if that changes anything.

Sorjal

@jrey Just to clarify, I was mentioning the option to disable the monitoring actions, not the monitoring itself.

For example, here's a brief bit from my own gateway log...

If the monitoring actions were turned on and say killing states, I'd never connect to anything. Leaving monitoring on let's me send logs to my isp for them to ignore and say everything is fine. :)

jrey

@Sorjal said in Always Wan-ip but gateway is 100% packet loss:

I was mentioning the option to disable the monitoring actions

right you are, sorry --- I misread the option you suggested "actions" ---

@AcidSleeper
do this after the "Reject leases from" test but not at the same time.
the fact that there are a couple of times in your various logs, where it appears the gateway is offered, but doesn't appear to be set, would never let it run in the first place.

AcidSleeper

@jrey said in Always Wan-ip but gateway is 100% packet loss:

@AcidSleeper

Actually after another cup of coffee and more of your log files, it occurs to me that you might want to add one of the DHCP servers responding in

Interfaces -> WAN

put one of the addresses in here (not both) either the 192.121.xxx.2 or .3
it just seems odd that is such a small space /25 they would have 2 servers handing out addresses, unless (read my IM) the XXX is in different segment, which would also be an ISP why question? From the logs I've seen over the past day or so, you seem to get IP from .2 most often, so start by rejecting the .3 in this field,

Let's see if that changes anything.

Nothing changed in gateway. It turned offline, so same problem.

Logs:
231206-pfsense-general-log.txt
231206-pfsense-gateway-log.txt
231206-pfsense-resolver-log.txt
231206-pfsense-dhcp-log.txt

AcidSleeper

@Sorjal said in Always Wan-ip but gateway is 100% packet loss:

@jrey Just to clarify, I was mentioning the option to disable the monitoring actions, not the monitoring itself.

If the monitoring actions were turned on and say killing states, I'd never connect to anything. Leaving monitoring on let's me send logs to my isp for them to ignore and say everything is fine. :)

Sorry but it didnt work. I got this:

jrey

@AcidSleeper

Maybe, but the DHCP client logging is certainly vastly different from previous samples.

let's take smaller steps.

can we set the exclude for the DHCP to the .3. -- after the change applies (give it a couple of minutes)

shutdown pfSense, pause
restart the MC, pause
then restart pf. pause

then login.

once it is up and running (online or not) let's have a look at DHCP log.
for this single test
and also the contents of /etc/resolv.conf
and
/var/db/dhclient.leases.igc0

also Dec 6 16:13:25 php-fpm 400 /rc.linkup: The command '/sbin/dhclient -c /var/etc/dhclient_wan.conf -p /var/run/dhclient.igc0.pid igc0 > /tmp/igc0_output 2> /tmp/igc0_error_output'

also the file in bold (if it exists)

Thanks

AcidSleeper

@jrey said in Always Wan-ip but gateway is 100% packet loss:

@AcidSleeper

Maybe, but the DHCP client logging is certainly vastly different from previous samples.

let's take smaller steps.

can we set the exclude for the DHCP to the .3. -- after the change applies (give it a couple of minutes)

shutdown pfSense, pause
restart the MC, pause
then restart pf. pause

then login.

once it is up and running (online or not) let's have a look at DHCP log.
for this single test
and also the contents of /etc/resolv.conf
and
/var/db/dhclient.leases.igc0

also Dec 6 16:13:25 php-fpm 400 /rc.linkup: The command '/sbin/dhclient -c /var/etc/dhclient_wan.conf -p /var/run/dhclient.igc0.pid igc0 > /tmp/igc0_output 2> /tmp/igc0_error_output'

also the file in bold (if it exists)

Did it as your instructions, result:
Online (working 100%)

LOGS:
231207-pfsense-dhcp-log.txt
231207-pfsense-etc.resolv.conf-log.txt
231207-pfsense-vi-dhclient.lease.txt
Nothing inside /tmp/igc0_error_output

After restart after a hour or so Im offline, nothing works.

LOGS:
231207-pfsense-dhcp-log-2.txt
231207-pfsense-vi-dhclient.lease-2.txt
Etc/resolv.conf is unchanged.
Nothing inside /tmp/igc0_error_output

Work continues.

AcidSleeper

If someone is wondering why there is no new posts its because @jrey is helping me directly in chat.

When a solution is found we will post it.

jrey

@AcidSleeper

After several messages, and dealing with an unhelpful ISP, time away etc, this now seems resolved.

(Thanks to your neighbour, for letting you go next door and try it on a different ISP, although we didn't use the data collected there, the result spoke volumes)

Briefly, to recap, the issue was that pfSense was not obtaining an IP/gateway on the WAN, unless the "MC" was rebooted.

The ISP uses two DHCP relays in a /25 scope, and provides a third DHCP server as "next-server" in response.
We'll call them DHCP.2, DHCP.3 with the relays in the same /25 scope and DHCP.244 (the upstream in a different segment)

once pfSense was able to obtain an IP (only after rebooting "MC" first) it would renew on schedule as expected. So the only issue was rebooting the pfSense without rebooting the "MC" first.

The solution that is now working was to:
Reject leases from (DHCP.3) address
change the Presets (timing) to "FreeBSD default"
add Send options: dhcp-server-identifier (DHCP.2)

The WAN port now obtains a valid IP/gateway combination within the /25 scope.
Works when there is:
a reboot of pfSense only, or;
a reboot of both "MC" and pfSense;
and renews the lease as scheduled (day 2 now, since last reboot?)

Happy New Year.

AcidSleeper

@jrey

Thanks so much for the help! Couldnt have done it without you! Thanks yet again!