Always Wan-ip but gateway is 100% packet loss
-
@Sorjal said in Always Wan-ip but gateway is 100% packet loss:
First in each Gateway (I have 1 IPv4 and 1 IPv6) I set up a different IP in each in the area "Monitor IP". For me I used two of Cloudflare's DNS addresses as they're pretty much always up (1.1.1.1 and 2606:4700:4700::1111). This makes the gateway monitoring a little more accurate as sometimes say the icmp request will make it through the ISP's gateway to the monitor IP and it will return those statistics. Still, as above, the gateways can be so bad that even requests to different addresses never make it there and back in time.
I only have IPv4 and I have added 8.8.8.8 to gateway monitoring.
More importantly I checked off the box next to "Gateway Action" to 'disable gateway monitoring action' so that if their stops responding to echo requests, the gateway is still considered up and pfsense doesn't take any actions such as Kill states on gateway failure.
I havent this box checked but I try anything. Will wait to do that until my answear to @jrey is looked at, so I dont mess up anything.
You could instead also try changing that option to 'Do no kill states on gateway failure'.
My settings:
-
A idea from a swedish forum.
What if there´s a problem when the link-speed is to be negotiated between MC and pfSense?
MC (Inteno) maximum 1GbE and PFsense 2.5GbE
Should I set the Speed and Duplex in Inferface -> Wan?
-
Possible, what speed is it showing?
Should I set the Speed and Duplex in Inferface -> Wan?
you can certainly force it to match, if it is not auto detecting the correct value.this sequence seems "interesting"
Dec 5 09:03:33 dhclient 77794 igc0 link state up -> down Dec 5 09:03:34 dhclient 77794 connection closed Dec 5 09:03:34 dhclient 77794 exiting. Dec 5 09:06:08 dhclient 71781 PREINIT Dec 5 09:06:08 dhclient 70811 DHCPREQUEST on igc0 to 255.255.255.255 port 67 Dec 5 09:06:09 dhclient 70811 DHCPACK from 192.121.XXX.2 Dec 5 09:06:09 dhclient 72733 REBOOT Dec 5 09:06:09 dhclient 74082 Starting add_new_address() Dec 5 09:06:09 dhclient 74466 ifconfig igc0 inet 192.121.XXX.50 netmask 255.255.255.128 broadcast 192.121.XXX.127 Dec 5 09:06:09 dhclient 75182 New IP Address (igc0): 192.121.XXX.50 Dec 5 09:06:09 dhclient 75852 New Subnet Mask (igc0): 255.255.255.128 Dec 5 09:06:09 dhclient 76620 New Broadcast Address (igc0): 192.121.XXX.127 Dec 5 09:06:09 dhclient 77335 New Routers (igc0): 192.121.XXX.1 Dec 5 09:06:09 dhclient 78369 Adding new routes to interface: igc0 Dec 5 09:06:09 dhclient 79802 /sbin/route add -host 192.121.XXX.1 -iface igc0 Dec 5 09:06:09 dhclient 80974 /sbin/route add default 192.121.XXX.1 Dec 5 09:06:09 dhclient 82065 Creating resolv.conf Dec 5 09:06:09 dhclient 70811 bound to 192.121.XXX.50 -- renewal in 43170 seconds. Dec 5 09:09:31 dhclient 2416 dhclient already running, pid: 77427. Dec 5 09:09:31 dhclient 2416 exiting. Dec 5 09:09:31 dhclient 2775 dhclient already running, pid: 77427. Dec 5 09:09:31 dhclient 2775 exiting. Dec 5 09:09:34 dhclient 59957 PREINIT Dec 5 09:09:34 dhclient 78067 DHCPREQUEST on igc0 to 255.255.255.255 port 67 Dec 5 09:09:34 dhclient 78067 DHCPACK from 192.121.XXX.2 Dec 5 09:09:34 dhclient 61008 REBOOT Dec 5 09:09:34 dhclient 62192 Starting add_new_address() Dec 5 09:09:34 dhclient 63168 ifconfig igc0 inet 192.121.XXX.50 netmask 255.255.255.128 broadcast 192.121.XXX.127 Dec 5 09:09:34 dhclient 64280 New IP Address (igc0): 192.121.XXX.50 Dec 5 09:09:34 dhclient 65065 New Subnet Mask (igc0): 255.255.255.128 Dec 5 09:09:34 dhclient 65381 New Broadcast Address (igc0): 192.121.XXX.127 Dec 5 09:09:34 dhclient 66014 New Routers (igc0): 192.121.XXX.1 Dec 5 09:09:34 dhclient 66921 Adding new routes to interface: igc0 Dec 5 09:09:34 dhclient 68062 Creating resolv.conf Dec 5 09:09:34 dhclient 78067 bound to 192.121.XXX.50 -- renewal in 43170 seconds.Still have to piece together the other 3 log files to see the entire sequence of events, specifically
Dec 5 09:06:09 dhclient 77335 New Routers (igc0): 192.121.XXX.1 Dec 5 09:06:09 dhclient 78369 Adding new routes to interface: igc0 Dec 5 09:06:09 dhclient 79802 /sbin/route add -host 192.121.XXX.1 -iface igc0 Dec 5 09:06:09 dhclient 80974 /sbin/route add default 192.121.XXX.1 Dec 5 09:06:09 dhclient 82065 Creating resolv.confDec 5 09:09:30 kernel Uptime: 12m25s Dec 5 09:09:30 kernel ---<<BOOT>>---compared to the one 3 minutes later
Dec 5 09:09:34 dhclient 66014 New Routers (igc0): 192.121.XXX.1 Dec 5 09:09:34 dhclient 66921 Adding new routes to interface: igc0 Dec 5 09:09:34 dhclient 68062 Creating resolv.conf Dec 5 09:09:34 dhclient 78067 bound to 192.121.XXX.50 -- renewal in 43170 seconds.would be interesting to know what is in /etc/resolv.conf under both conditions
"Online working" and "when not"also the contents of /var/db/dhclient.leases.igc0 under both conditions.
"Gateway Action" to 'disable gateway monitoring action'
this will turn off the monitoring, and you could try it, however you have cases where it is working. You are not pinging the ISP, you are pinging a remote. ISP shouldn't be blocking that. You can find out, when you are online and it is working... do a trace route to 1.1.1.1
you also have a couple of entries that imply the dhclient is not running
DHCP Client not running on wan (igc0)start tracking if you can, when you get the IP assigned from their .2 vs their .3
ie when you get the address assigned from .2 does it work and from .3 not work or visa versa. -
Actually after another cup of coffee and more of your log files, it occurs to me that you might want to add one of the DHCP servers responding in
Interfaces -> WAN
put one of the addresses in here (not both) either the 192.121.xxx.2 or .3
it just seems odd that is such a small space /25 they would have 2 servers handing out addresses, unless (read my IM) the XXX is in different segment, which would also be an ISP why question? From the logs I've seen over the past day or so, you seem to get IP from .2 most often, so start by rejecting the .3 in this field,Let's see if that changes anything.
-
@jrey Just to clarify, I was mentioning the option to disable the monitoring actions, not the monitoring itself.
For example, here's a brief bit from my own gateway log...
If the monitoring actions were turned on and say killing states, I'd never connect to anything. Leaving monitoring on let's me send logs to my isp for them to ignore and say everything is fine. :)
-
@Sorjal said in Always Wan-ip but gateway is 100% packet loss:
I was mentioning the option to disable the monitoring actions
right you are, sorry --- I misread the option you suggested "actions" ---
@AcidSleeper
do this after the "Reject leases from" test but not at the same time.
the fact that there are a couple of times in your various logs, where it appears the gateway is offered, but doesn't appear to be set, would never let it run in the first place. -
@jrey said in Always Wan-ip but gateway is 100% packet loss:
Actually after another cup of coffee and more of your log files, it occurs to me that you might want to add one of the DHCP servers responding in
Interfaces -> WAN
put one of the addresses in here (not both) either the 192.121.xxx.2 or .3
it just seems odd that is such a small space /25 they would have 2 servers handing out addresses, unless (read my IM) the XXX is in different segment, which would also be an ISP why question? From the logs I've seen over the past day or so, you seem to get IP from .2 most often, so start by rejecting the .3 in this field,Let's see if that changes anything.
Nothing changed in gateway. It turned offline, so same problem.
Logs:
231206-pfsense-general-log.txt
231206-pfsense-gateway-log.txt
231206-pfsense-resolver-log.txt
231206-pfsense-dhcp-log.txt -
@Sorjal said in Always Wan-ip but gateway is 100% packet loss:
@jrey Just to clarify, I was mentioning the option to disable the monitoring actions, not the monitoring itself.
If the monitoring actions were turned on and say killing states, I'd never connect to anything. Leaving monitoring on let's me send logs to my isp for them to ignore and say everything is fine. :)
Sorry but it didnt work. I got this:
-
Maybe, but the DHCP client logging is certainly vastly different from previous samples.
let's take smaller steps.
can we set the exclude for the DHCP to the .3. -- after the change applies (give it a couple of minutes)
shutdown pfSense, pause
restart the MC, pause
then restart pf. pausethen login.
once it is up and running (online or not) let's have a look at DHCP log.
for this single test
and also the contents of /etc/resolv.conf
and
/var/db/dhclient.leases.igc0also Dec 6 16:13:25 php-fpm 400 /rc.linkup: The command '/sbin/dhclient -c /var/etc/dhclient_wan.conf -p /var/run/dhclient.igc0.pid igc0 > /tmp/igc0_output 2> /tmp/igc0_error_output'
also the file in bold (if it exists)
Thanks
-
@jrey said in Always Wan-ip but gateway is 100% packet loss:
Maybe, but the DHCP client logging is certainly vastly different from previous samples.
let's take smaller steps.
can we set the exclude for the DHCP to the .3. -- after the change applies (give it a couple of minutes)
shutdown pfSense, pause
restart the MC, pause
then restart pf. pausethen login.
once it is up and running (online or not) let's have a look at DHCP log.
for this single test
and also the contents of /etc/resolv.conf
and
/var/db/dhclient.leases.igc0also Dec 6 16:13:25 php-fpm 400 /rc.linkup: The command '/sbin/dhclient -c /var/etc/dhclient_wan.conf -p /var/run/dhclient.igc0.pid igc0 > /tmp/igc0_output 2> /tmp/igc0_error_output'
also the file in bold (if it exists)
Did it as your instructions, result:
Online (working 100%)LOGS:
231207-pfsense-dhcp-log.txt
231207-pfsense-etc.resolv.conf-log.txt
231207-pfsense-vi-dhclient.lease.txt
Nothing inside /tmp/igc0_error_outputAfter restart after a hour or so Im offline, nothing works.
LOGS:
231207-pfsense-dhcp-log-2.txt
231207-pfsense-vi-dhclient.lease-2.txt
Etc/resolv.conf is unchanged.
Nothing inside /tmp/igc0_error_outputWork continues.
-
If someone is wondering why there is no new posts its because @jrey is helping me directly in chat.
When a solution is found we will post it.
-
After several messages, and dealing with an unhelpful ISP, time away etc, this now seems resolved.
(Thanks to your neighbour, for letting you go next door and try it on a different ISP, although we didn't use the data collected there, the result spoke volumes)
Briefly, to recap, the issue was that pfSense was not obtaining an IP/gateway on the WAN, unless the "MC" was rebooted.
The ISP uses two DHCP relays in a /25 scope, and provides a third DHCP server as "next-server" in response.
We'll call them DHCP.2, DHCP.3 with the relays in the same /25 scope and DHCP.244 (the upstream in a different segment)once pfSense was able to obtain an IP (only after rebooting "MC" first) it would renew on schedule as expected. So the only issue was rebooting the pfSense without rebooting the "MC" first.
The solution that is now working was to:
Reject leases from (DHCP.3) address
change the Presets (timing) to "FreeBSD default"
add Send options: dhcp-server-identifier (DHCP.2)The WAN port now obtains a valid IP/gateway combination within the /25 scope.
Works when there is:
a reboot of pfSense only, or;
a reboot of both "MC" and pfSense;
and renews the lease as scheduled (day 2 now, since last reboot?)Happy New Year.
-
Thanks so much for the help! Couldnt have done it without you! Thanks yet again!