Intermittent "no route to host" on my LAN-port

tsmalmbe

ping: sendto: No route to host
ping: sendto: No route to host
ping: sendto: No route to host
64 bytes from 192.168.1.9: icmp_seq=3 ttl=64 time=1.511 ms
64 bytes from 192.168.1.9: icmp_seq=4 ttl=64 time=1.424 ms
64 bytes from 192.168.1.9: icmp_seq=5 ttl=64 time=0.856 ms
64 bytes from 192.168.1.9: icmp_seq=6 ttl=64 time=0.881 ms
64 bytes from 192.168.1.9: icmp_seq=7 ttl=64 time=0.800 ms
ping: sendto: No route to host
ping: sendto: No route to host
ping: sendto: No route to host
ping: sendto: No route to host
ping: sendto: No route to host

This is from the pfSense shell, trying to ping my LAN. PfSense is 192.168.1.1. I have two VLAN's configured on my LAN-port. Hardware is a Watchguard X550. A reboot always helps. I have not seen this issue on for instance my WAN port. Both my WAn and my LAN ports are connected to a Procurve switch. I have a couple of OpenVPN servers configured. I have no special NAT-rules (just the defaults) and no manual route configurations. 192.168.1.9 is wired to the Procurve switch directly.

Other devices within the network and connected to the Procurve can always ping eachother without issues. It's just pfsense which is a problem.

This happens every few days or so, sometimes a few times per day even. Sometimes I also lose my WAN-connectivity in a similar way.

Can this be a hardware thing? I have had the firewall operational for a few years and these problems have now been ongoing for 6 months.

I've lookead at the logs in /var/log and cannot see anything that could be related.

Where should I even start?

tsmalmbe

Routing tables

Internet:
Destination Gateway Flags Netif Expire
default atm-gw-178.nblnetw UGS sk0
10.10.1.0 link#4 U sk3
10.10.1.1 link#4 UHS lo0
10.99.0.0 10.100.100.2 UGS ovpns5
10.100.100.1 link#12 UHS lo0
10.100.100.2 link#12 UH ovpns5
10.100.101.1 link#13 UHS lo0
10.100.101.2 link#13 UH ovpns8
10.200.200.0 10.200.200.1 UGS ovpns1
10.200.200.1 link#11 UHS lo0
10.200.200.2 link#11 UH ovpns1
10.200.210.0 10.100.100.2 UGS ovpns5
nblzone-gw.lau.hel atm-gw-178.nblnetw UGHS sk0
localhost link#8 UH lo0
192.168.1.0 link#2 U sk1
firewall link#2 UHS lo0
192.168.1.3 link#2 UHS lo0
wiki link#2 UHS lo0
192.168.2.21/32 link#2 U sk1
192.168.10.0 link#9 U sk1_vlan
192.168.10.1 link#9 UHS lo0
192.168.20.0 link#10 U sk1_vlan
192.168.20.1 link#10 UHS lo0
192.168.69.0 10.100.101.2 UGS ovpns8
192.168.100.0 link#3 U sk2
192.168.100.1 link#3 UHS lo0
217.30.178.0 link#1 U sk0
xdsl-178-237.nblne link#1 UHS lo0

tsmalmbe

Is there something I could have left "empty" or as default regarding gateways or routing when I have added OpenVPN's and VLAN's?

tsmalmbe

PING 8.8.8.8 (8.8.8.8): 56 data bytes
ping: sendto: No route to host

traceroute to 8.8.8.8 (8.8.8.8), 64 hops max, 40 byte packets
1 nblzone-gw.lau.hel.fi.nblnet.com (83.145.193.133) 15.589 ms 15.845 ms 15.937 ms
2 pertr1-te0-3-0-5.lau.hel.fi.nblnet.com (83.145.255.246) 15.966 ms 16.721 ms 17.708 ms
3 rtr3-te7-4.lau.hel.fi.nblnet.com (188.117.15.254) 48.247 ms 16.310 ms 15.609 ms
4 rtr1-po3.lau.hel.fi.nblnet.com (83.145.254.70) 16.154 ms 16.124 ms
rtr1-po4.pas.hel.fi.nblnet.com (83.145.255.109) 70.188 ms
5 hls-b3-link.telia.net (213.248.84.237) 16.024 ms
hls-b2-link.telia.net (80.239.132.5) 15.918 ms 16.022 ms
6 s-bb4-link.telia.net (62.115.123.30) 22.409 ms
s-bb3-link.telia.net (62.115.134.0) 23.252 ms
s-bb3-link.telia.net (62.115.113.104) 24.018 ms
7 s-b5-link.telia.net (213.155.133.17) 23.561 ms
s-b5-link.telia.net (80.91.249.219) 22.708 ms 22.793 ms
8 google-ic-314684-s-b5.c.telia.net (62.115.61.30) 23.393 ms 23.467 ms 22.616 ms
9 74.125.37.237 (74.125.37.237) 23.490 ms 23.004 ms
216.239.54.213 (216.239.54.213) 23.726 ms
10 72.14.236.85 (72.14.236.85) 22.901 ms
209.85.245.63 (209.85.245.63) 23.376 ms
216.239.48.1 (216.239.48.1) 23.174 ms
11 google-public-dns-a.google.com (8.8.8.8) 23.060 ms 23.278 ms 22.818 ms

traceroute to 192.168.1.9 (192.168.1.9), 64 hops max, 40 byte packets
traceroute: sendto: No route to host
1 traceroute: wrote 192.168.1.9 40 chars, ret=-1
*traceroute: sendto: No route to host
traceroute: wrote 192.168.1.9 40 chars, ret=-1
*traceroute: sendto: No route to host
traceroute: wrote 192.168.1.9 40 chars, ret=-1

PING 192.168.1.9 (192.168.1.9): 56 data bytes
ping: sendto: No route to host
ping: sendto: No route to host

tsmalmbe

After reboot

PING 192.168.1.9 (192.168.1.9): 56 data bytes
64 bytes from 192.168.1.9: icmp_seq=0 ttl=64 time=0.942 ms
64 bytes from 192.168.1.9: icmp_seq=1 ttl=64 time=0.789 ms
64 bytes from 192.168.1.9: icmp_seq=2 ttl=64 time=0.752 ms
^C

traceroute to 192.168.1.9 (192.168.1.9), 64 hops max, 40 byte packets
1 wlan (192.168.1.9) 0.934 ms 0.897 ms 0.888 ms

tsmalmbe

netstat -r after reboot

Routing tables

Internet:
Destination Gateway Flags Netif Expire
default atm-gw-178.nblnetw UGS sk0
10.10.1.0 link#4 U sk3
10.10.1.1 link#4 UHS lo0
10.99.0.0 10.100.100.2 UGS ovpns5
10.100.100.1 link#12 UHS lo0
10.100.100.2 link#12 UH ovpns5
10.100.101.1 link#13 UHS lo0
10.100.101.2 link#13 UH ovpns8
10.200.200.0 10.200.200.1 UGS ovpns1
10.200.200.1 link#11 UHS lo0
10.200.200.2 link#11 UH ovpns1
10.200.210.0 10.100.100.2 UGS ovpns5
nblzone-gw.lau.hel atm-gw-178.nblnetw UGHS sk0
localhost link#8 UH lo0
192.168.1.0 link#2 U sk1
firewall link#2 UHS lo0
192.168.1.3 link#2 UHS lo0
wiki link#2 UHS lo0
192.168.2.21/32 link#2 U sk1
192.168.10.0 link#9 U sk1_vlan
192.168.10.1 link#9 UHS lo0
192.168.20.0 link#10 U sk1_vlan
192.168.20.1 link#10 UHS lo0
192.168.69.0 10.100.101.2 UGS ovpns8
192.168.100.0 link#3 U sk2
192.168.100.1 link#3 UHS lo0
217.30.178.0 link#1 U sk0
xdsl-178-237.nblne link#1 UHS lo0

bingo600

If you are executing the ping from the pfsense shell , and you get a "no route to host" on a directly connected interface (as you have).
Then the obvious reason is that the vlan interface gets "disconnected" , what does the procurve say in the logs ?.

Do you have any "link renegotiations / drops" , on the link to the pfsense ?

/Bingo

tsmalmbe

I will check this. My lan port is connected "untagged" while I am also running two other VLAN's on the same port as "tagged".

This is how the procurve log looks like. I just had to reboot pfsense a minute ago, but no relevant logs prior to that. Port 1 is my LAN port

05/14/17 13:47:17 SNTP: updated time by -4 seconds
05/15/17 08:43:51 ports: port 34 is now off-line
05/15/17 08:43:51 ports: port 1 is now off-line
05/15/17 08:43:51 ports: port 37 is now off-line
05/15/17 08:43:51 ports: port 41 is now off-line
05/15/17 08:43:54 ports: port 1 is Blocked by LACP
05/15/17 08:43:54 ports: port 34 is Blocked by LACP
05/15/17 08:43:54 ports: port 37 is Blocked by LACP
05/15/17 08:43:55 ports: port 41 is Blocked by LACP
05/15/17 08:43:57 ports: port 1 is now on-line

tsmalmbe

For what it is worth, I tested to ping my lan as described earlier: "no route to host". Then I pinged a VLAN which is on the same port, success and ping goes through without any problems.

tsmalmbe

I disabled all hardware offloading (had two out of three disabled previously). Just to see if it makes a difference. I have rebooted twice today already.

bingo600

@tsmalmbe:

05/14/17 13:47:17 SNTP: updated time by -4 seconds
05/15/17 08:43:51 ports: port 34 is now off-line
05/15/17 08:43:51 ports: port 1 is now off-line
05/15/17 08:43:51 ports: port 37 is now off-line
05/15/17 08:43:51 ports: port 41 is now off-line
05/15/17 08:43:54 ports: port 1 is Blocked by LACP
05/15/17 08:43:54 ports: port 34 is Blocked by LACP
05/15/17 08:43:54 ports: port 37 is Blocked by LACP
05/15/17 08:43:55 ports: port 41 is Blocked by LACP
05/15/17 08:43:57 ports: port 1 is now on-line

Next time login to the procurve and snag the log before rebooting the pfsense.

I'm a pfsense newbie (but know networking) , and would not expect it to participate in STP (spanning tree protocol) as it's a L3 firewall.
But then again … It has bridge mode ... and should be STP capable (at least i that mode)

So you have 2 tagged vlans and one untagged vlan on that interface ?

Is it only the untagged vlan that have this problem , or does the same problem occur (at the same time) on the other tagged vlans ?

/Bingo

tsmalmbe

Wierd as it seems, it is only the untagged VLAN that starts to behave badly, the tagged VLAN's are completely fine.

tsmalmbe

Surely there has to be a loglevel that can reveal what the problem is?

ping: sendto: No route to host
ping: sendto: No route to host
ping: sendto: No route to host
ping: sendto: No route to host
ping: sendto: No route to host
ping: sendto: No route to host
64 bytes from 192.168.1.9: icmp_seq=168 ttl=64 time=1.496 ms
64 bytes from 192.168.1.9: icmp_seq=169 ttl=64 time=0.721 ms
64 bytes from 192.168.1.9: icmp_seq=170 ttl=64 time=0.742 ms
64 bytes from 192.168.1.9: icmp_seq=171 ttl=64 time=0.915 ms
64 bytes from 192.168.1.9: icmp_seq=172 ttl=64 time=0.784 ms
64 bytes from 192.168.1.9: icmp_seq=173 ttl=64 time=0.510 ms
64 bytes from 192.168.1.9: icmp_seq=174 ttl=64 time=1.138 ms
64 bytes from 192.168.1.9: icmp_seq=175 ttl=64 time=0.760 ms
64 bytes from 192.168.1.9: icmp_seq=176 ttl=64 time=0.586 ms
64 bytes from 192.168.1.9: icmp_seq=177 ttl=64 time=0.664 ms
64 bytes from 192.168.1.9: icmp_seq=178 ttl=64 time=0.724 ms
ping: sendto: No route to host
ping: sendto: No route to host
ping: sendto: No route to host
ping: sendto: No route to host
ping: sendto: No route to host
ping: sendto: No route to host
64 bytes from 192.168.1.9: icmp_seq=185 ttl=64 time=1.198 ms
64 bytes from 192.168.1.9: icmp_seq=186 ttl=64 time=1.222 ms
64 bytes from 192.168.1.9: icmp_seq=187 ttl=64 time=0.845 ms
64 bytes from 192.168.1.9: icmp_seq=188 ttl=64 time=0.760 ms
64 bytes from 192.168.1.9: icmp_seq=189 ttl=64 time=0.890 ms
64 bytes from 192.168.1.9: icmp_seq=190 ttl=64 time=0.815 ms
ping: sendto: No route to host
ping: sendto: No route to host
ping: sendto: No route to host
ping: sendto: No route to host
ping: sendto: No route to host
ping: sendto: No route to host
ping: sendto: No route to host
ping: sendto: No route to host
ping: sendto: No route to host
ping: sendto: No route to host
ping: sendto: No route to host
ping: sendto: No route to host
ping: sendto: No route to host
ping: sendto: No route to host
ping: sendto: No route to host
64 bytes from 192.168.1.9: icmp_seq=206 ttl=64 time=1.395 ms
64 bytes from 192.168.1.9: icmp_seq=207 ttl=64 time=1.413 ms
64 bytes from 192.168.1.9: icmp_seq=208 ttl=64 time=0.548 ms
64 bytes from 192.168.1.9: icmp_seq=209 ttl=64 time=0.767 ms
64 bytes from 192.168.1.9: icmp_seq=210 ttl=64 time=0.872 ms
64 bytes from 192.168.1.9: icmp_seq=211 ttl=64 time=0.613 ms
64 bytes from 192.168.1.9: icmp_seq=212 ttl=64 time=0.740 ms
64 bytes from 192.168.1.9: icmp_seq=213 ttl=64 time=0.770 ms
64 bytes from 192.168.1.9: icmp_seq=214 ttl=64 time=0.585 ms
64 bytes from 192.168.1.9: icmp_seq=215 ttl=64 time=0.809 ms
64 bytes from 192.168.1.9: icmp_seq=216 ttl=64 time=0.738 ms
64 bytes from 192.168.1.9: icmp_seq=217 ttl=64 time=0.865 ms
64 bytes from 192.168.1.9: icmp_seq=218 ttl=64 time=0.780 ms
64 bytes from 192.168.1.9: icmp_seq=219 ttl=64 time=0.705 ms
ping: sendto: No route to host
ping: sendto: No route to host
ping: sendto: No route to host
ping: sendto: No route to host
ping: sendto: No route to host
ping: sendto: No route to host
ping: sendto: No route to host
ping: sendto: No route to host
64 bytes from 192.168.1.9: icmp_seq=228 ttl=64 time=1.111 ms

bingo600

Can you ping the pfSense ip address , on the affected untagged vlan, while the problem is there??

/Bingo

tsmalmbe

@bingo600:

Can you ping the pfSense ip address , on the affected untagged vlan, while the problem is there??

A lan wks -> pfSense ip FAIL
pfSense -> a lan wks FAIL
pfSense -> a tagged lan server on the same port as the untagged lan SUCCESS
A lan wks -> a tagged lan server on the same port as the untagged lan SUCCESS

Like that.

tsmalmbe

I now moved my LAN from untagged to tagged (I tagged it on pfSense and on the switch). It will be a bitch if my switch dies to recover, but theres always something you can do over the terminal.

Anyway, let's see if this solves the issue. Then we know that VLANs and raw LAN's and pfSense and Watchguard and Procurve donät work together well.

Derelict

Some people say don't mix tagged and untagged traffic on an interface for a reason.

I would suspect an ARP issue there, but those intervals are awfully short for that. Could also be a simple no carrier on the ethernet interface. Have you tried another cable? Another switchport? But if it is only the default VLAN and not the tagged interfaces, that pretty much rules out layer 1.

Dealing with a tagged port is not really a bitch to deal with if you have the right tools.

![Screen Shot 2017-05-23 at 2.58.57 AM.png](/public/imported_attachments/1/Screen Shot 2017-05-23 at 2.58.57 AM.png)
![Screen Shot 2017-05-23 at 2.58.57 AM.png_thumb](/public/imported_attachments/1/Screen Shot 2017-05-23 at 2.58.57 AM.png_thumb)

tsmalmbe

ARP would've been my guess also, but then I don't really understand why it only starts after a while, and then the problem doesn't go away by itself - and the cycle is quite short as you said. But let's see.

The bitchy part is mostly if I need to remove the fw and hook up a laptop directly to the LAN, then it will require a bit of fiddling to get that going. As long as you document the VLAN's it's not such a big deal. And perhaps the correct way is to use the switch to handle the mixing and matching of VLAN's on the switch ports and then just either show all tagged or all untagged to the firewall.

tsmalmbe

It gets worse.

My two OpenVPN tunnels are now doing the same thing.
Request timed out.
Request timed out.
Reply from 10.99.10.1: bytes=32 time=48ms TTL=63
Reply from 10.99.10.1: bytes=32 time=48ms TTL=63
Reply from 10.99.10.1: bytes=32 time=48ms TTL=63
Request timed out.
Reply from 10.99.10.1: bytes=32 time=49ms TTL=63
Reply from 10.99.10.1: bytes=32 time=52ms TTL=63
Reply from 10.99.10.1: bytes=32 time=47ms TTL=63
Request timed out.
Reply from 10.99.10.1: bytes=32 time=49ms TTL=63
Reply from 10.99.10.1: bytes=32 time=48ms TTL=63
Reply from 10.99.10.1: bytes=32 time=48ms TTL=63
Reply from 10.99.10.1: bytes=32 time=48ms TTL=63
Reply from 10.99.10.1: bytes=32 time=49ms TTL=63
Request timed out.
Request timed out.
Reply from 10.99.10.1: bytes=32 time=64ms TTL=63
Reply from 10.99.10.1: bytes=32 time=48ms TTL=63
Reply from 10.99.10.1: bytes=32 time=47ms TTL=63
Reply from 10.99.10.1: bytes=32 time=48ms TTL=63
Reply from 10.99.10.1: bytes=32 time=48ms TTL=63
Reply from 10.99.10.1: bytes=32 time=49ms TTL=63
Request timed out.
Request timed out.
Reply from 10.99.10.1: bytes=32 time=149ms TTL=63
Reply from 10.99.10.1: bytes=32 time=47ms TTL=63
Reply from 10.99.10.1: bytes=32 time=49ms TTL=63
Reply from 10.99.10.1: bytes=32 time=49ms TTL=63
Reply from 10.99.10.1: bytes=32 time=49ms TTL=63

And

Reply from 192.168.69.1: bytes=32 time=126ms TTL=63
Reply from 192.168.69.1: bytes=32 time=36ms TTL=63
Request timed out.
Request timed out.
Reply from 192.168.69.1: bytes=32 time=70ms TTL=63
Reply from 192.168.69.1: bytes=32 time=110ms TTL=63
Reply from 192.168.69.1: bytes=32 time=61ms TTL=63
Request timed out.
Request timed out.
Reply from 192.168.69.1: bytes=32 time=108ms TTL=63
Reply from 192.168.69.1: bytes=32 time=68ms TTL=63
Reply from 192.168.69.1: bytes=32 time=107ms TTL=63
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Reply from 192.168.69.1: bytes=32 time=109ms TTL=63
Request timed out.
Request timed out.
Reply from 192.168.69.1: bytes=32 time=68ms TTL=63
Reply from 192.168.69.1: bytes=32 time=100ms TTL=63
Reply from 192.168.69.1: bytes=32 time=75ms TTL=63
Reply from 192.168.69.1: bytes=32 time=103ms TTL=63
Reply from 192.168.69.1: bytes=32 time=65ms TTL=63
Request timed out.
Request timed out.
Reply from 192.168.69.1: bytes=32 time=108ms TTL=63
Reply from 192.168.69.1: bytes=32 time=68ms TTL=63
Reply from 192.168.69.1: bytes=32 time=93ms TTL=63
Reply from 192.168.69.1: bytes=32 time=67ms TTL=63
Reply from 192.168.69.1: bytes=32 time=106ms TTL=63
Request timed out.

Also pingig the firewall

Reply from 192.168.1.1: bytes=32 time=1ms TTL=64
Reply from 192.168.1.1: bytes=32 time=1ms TTL=64
Request timed out.
Reply from 192.168.1.1: bytes=32 time=1ms TTL=64
Reply from 192.168.1.1: bytes=32 time=1ms TTL=64
Reply from 192.168.1.1: bytes=32 time=1ms TTL=64
Reply from 192.168.1.1: bytes=32 time=2ms TTL=64
Reply from 192.168.1.1: bytes=32 time=1ms TTL=64
Reply from 192.168.1.1: bytes=32 time=1ms TTL=64
Request timed out.
Reply from 192.168.1.1: bytes=32 time=2ms TTL=64
Reply from 192.168.1.1: bytes=32 time=1ms TTL=64
Reply from 192.168.1.1: bytes=32 time=2ms TTL=64
Reply from 192.168.1.1: bytes=32 time=1ms TTL=64
Request timed out.
Reply from 192.168.1.1: bytes=32 time=2ms TTL=64
Request timed out.
Reply from 192.168.1.1: bytes=32 time=2ms TTL=64
Reply from 192.168.1.1: bytes=32 time=2ms TTL=64
Request timed out.
Reply from 192.168.1.1: bytes=32 time=4ms TTL=64
Reply from 192.168.1.1: bytes=32 time=1ms TTL=64
Reply from 192.168.1.1: bytes=32 time=1ms TTL=64
Reply from 192.168.1.1: bytes=32 time=1ms TTL=64
Request timed out.
Reply from 192.168.1.1: bytes=32 time=2ms TTL=64

and poinging my wan

Reply from 188.117.46.161: bytes=32 time=17ms TTL=56
Request timed out.
Reply from 188.117.46.161: bytes=32 time=19ms TTL=56
Reply from 188.117.46.161: bytes=32 time=21ms TTL=56
Reply from 188.117.46.161: bytes=32 time=45ms TTL=56
Reply from 188.117.46.161: bytes=32 time=17ms TTL=56
Reply from 188.117.46.161: bytes=32 time=19ms TTL=56
Reply from 188.117.46.161: bytes=32 time=17ms TTL=56
Reply from 188.117.46.161: bytes=32 time=17ms TTL=56
Reply from 188.117.46.161: bytes=32 time=17ms TTL=56
Request timed out.
Reply from 188.117.46.161: bytes=32 time=18ms TTL=56
Reply from 188.117.46.161: bytes=32 time=18ms TTL=56
Reply from 188.117.46.161: bytes=32 time=17ms TTL=56
Reply from 188.117.46.161: bytes=32 time=18ms TTL=56
Request timed out.
Reply from 188.117.46.161: bytes=32 time=19ms TTL=56

I really hate this shit at the moment.

tsmalmbe

So now everything is pretty fucked until I once again reboot the firewall.