Intermittent "no route to host" on my LAN-port

tsmalmbe · May 23, 2017, 1:56 PM

ARP would've been my guess also, but then I don't really understand why it only starts after a while, and then the problem doesn't go away by itself - and the cycle is quite short as you said. But let's see.

The bitchy part is mostly if I need to remove the fw and hook up a laptop directly to the LAN, then it will require a bit of fiddling to get that going. As long as you document the VLAN's it's not such a big deal. And perhaps the correct way is to use the switch to handle the mixing and matching of VLAN's on the switch ports and then just either show all tagged or all untagged to the firewall.

tsmalmbe · May 23, 2017, 2:49 PM

It gets worse.

My two OpenVPN tunnels are now doing the same thing.
Request timed out.
Request timed out.
Reply from 10.99.10.1: bytes=32 time=48ms TTL=63
Reply from 10.99.10.1: bytes=32 time=48ms TTL=63
Reply from 10.99.10.1: bytes=32 time=48ms TTL=63
Request timed out.
Reply from 10.99.10.1: bytes=32 time=49ms TTL=63
Reply from 10.99.10.1: bytes=32 time=52ms TTL=63
Reply from 10.99.10.1: bytes=32 time=47ms TTL=63
Request timed out.
Reply from 10.99.10.1: bytes=32 time=49ms TTL=63
Reply from 10.99.10.1: bytes=32 time=48ms TTL=63
Reply from 10.99.10.1: bytes=32 time=48ms TTL=63
Reply from 10.99.10.1: bytes=32 time=48ms TTL=63
Reply from 10.99.10.1: bytes=32 time=49ms TTL=63
Request timed out.
Request timed out.
Reply from 10.99.10.1: bytes=32 time=64ms TTL=63
Reply from 10.99.10.1: bytes=32 time=48ms TTL=63
Reply from 10.99.10.1: bytes=32 time=47ms TTL=63
Reply from 10.99.10.1: bytes=32 time=48ms TTL=63
Reply from 10.99.10.1: bytes=32 time=48ms TTL=63
Reply from 10.99.10.1: bytes=32 time=49ms TTL=63
Request timed out.
Request timed out.
Reply from 10.99.10.1: bytes=32 time=149ms TTL=63
Reply from 10.99.10.1: bytes=32 time=47ms TTL=63
Reply from 10.99.10.1: bytes=32 time=49ms TTL=63
Reply from 10.99.10.1: bytes=32 time=49ms TTL=63
Reply from 10.99.10.1: bytes=32 time=49ms TTL=63

And

Reply from 192.168.69.1: bytes=32 time=126ms TTL=63
Reply from 192.168.69.1: bytes=32 time=36ms TTL=63
Request timed out.
Request timed out.
Reply from 192.168.69.1: bytes=32 time=70ms TTL=63
Reply from 192.168.69.1: bytes=32 time=110ms TTL=63
Reply from 192.168.69.1: bytes=32 time=61ms TTL=63
Request timed out.
Request timed out.
Reply from 192.168.69.1: bytes=32 time=108ms TTL=63
Reply from 192.168.69.1: bytes=32 time=68ms TTL=63
Reply from 192.168.69.1: bytes=32 time=107ms TTL=63
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Reply from 192.168.69.1: bytes=32 time=109ms TTL=63
Request timed out.
Request timed out.
Reply from 192.168.69.1: bytes=32 time=68ms TTL=63
Reply from 192.168.69.1: bytes=32 time=100ms TTL=63
Reply from 192.168.69.1: bytes=32 time=75ms TTL=63
Reply from 192.168.69.1: bytes=32 time=103ms TTL=63
Reply from 192.168.69.1: bytes=32 time=65ms TTL=63
Request timed out.
Request timed out.
Reply from 192.168.69.1: bytes=32 time=108ms TTL=63
Reply from 192.168.69.1: bytes=32 time=68ms TTL=63
Reply from 192.168.69.1: bytes=32 time=93ms TTL=63
Reply from 192.168.69.1: bytes=32 time=67ms TTL=63
Reply from 192.168.69.1: bytes=32 time=106ms TTL=63
Request timed out.

Also pingig the firewall

Reply from 192.168.1.1: bytes=32 time=1ms TTL=64
Reply from 192.168.1.1: bytes=32 time=1ms TTL=64
Request timed out.
Reply from 192.168.1.1: bytes=32 time=1ms TTL=64
Reply from 192.168.1.1: bytes=32 time=1ms TTL=64
Reply from 192.168.1.1: bytes=32 time=1ms TTL=64
Reply from 192.168.1.1: bytes=32 time=2ms TTL=64
Reply from 192.168.1.1: bytes=32 time=1ms TTL=64
Reply from 192.168.1.1: bytes=32 time=1ms TTL=64
Request timed out.
Reply from 192.168.1.1: bytes=32 time=2ms TTL=64
Reply from 192.168.1.1: bytes=32 time=1ms TTL=64
Reply from 192.168.1.1: bytes=32 time=2ms TTL=64
Reply from 192.168.1.1: bytes=32 time=1ms TTL=64
Request timed out.
Reply from 192.168.1.1: bytes=32 time=2ms TTL=64
Request timed out.
Reply from 192.168.1.1: bytes=32 time=2ms TTL=64
Reply from 192.168.1.1: bytes=32 time=2ms TTL=64
Request timed out.
Reply from 192.168.1.1: bytes=32 time=4ms TTL=64
Reply from 192.168.1.1: bytes=32 time=1ms TTL=64
Reply from 192.168.1.1: bytes=32 time=1ms TTL=64
Reply from 192.168.1.1: bytes=32 time=1ms TTL=64
Request timed out.
Reply from 192.168.1.1: bytes=32 time=2ms TTL=64

and poinging my wan

Reply from 188.117.46.161: bytes=32 time=17ms TTL=56
Request timed out.
Reply from 188.117.46.161: bytes=32 time=19ms TTL=56
Reply from 188.117.46.161: bytes=32 time=21ms TTL=56
Reply from 188.117.46.161: bytes=32 time=45ms TTL=56
Reply from 188.117.46.161: bytes=32 time=17ms TTL=56
Reply from 188.117.46.161: bytes=32 time=19ms TTL=56
Reply from 188.117.46.161: bytes=32 time=17ms TTL=56
Reply from 188.117.46.161: bytes=32 time=17ms TTL=56
Reply from 188.117.46.161: bytes=32 time=17ms TTL=56
Request timed out.
Reply from 188.117.46.161: bytes=32 time=18ms TTL=56
Reply from 188.117.46.161: bytes=32 time=18ms TTL=56
Reply from 188.117.46.161: bytes=32 time=17ms TTL=56
Reply from 188.117.46.161: bytes=32 time=18ms TTL=56
Request timed out.
Reply from 188.117.46.161: bytes=32 time=19ms TTL=56

I really hate this shit at the moment.

tsmalmbe · May 23, 2017, 2:56 PM

So now everything is pretty fucked until I once again reboot the firewall.

tsmalmbe · May 24, 2017, 4:29 PM

When the problem (which is now different but the same) occurs, the firewall can now ping everything so no problem there. But now I lose

VLAN (LAN) -> ovpns5,
VLAN (LAN)->VLAN (SERVERS1) and
VLAN (LAN) -> VLAN (SERVERS2).

VLAN (LAN) to the internet is all fine and working.

There has to be some way to debug this.

tsmalmbe · May 24, 2017, 4:47 PM

Dumping the routing when the issue occurs:

[2.3.4-RELEASE][root@firewall.ccccccccc.fi]/root: netstat -rn
Routing tables

Internet:
Destination Gateway Flags Netif Expire
default 217.30.178.254 UGS sk0
10.10.1.0/24 link#4 U sk3
10.10.1.1 link#4 UHS lo0
10.99.0.0/16 10.100.100.2 UGS ovpns5
10.100.100.1 link#13 UHS lo0
10.100.100.2 link#13 UH ovpns5
10.100.101.1 link#14 UHS lo0
10.100.101.2 link#14 UH ovpns8
10.200.200.0/24 10.200.200.2 UGS ovpns1
10.200.200.1 link#12 UHS lo0
10.200.200.2 link#12 UH ovpns1
10.200.210.0/24 10.100.100.2 UGS ovpns5
83.145.193.133 217.30.178.254 UGHS sk0
127.0.0.1 link#8 UH lo0
192.168.1.0/24 link#11 U sk1_vlan
192.168.1.1 link#11 UHS lo0
192.168.1.3 link#11 UHS lo0
192.168.2.21 link#11 UHS lo0
192.168.2.21/32 link#11 U sk1_vlan
192.168.10.0/24 link#9 U sk1_vlan
192.168.10.1 link#9 UHS lo0
192.168.20.0/24 link#10 U sk1_vlan
192.168.20.1 link#10 UHS lo0
192.168.69.0/24 10.100.101.2 UGS ovpns8
192.168.100.0/24 link#3 U sk2
192.168.100.1 link#3 UHS lo0
217.30.178.0/24 link#1 U sk0
217.30.178.237 link#1 UHS lo0

tsmalmbe · May 24, 2017, 4:47 PM

After reboot

Routing tables

Internet:
Destination Gateway Flags Netif Expire
default 217.30.178.254 UGS sk0
10.10.1.0/24 link#4 U sk3
10.10.1.1 link#4 UHS lo0
10.99.0.0/16 10.100.100.2 UGS ovpns5
10.100.100.1 link#13 UHS lo0
10.100.100.2 link#13 UH ovpns5
10.100.101.1 link#14 UHS lo0
10.100.101.2 link#14 UH ovpns8
10.200.200.0/24 10.200.200.2 UGS ovpns1
10.200.200.1 link#12 UHS lo0
10.200.200.2 link#12 UH ovpns1
10.200.210.0/24 10.100.100.2 UGS ovpns5
83.145.193.133 217.30.178.254 UGHS sk0
127.0.0.1 link#8 UH lo0
192.168.1.0/24 link#11 U sk1_vlan
192.168.1.1 link#11 UHS lo0
192.168.1.3 link#11 UHS lo0
192.168.2.21 link#11 UHS lo0
192.168.2.21/32 link#11 U sk1_vlan
192.168.10.0/24 link#9 U sk1_vlan
192.168.10.1 link#9 UHS lo0
192.168.20.0/24 link#10 U sk1_vlan
192.168.20.1 link#10 UHS lo0
192.168.69.0/24 10.100.101.2 UGS ovpns8
192.168.100.0/24 link#3 U sk2
192.168.100.1 link#3 UHS lo0
217.30.178.0/24 link#1 U sk0
217.30.178.237 link#1 UHS lo0

Derelict · May 24, 2017, 7:45 PM

As far as I know, almost nobody is using those sk NICs.

At the same time I haven't heard of anything like what you're seeing either.

Derelict · May 24, 2017, 7:50 PM

192.168.1.0/24 link#11 U sk1_vlan
192.168.1.1 link#11 UHS lo0
192.168.1.3 link#11 UHS lo0
192.168.2.21 link#11 UHS lo0
192.168.2.21/32 link#11 U sk1_vlan

Why are both of those on link #11?

netstat -rnWfinet

tsmalmbe · May 25, 2017, 8:19 AM

1.1 is the router itself
1.3 is a virtual ip
2.21 is a virtual ip

192.168.1.3/32 LAN IP Alias Virtual IP for wpad.malmberg.fi
192.168.2.21/32 LAN IP Alias Virtual IP for wiki.malmberg.fi

I also realized that I have had it defined as 1.3/24 instead of 1.3/32 - this I have now changed.

I have run your suggested netstat -command BEFORE and AFTER this change.

BEFORE

netstat -rnWfinet
Routing tables

Internet:
Destination Gateway Flags Use Mtu Netif Expire
default 217.30.178.254 UGS 307699 1500 sk0
10.10.1.0/24 link#4 U 401 1500 sk3
10.10.1.1 link#4 UHS 0 16384 lo0
10.99.0.0/16 10.100.100.2 UGS 66905 1500 ovpns5
10.100.100.1 link#13 UHS 0 16384 lo0
10.100.100.2 link#13 UH 6 1500 ovpns5
10.100.101.1 link#14 UHS 0 16384 lo0
10.100.101.2 link#14 UH 0 1500 ovpns8
10.200.200.0/24 10.200.200.2 UGS 0 1500 ovpns1
10.200.200.1 link#12 UHS 0 16384 lo0
10.200.200.2 link#12 UH 0 1500 ovpns1
10.200.210.0/24 10.100.100.2 UGS 0 1500 ovpns5
83.145.193.133 217.30.178.254 UGHS 111003 1500 sk0
127.0.0.1 link#8 UH 354961 16384 lo0
192.168.1.0/24 link#11 U 2433840 1500 sk1_vlan5
192.168.1.1 link#11 UHS 0 16384 lo0
192.168.1.3 link#11 UHS 0 16384 lo0
192.168.2.21 link#11 UHS 810 16384 lo0
192.168.2.21/32 link#11 U 0 1500 sk1_vlan5
192.168.10.0/24 link#9 U 487 1500 sk1_vlan6
192.168.10.1 link#9 UHS 0 16384 lo0
192.168.20.0/24 link#10 U 36126 1500 sk1_vlan7
192.168.20.1 link#10 UHS 0 16384 lo0
192.168.69.0/24 10.100.101.2 UGS 0 1500 ovpns8
192.168.100.0/24 link#3 U 0 1500 sk2
192.168.100.1 link#3 UHS 0 16384 lo0
217.30.178.0/24 link#1 U 0 1500 sk0
217.30.178.237 link#1 UHS 0 16384 lo0

AFTER

Routing tables

Internet:
Destination Gateway Flags Use Mtu Netif Expire
default 217.30.178.254 UGS 309567 1500 sk0
10.10.1.0/24 link#4 U 401 1500 sk3
10.10.1.1 link#4 UHS 0 16384 lo0
10.99.0.0/16 10.100.100.2 UGS 67281 1500 ovpns5
10.100.100.1 link#13 UHS 0 16384 lo0
10.100.100.2 link#13 UH 6 1500 ovpns5
10.100.101.1 link#14 UHS 0 16384 lo0
10.100.101.2 link#14 UH 0 1500 ovpns8
10.200.200.0/24 10.200.200.2 UGS 0 1500 ovpns1
10.200.200.1 link#12 UHS 0 16384 lo0
10.200.200.2 link#12 UH 0 1500 ovpns1
10.200.210.0/24 10.100.100.2 UGS 0 1500 ovpns5
83.145.193.133 217.30.178.254 UGHS 111477 1500 sk0
127.0.0.1 link#8 UH 357517 16384 lo0
192.168.1.0/24 link#11 U 2435972 1500 sk1_vlan5
192.168.1.1 link#11 UHS 0 16384 lo0
192.168.1.3 link#11 UHS 0 16384 lo0
192.168.1.3/32 link#11 U 0 1500 sk1_vlan5
192.168.2.21 link#11 UHS 0 16384 lo0
192.168.2.21/32 link#11 U 0 1500 sk1_vlan5
192.168.10.0/24 link#9 U 490 1500 sk1_vlan6
192.168.10.1 link#9 UHS 0 16384 lo0
192.168.20.0/24 link#10 U 36269 1500 sk1_vlan7
192.168.20.1 link#10 UHS 0 16384 lo0
192.168.69.0/24 10.100.101.2 UGS 0 1500 ovpns8
192.168.100.0/24 link#3 U 0 1500 sk2
192.168.100.1 link#3 UHS 0 16384 lo0
217.30.178.0/24 link#1 U 0 1500 sk0
217.30.178.237 link#1 UHS 0 16384 lo0

tsmalmbe · May 25, 2017, 8:35 AM

I know this is not a perfect timeline, but two simultaneous windows though.

11:33:52.822204 ARP, Request who-has 192.168.20.7 tell 192.168.20.1, length 28
11:33:52.822442 ARP, Reply 192.168.20.7 is-at 7a:f5:5b:ff:c9:c4, length 42
11:33:52.822452 IP 192.168.1.191 > 192.168.20.7: ICMP echo request, id 6, seq 11405, length 40
11:33:52.822642 IP 192.168.20.7 > 192.168.1.191: ICMP echo reply, id 6, seq 11405, length 40
11:33:53.822241 ARP, Request who-has 192.168.20.7 tell 192.168.20.1, length 28
11:33:53.822469 ARP, Reply 192.168.20.7 is-at 7a:f5:5b:ff:c9:c4, length 42
11:33:53.822482 IP 192.168.1.191 > 192.168.20.7: ICMP echo request, id 6, seq 11407, length 40
11:33:53.822669 IP 192.168.20.7 > 192.168.1.191: ICMP echo reply, id 6, seq 11407, length 40
11:33:54.823260 ARP, Request who-has 192.168.20.7 tell 192.168.20.1, length 28
11:33:54.823490 ARP, Reply 192.168.20.7 is-at 7a:f5:5b:ff:c9:c4, length 42
11:33:54.823501 IP 192.168.1.191 > 192.168.20.7: ICMP echo request, id 6, seq 11409, length 40
11:33:54.823690 IP 192.168.20.7 > 192.168.1.191: ICMP echo reply, id 6, seq 11409, length 40
11:33:55.825579 IP 192.168.1.191 > 192.168.20.7: ICMP echo request, id 6, seq 11411, length 40
11:33:55.825812 IP 192.168.20.7 > 192.168.1.191: ICMP echo reply, id 6, seq 11411, length 40
11:33:56.826596 IP 192.168.1.191 > 192.168.20.7: ICMP echo request, id 6, seq 11414, length 40
11:33:56.826836 IP 192.168.20.7 > 192.168.1.191: ICMP echo reply, id 6, seq 11414, length 40
11:33:57.829819 IP 192.168.1.191 > 192.168.20.7: ICMP echo request, id 6, seq 11417, length 40
11:33:57.830054 IP 192.168.20.7 > 192.168.1.191: ICMP echo reply, id 6, seq 11417, length 40

Reply from 192.168.1.1: Destination host unreachable.
Reply from 192.168.1.1: Destination host unreachable.
Reply from 192.168.1.1: Destination host unreachable.
Reply from 192.168.1.1: Destination host unreachable.
Reply from 192.168.1.1: Destination host unreachable.
Reply from 192.168.1.1: Destination host unreachable.
Reply from 192.168.20.7: bytes=32 time=3ms TTL=63
Reply from 192.168.20.7: bytes=32 time=114ms TTL=63
Reply from 192.168.20.7: bytes=32 time=2ms TTL=63
Reply from 192.168.20.7: bytes=32 time=3ms TTL=63
Reply from 192.168.20.7: bytes=32 time=3ms TTL=63
Reply from 192.168.20.7: bytes=32 time=6ms TTL=63
Reply from 192.168.20.7: bytes=32 time=1ms TTL=63
Reply from 192.168.20.7: bytes=32 time=3ms TTL=63
Reply from 192.168.20.7: bytes=32 time=2ms TTL=63
Reply from 192.168.20.7: bytes=32 time=2ms TTL=63
Reply from 192.168.20.7: bytes=32 time=1ms TTL=63
Reply from 192.168.20.7: bytes=32 time=3ms TTL=63

So WTF is going on with these ARP's.

tsmalmbe · May 25, 2017, 8:38 AM

11:35:52.476415 ARP, Request who-has 192.168.20.7 tell 192.168.20.1, length 28
11:35:52.476692 ARP, Reply 192.168.20.7 is-at 7a:f5:5b:ff:c9:c4, length 42
11:35:52.476703 IP 192.168.1.191 > 192.168.20.7: ICMP echo request, id 6, seq 11614, length 40
11:35:52.476892 IP 192.168.20.7 > 192.168.1.191: ICMP echo reply, id 6, seq 11614, length 40
11:35:53.475585 ARP, Request who-has 192.168.20.7 tell 192.168.20.1, length 28
11:35:53.475821 ARP, Reply 192.168.20.7 is-at 7a:f5:5b:ff:c9:c4, length 42
11:35:53.475833 IP 192.168.1.191 > 192.168.20.7: ICMP echo request, id 6, seq 11616, length 40
11:35:53.476019 IP 192.168.20.7 > 192.168.1.191: ICMP echo reply, id 6, seq 11616, length 40
11:35:53.890421 IP 192.168.20.7.138 > 192.168.1.255.138: UDP, length 227
11:35:54.477796 IP 192.168.1.191 > 192.168.20.7: ICMP echo request, id 6, seq 11619, length 40
11:35:54.478040 IP 192.168.20.7 > 192.168.1.191: ICMP echo reply, id 6, seq 11619, length 40
11:35:55.477931 IP 192.168.1.191 > 192.168.20.7: ICMP echo request, id 6, seq 11622, length 40
11:35:55.478166 IP 192.168.20.7 > 192.168.1.191: ICMP echo reply, id 6, seq 11622, length 40
11:35:56.478949 IP 192.168.1.191 > 192.168.20.7: ICMP echo request, id 6, seq 11625, length 40
11:35:56.479188 IP 192.168.20.7 > 192.168.1.191: ICMP echo reply, id 6, seq 11625, length 40
11:35:57.479970 IP 192.168.1.191 > 192.168.20.7: ICMP echo request, id 6, seq 11628, length 40
11:35:57.480212 IP 192.168.20.7 > 192.168.1.191: ICMP echo reply, id 6, seq 11628, length 40
11:35:58.480948 IP 192.168.1.191 > 192.168.20.7: ICMP echo request, id 6, seq 11631, length 40
11:35:58.481133 IP 192.168.20.7 > 192.168.1.191: ICMP echo reply, id 6, seq 11631, length 40
11:35:59.482083 ARP, Request who-has 192.168.20.7 tell 192.168.20.1, length 28
11:35:59.482256 ARP, Reply 192.168.20.7 is-at 7a:f5:5b:ff:c9:c4, length 42
11:35:59.482261 IP 192.168.1.191 > 192.168.20.7: ICMP echo request, id 6, seq 11634, length 40
11:35:59.482455 IP 192.168.20.7 > 192.168.1.191: ICMP echo reply, id 6, seq 11634, length 40
11:36:00.483100 IP 192.168.1.191 > 192.168.20.7: ICMP echo request, id 6, seq 11637, length 40
11:36:00.483379 IP 192.168.20.7 > 192.168.1.191: ICMP echo reply, id 6, seq 11637, length 40
11:36:01.483127 IP 192.168.1.191 > 192.168.20.7: ICMP echo request, id 6, seq 11640, length 40
11:36:01.483305 IP 192.168.20.7 > 192.168.1.191: ICMP echo reply, id 6, seq 11640, length 40
11:36:02.485267 IP 192.168.1.191 > 192.168.20.7: ICMP echo request, id 6, seq 11643, length 40
11:36:02.485527 IP 192.168.20.7 > 192.168.1.191: ICMP echo reply, id 6, seq 11643, length 40
11:36:03.485210 IP 192.168.1.191 > 192.168.20.7: ICMP echo request, id 6, seq 11646, length 40
11:36:03.485452 IP 192.168.20.7 > 192.168.1.191: ICMP echo reply, id 6, seq 11646, length 40
11:36:04.485242 IP 192.168.1.191 > 192.168.20.7: ICMP echo request, id 6, seq 11649, length 40
11:36:04.485476 IP 192.168.20.7 > 192.168.1.191: ICMP echo reply, id 6, seq 11649, length 40
11:36:04.485776 ARP, Request who-has 192.168.20.1 tell 192.168.20.7, length 42
11:36:04.485786 ARP, Reply 192.168.20.1 is-at 00:90:7f:40:45:54, length 28
11:36:04.597913 IP 192.168.20.7.139 > 192.168.1.166.57005: tcp 4
11:36:05.369166 IP 192.168.1.14.137 > 192.168.20.7.137: UDP, length 50
11:36:05.369769 IP 192.168.20.7.137 > 192.168.1.14.137: UDP, length 62
11:36:05.370391 IP 192.168.1.14.138 > 192.168.20.7.138: UDP, length 177
11:36:05.374596 IP 192.168.1.14.3514 > 192.168.20.7.139: tcp 0
11:36:05.374863 IP 192.168.20.7.139 > 192.168.1.14.3514: tcp 0
11:36:05.375069 IP 192.168.1.14.3514 > 192.168.20.7.139: tcp 0
11:36:05.388350 IP 192.168.1.14.3514 > 192.168.20.7.139: tcp 72
11:36:05.388543 IP 192.168.20.7.139 > 192.168.1.14.3514: tcp 0
11:36:05.389041 IP 192.168.20.7.139 > 192.168.1.14.3514: tcp 4
11:36:05.389148 IP 192.168.1.14.3514 > 192.168.20.7.139: tcp 0
11:36:05.389347 IP 192.168.1.14.3514 > 192.168.20.7.139: tcp 194
11:36:05.394133 IP 192.168.20.7.57303 > 192.168.1.1.53: UDP, length 37
11:36:05.428884 IP 192.168.20.7.139 > 192.168.1.14.3514: tcp 0
11:36:05.487307 IP 192.168.1.191 > 192.168.20.7: ICMP echo request, id 6, seq 11652, length 40
11:36:05.487495 IP 192.168.20.7 > 192.168.1.191: ICMP echo reply, id 6, seq 11652, length 40
11:36:06.488447 IP 192.168.1.191 > 192.168.20.7: ICMP echo request, id 6, seq 11655, length 40
11:36:06.488719 IP 192.168.20.7 > 192.168.1.191: ICMP echo reply, id 6, seq 11655, length 40
11:36:07.488499 IP 192.168.1.191 > 192.168.20.7: ICMP echo request, id 6, seq 11658, length 40
11:36:07.488743 IP 192.168.20.7 > 192.168.1.191: ICMP echo reply, id 6, seq 11658, length 40
11:36:08.490590 IP 192.168.1.191 > 192.168.20.7: ICMP echo request, id 6, seq 11661, length 40
11:36:08.490764 IP 192.168.20.7 > 192.168.1.191: ICMP echo reply, id 6, seq 11661, length 40
11:36:09.491648 IP 192.168.1.191 > 192.168.20.7: ICMP echo request, id 6, seq 11664, length 40
11:36:09.491889 IP 192.168.20.7 > 192.168.1.191: ICMP echo reply, id 6, seq 11664, length 40
11:36:10.399253 IP 192.168.20.7.57303 > 192.168.1.1.53: UDP, length 37
11:36:10.722106 IP 192.168.1.1.53 > 192.168.20.7.57303: UDP, length 103
11:36:10.722473 IP 192.168.20.7.32781 > 192.168.1.1.53: UDP, length 49
11:36:10.743034 IP 192.168.1.1.53 > 192.168.20.7.32781: UDP, length 115
11:36:10.744939 IP 192.168.20.7.139 > 192.168.1.14.3514: tcp 163
11:36:10.744963 IP 192.168.20.1 > 192.168.20.7: ICMP host 192.168.1.14 unreachable, length 60
11:36:10.945952 IP 192.168.20.7.139 > 192.168.1.14.3514: tcp 163
11:36:10.945993 IP 192.168.20.1 > 192.168.20.7: ICMP host 192.168.1.14 unreachable, length 60
11:36:11.146854 IP 192.168.20.7.139 > 192.168.1.14.3514: tcp 163
11:36:11.146915 ARP, Request who-has 192.168.20.7 tell 192.168.20.1, length 28
11:36:11.147147 ARP, Reply 192.168.20.7 is-at 7a:f5:5b:ff:c9:c4, length 42
11:36:11.147157 IP 192.168.20.1 > 192.168.20.7: ICMP host 192.168.1.14 unreachable, length 60
11:36:11.347851 IP 192.168.20.7.139 > 192.168.1.14.3514: tcp 163
11:36:11.347879 IP 192.168.20.1 > 192.168.20.7: ICMP host 192.168.1.14 unreachable, length 60
11:36:11.548874 IP 192.168.20.7.139 > 192.168.1.14.3514: tcp 163
11:36:11.548950 IP 192.168.20.1 > 192.168.20.7: ICMP host 192.168.1.14 unreachable, length 60
11:36:11.749860 IP 192.168.20.7.139 > 192.168.1.14.3514: tcp 163
11:36:11.749892 IP 192.168.20.1 > 192.168.20.7: ICMP host 192.168.1.14 unreachable, length 60
11:36:11.950968 IP 192.168.20.7.139 > 192.168.1.14.3514: tcp 163
11:36:11.951009 IP 192.168.20.1 > 192.168.20.7: ICMP host 192.168.1.14 unreachable, length 60
11:36:12.151867 IP 192.168.20.7.139 > 192.168.1.14.3514: tcp 163
11:36:12.151908 IP 192.168.20.1 > 192.168.20.7: ICMP host 192.168.1.14 unreachable, length 60
11:36:12.352873 IP 192.168.20.7.139 > 192.168.1.14.3514: tcp 163
11:36:12.352916 IP 192.168.20.1 > 192.168.20.7: ICMP host 192.168.1.14 unreachable, length 60
11:36:12.553878 IP 192.168.20.7.139 > 192.168.1.14.3514: tcp 163
11:36:12.553927 IP 192.168.20.1 > 192.168.20.7: ICMP host 192.168.1.14 unreachable, length 60
11:36:12.754878 IP 192.168.20.7.139 > 192.168.1.14.3514: tcp 163
11:36:12.754921 IP 192.168.20.1 > 192.168.20.7: ICMP host 192.168.1.14 unreachable, length 60
11:36:12.955897 IP 192.168.20.7.139 > 192.168.1.14.3514: tcp 163
11:36:12.955969 IP 192.168.20.1 > 192.168.20.7: ICMP host 192.168.1.14 unreachable, length 60
11:36:13.156900 IP 192.168.20.7.139 > 192.168.1.14.3514: tcp 163
11:36:13.156968 IP 192.168.20.1 > 192.168.20.7: ICMP host 192.168.1.14 unreachable, length 60
11:36:13.357906 IP 192.168.20.7.139 > 192.168.1.14.3514: tcp 163
11:36:13.357972 IP 192.168.20.1 > 192.168.20.7: ICMP host 192.168.1.14 unreachable, length 60
11:36:13.558892 IP 192.168.20.7.139 > 192.168.1.14.3514: tcp 163
11:36:13.558928 IP 192.168.20.1 > 192.168.20.7: ICMP host 192.168.1.14 unreachable, length 60
11:36:13.759896 IP 192.168.20.7.139 > 192.168.1.14.3514: tcp 163
11:36:13.759921 IP 192.168.20.1 > 192.168.20.7: ICMP host 192.168.1.14 unreachable, length 60
11:36:13.960906 IP 192.168.20.7.139 > 192.168.1.14.3514: tcp 163
11:36:13.960936 IP 192.168.20.1 > 192.168.20.7: ICMP host 192.168.1.14 unreachable, length 60
11:36:14.161910 IP 192.168.20.7.139 > 192.168.1.14.3514: tcp 163

tsmalmbe · May 25, 2017, 8:56 AM

netstat -m
1855/1940/3795 mbufs in use (current/cache/total)
1438/338/1776/26368 mbuf clusters in use (current/cache/total/max)
1438/333 mbuf+clusters out of packet secondary zone in use (current/cache)
0/120/120/13184 4k (page size) jumbo clusters in use (current/cache/total/max)
0/0/0/3906 9k jumbo clusters in use (current/cache/total/max)
0/0/0/2197 16k jumbo clusters in use (current/cache/total/max)
3352K/1641K/4993K bytes allocated to network (current/cache/total)
0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
0/0/0 requests for mbufs delayed (mbufs/clusters/mbuf+clusters)
0/0/0 requests for jumbo clusters delayed (4k/9k/16k)
0/0/0 requests for jumbo clusters denied (4k/9k/16k)
0/10/6656 sfbufs in use (current/peak/max)
0 requests for sfbufs denied
0 requests for sfbufs delayed
0 requests for I/O initiated by sendfile

I also did service netif restart and then no traffic on any interface worked so I had to reboot again.

Derelict · May 25, 2017, 4:41 PM

1.1 is the router itself
1.3 is a virtual ip
2.21 is a virtual ip

192.168.1.3/32 LAN IP Alias Virtual IP for wpad.malmberg.fi
192.168.2.21/32 LAN IP Alias Virtual IP for wiki.malmberg.fi

So you're playing games with multiple Layer 3 networks on VLAN 5 and you are having problems. Perhaps don't do silly things like that. Makes me wonder what other questionable design decisions you have made elsewhere. Seems they are coming home to roost. What kind of switch are you using? If you post more packet captures please indicate where they were taken and how.

tsmalmbe · May 25, 2017, 6:58 PM

So, the 192.168.2-base ip is now gone. I will look into adding a network solely for my HAProxy addresses some other way, perhaps using a VLAN which is only available on pfSense or something along those lines. But I will leave that for now as it most obviously can be a part of the problem. I added 1.6/32 as a VIP instead to serve my wiki.

My packet dumps are from pfSense.

My switch is a Procurve 2848 (J4904A).

Let's see if the removal of the VIP makes a difference.

It would be foolish to say that this was the only questionable design I have made, although most of my decisions are preceded by somewhat thorough investigations.

Routing tables

Internet:
Destination Gateway Flags Use Mtu Netif Expire
default 217.30.178.254 UGS 275029 1500 sk0
10.10.1.0/24 link#4 U 0 1500 sk3
10.10.1.1 link#4 UHS 0 16384 lo0
10.99.0.0/16 10.100.100.2 UGS 67877 1500 ovpns5
10.100.100.1 link#13 UHS 0 16384 lo0
10.100.100.2 link#13 UH 0 1500 ovpns5
10.100.101.1 link#14 UHS 0 16384 lo0
10.100.101.2 link#14 UH 0 1500 ovpns8
10.200.200.0/24 10.200.200.2 UGS 0 1500 ovpns1
10.200.200.1 link#12 UHS 0 16384 lo0
10.200.200.2 link#12 UH 0 1500 ovpns1
10.200.210.0/24 10.100.100.2 UGS 0 1500 ovpns5
83.145.193.133 217.30.178.254 UGHS 72257 1500 sk0
127.0.0.1 link#8 UH 231466 16384 lo0
192.168.1.0/24 link#11 U 640239 1500 sk1_vlan5
192.168.1.1 link#11 UHS 0 16384 lo0
192.168.1.3 link#11 UHS 0 16384 lo0
192.168.1.3/32 link#11 U 0 1500 sk1_vlan5
192.168.1.6 link#11 UHS 0 16384 lo0
192.168.1.6/32 link#11 U 0 1500 sk1_vlan5
192.168.10.0/24 link#9 U 319 1500 sk1_vlan6
192.168.10.1 link#9 UHS 0 16384 lo0
192.168.20.0/24 link#10 U 65604 1500 sk1_vlan7
192.168.20.1 link#10 UHS 0 16384 lo0
192.168.69.0/24 10.100.101.2 UGS 0 1500 ovpns8
192.168.100.0/24 link#3 U 0 1500 sk2
192.168.100.1 link#3 UHS 0 16384 lo0
217.30.178.0/24 link#1 U 0 1500 sk0
217.30.178.237 link#1 UHS 0 16384 lo0

tsmalmbe · May 25, 2017, 9:52 PM

Made no real difference.

00:44:13.624423 IP 192.168.1.191.60359 > 192.168.1.1.53: UDP, length 38
00:44:13.624597 IP 192.168.1.1.53 > 192.168.1.191.60359: UDP, length 54
00:44:13.627115 IP 192.168.1.191.54459 > 192.168.1.1.53: UDP, length 38
00:44:13.627161 IP 192.168.1.1.53 > 192.168.1.191.54459: UDP, length 54
00:44:13.920983 IP 192.168.1.191.41804 > 192.168.1.1.443: tcp 1
00:44:13.921020 IP 192.168.1.1.443 > 192.168.1.191.41804: tcp 0
00:44:14.113198 IP 192.168.1.191.41889 > 192.168.1.1.3128: tcp 0
00:44:14.113255 IP 192.168.1.1.3128 > 192.168.1.191.41889: tcp 0
00:44:14.114696 IP 192.168.1.191.41889 > 192.168.1.1.3128: tcp 0
00:44:14.132373 IP 192.168.1.191.41889 > 192.168.1.1.3128: tcp 0
00:44:14.132397 IP 192.168.1.1.3128 > 192.168.1.191.41889: tcp 0
00:44:14.132481 IP 192.168.1.1.3128 > 192.168.1.191.41889: tcp 0
00:44:14.133968 IP 192.168.1.191.41889 > 192.168.1.1.3128: tcp 0
00:44:14.372141 IP 192.168.1.1 > 192.168.1.191: ICMP host 192.168.20.7 unreachable, length 36
00:44:15.372264 IP 192.168.1.1 > 192.168.1.191: ICMP host 192.168.20.7 unreachable, length 36
00:44:15.647337 IP 192.168.1.191.41827 > 192.168.1.1.3128: tcp 46
00:44:15.647388 IP 192.168.1.1.3128 > 192.168.1.191.41827: tcp 0
00:44:15.647834 IP 192.168.1.191.41827 > 192.168.1.1.3128: tcp 31
00:44:15.647850 IP 192.168.1.1.3128 > 192.168.1.191.41827: tcp 0
00:44:15.648333 IP 192.168.1.191.41827 > 192.168.1.1.3128: tcp 0
00:44:15.648354 IP 192.168.1.1.3128 > 192.168.1.191.41827: tcp 0
00:44:15.648586 IP 192.168.1.1.3128 > 192.168.1.191.41827: tcp 0
00:44:15.649732 IP 192.168.1.191.41827 > 192.168.1.1.3128: tcp 0
00:44:16.373381 IP 192.168.1.1 > 192.168.1.191: ICMP host 192.168.20.7 unreachable, length 36
00:44:17.374319 IP 192.168.1.1 > 192.168.1.191: ICMP host 192.168.20.7 unreachable, length 36
00:44:18.375471 IP 192.168.1.1 > 192.168.1.191: ICMP host 192.168.20.7 unreachable, length 36
00:44:19.377456 IP 192.168.1.1 > 192.168.1.191: ICMP host 192.168.20.7 unreachable, length 36
00:44:20.212401 IP 192.168.1.191.40996 > 192.168.1.1.3128: tcp 1
00:44:20.212425 IP 192.168.1.1.3128 > 192.168.1.191.40996: tcp 0
00:44:20.378476 IP 192.168.1.1 > 192.168.1.191: ICMP host 192.168.20.7 unreachable, length 36
00:44:20.414702 IP 192.168.1.191.41703 > 192.168.1.1.3128: tcp 36
00:44:20.414729 IP 192.168.1.1.3128 > 192.168.1.191.41703: tcp 0
00:44:21.072818 ARP, Request who-has 192.168.1.169 tell 192.168.1.1, length 28
00:44:21.134440 ARP, Reply 192.168.1.169 is-at d8:0f:99:2d:58:a5, length 46
00:44:21.379496 IP 192.168.1.1 > 192.168.1.191: ICMP host 192.168.20.7 unreachable, length 36
00:44:22.380614 IP 192.168.1.1 > 192.168.1.191: ICMP host 192.168.20.7 unreachable, length 36
00:44:23.381640 IP 192.168.1.1 > 192.168.1.191: ICMP host 192.168.20.7 unreachable, length 36
00:44:23.606706 IP 192.168.1.191.41861 > 192.168.1.1.443: tcp 1
00:44:23.606781 IP 192.168.1.1.443 > 192.168.1.191.41861: tcp 0
00:44:23.929637 IP 192.168.1.191.41804 > 192.168.1.1.443: tcp 1
00:44:23.929717 IP 192.168.1.1.443 > 192.168.1.191.41804: tcp 0
00:44:24.382789 IP 192.168.1.1 > 192.168.1.191: ICMP host 192.168.20.7 unreachable, length 36
00:44:25.383800 IP 192.168.1.1 > 192.168.1.191: ICMP host 192.168.20.7 unreachable, length 36
00:44:26.182792 ARP, Request who-has 192.168.1.1 tell 192.168.1.169, length 46
00:44:26.182806 ARP, Reply 192.168.1.1 is-at 00:90:7f:40:45:54, length 28
00:44:26.385013 IP 192.168.1.1 > 192.168.1.191: ICMP host 192.168.20.7 unreachable, length 36
00:44:27.385931 IP 192.168.1.1 > 192.168.1.191: ICMP host 192.168.20.7 unreachable, length 36
00:44:28.388054 IP 192.168.1.1 > 192.168.1.191: ICMP host 192.168.20.7 unreachable, length 36
00:44:28.600625 IP 192.168.1.191.39585 > 192.168.1.1.3128: tcp 805
00:44:28.600657 IP 192.168.1.1.3128 > 192.168.1.191.39585: tcp 0
00:44:29.374504 IP 192.168.1.1.3128 > 192.168.1.191.39585: tcp 1460
00:44:29.374521 IP 192.168.1.1.3128 > 192.168.1.191.39585: tcp 279
00:44:29.377592 IP 192.168.1.191.39585 > 192.168.1.1.3128: tcp 0
00:44:29.389036 IP 192.168.1.1 > 192.168.1.191: ICMP host 192.168.20.7 unreachable, length 36
00:44:30.214045 IP 192.168.1.191.40996 > 192.168.1.1.3128: tcp 1
00:44:30.214070 IP 192.168.1.1.3128 > 192.168.1.191.40996: tcp 0
00:44:30.390005 IP 192.168.1.1 > 192.168.1.191: ICMP host 192.168.20.7 unreachable, length 36
00:44:31.391255 IP 192.168.1.1 > 192.168.1.191: ICMP host 192.168.20.7 unreachable, length 36
00:44:32.392346 IP 192.168.1.1 > 192.168.1.191: ICMP host 192.168.20.7 unreachable, length 36
00:44:33.394514 IP 192.168.1.1 > 192.168.1.191: ICMP host 192.168.20.7 unreachable, length 36
00:44:33.607239 IP 192.168.1.191.41861 > 192.168.1.1.443: tcp 1
00:44:33.607283 IP 192.168.1.1.443 > 192.168.1.191.41861: tcp 0
00:44:33.933459 IP 192.168.1.191.41804 > 192.168.1.1.443: tcp 1
00:44:33.933483 IP 192.168.1.1.443 > 192.168.1.191.41804: tcp 0
00:44:34.395412 IP 192.168.1.1 > 192.168.1.191: ICMP host 192.168.20.7 unreachable, length 36
00:44:35.396425 IP 192.168.1.1 > 192.168.1.191: ICMP host 192.168.20.7 unreachable, length 36
00:44:36.397542 IP 192.168.1.1 > 192.168.1.191: ICMP host 192.168.20.7 unreachable, length 36
00:44:37.398571 IP 192.168.1.1 > 192.168.1.191: ICMP host 192.168.20.7 unreachable, length 36
00:44:38.021286 IP 192.168.1.1.3128 > 192.168.1.191.39609: tcp 92
00:44:38.222530 IP 192.168.1.191.39609 > 192.168.1.1.3128: tcp 0
00:44:38.399706 IP 192.168.1.1 > 192.168.1.191: ICMP host 192.168.20.7 unreachable, length 36
00:44:39.400709 IP 192.168.1.1 > 192.168.1.191: ICMP host 192.168.20.7 unreachable, length 36
00:44:40.220600 IP 192.168.1.191.40996 > 192.168.1.1.3128: tcp 1
00:44:40.220682 IP 192.168.1.1.3128 > 192.168.1.191.40996: tcp 0
00:44:40.308870 IP 192.168.1.191.39609 > 192.168.1.1.3128: tcp 388
00:44:40.308948 IP 192.168.1.1.3128 > 192.168.1.191.39609: tcp 0
00:44:40.309052 IP 192.168.1.191.39609 > 192.168.1.1.3128: tcp 156
00:44:40.309069 IP 192.168.1.1.3128 > 192.168.1.191.39609: tcp 0
00:44:40.401746 IP 192.168.1.1 > 192.168.1.191: ICMP host 192.168.20.7 unreachable, length 36
00:44:40.533902 IP 192.168.1.1.3128 > 192.168.1.191.39609: tcp 107
00:44:40.593171 IP 192.168.1.1.3128 > 192.168.1.191.39609: tcp 401
00:44:40.595328 IP 192.168.1.191.39609 > 192.168.1.1.3128: tcp 0
00:44:41.402919 IP 192.168.1.1 > 192.168.1.191: ICMP host 192.168.20.7 unreachable, length 36
00:44:42.404878 IP 192.168.1.1 > 192.168.1.191: ICMP host 192.168.20.7 unreachable, length 36
00:44:43.405796 IP 192.168.1.1 > 192.168.1.191: ICMP host 192.168.20.7 unreachable, length 36
00:44:43.604789 IP 192.168.1.191.41861 > 192.168.1.1.443: tcp 1
00:44:43.604823 IP 192.168.1.1.443 > 192.168.1.191.41861: tcp 0
00:44:43.938897 IP 192.168.1.191.41804 > 192.168.1.1.443: tcp 1
00:44:43.938918 IP 192.168.1.1.443 > 192.168.1.191.41804: tcp 0
00:44:44.406929 IP 192.168.1.1 > 192.168.1.191: ICMP host 192.168.20.7 unreachable, length 36
00:44:45.408085 IP 192.168.1.1 > 192.168.1.191: ICMP host 192.168.20.7 unreachable, length 36
00:44:46.410172 IP 192.168.1.1 > 192.168.1.191: ICMP host 192.168.20.7 unreachable, length 36
00:44:47.200184 IP 192.168.1.191.39609 > 192.168.1.1.3128: tcp 42
00:44:47.200211 IP 192.168.1.1.3128 > 192.168.1.191.39609: tcp 0
00:44:47.411191 IP 192.168.1.1 > 192.168.1.191: ICMP host 192.168.20.7 unreachable, length 36
00:44:48.209196 IP 192.168.1.191.39609 > 192.168.1.1.3128: tcp 471
00:44:48.209226 IP 192.168.1.1.3128 > 192.168.1.191.39609: tcp 0
00:44:48.405610 IP 192.168.1.1.3128 > 192.168.1.191.39609: tcp 97
00:44:48.412215 IP 192.168.1.1 > 192.168.1.191: ICMP host 192.168.20.7 unreachable, length 36
00:44:48.412443 IP 192.168.1.1.3128 > 192.168.1.191.39609: tcp 197

From pfSense, capturing 192.168.1.1 this time.

The pinging is done from 1.191.

Derelict · May 25, 2017, 10:27 PM

I would try a different NIC. Or capture on a monitor port on the switch and see if the pings are really going out on the wire.

Almost nobody uses those sk NICs.

tsmalmbe · May 26, 2017, 1:36 PM

I'll go ahead and move my ports around a bit later tonight just to cross out the possibility of hardware failure.

I've liked the idea of using these Watchguards as they are "proper" pedigree firewalls. I have been looking at migratiing to an XTM-series, but I just had huge issues getting the latest nano-release even starting on that one.

tsmalmbe · May 31, 2017, 7:38 PM

Both the primary WG and my secondary WG got totally screwed up and wont even boot in an orderly fashion.

Luckily I took a backup before venturing into testin.

As a workaround, I installed pfSense on proxmox, hooked up my VLANs and now this technically works. Technically, not optimally - because now I'm firewalling in the host-environment where my crownjewels are, instead of firewalling before even touching this hardware. But for now, I'm in business again.