pfSense unreachable if rebooted

[ctuchik]

Hi, I have a very functional virtualized pfsense VM, I have a few VLANs, I'm using pfblockerng for DNSBL and IP blacklists, some NATS, and a wireguard VPN.

However, while looking at the console in my hypervisor all looks fine after a reboot - the network goes completely dead on the LAN side.
I'm able to administer the hypervisor via a static IP on the hypervisor and my laptop.
I can't ping or access the webui after a reboot, and there's certainly no DHCP.
I've tried bringing the firewall down through the console but still can't get in.
Looking at the connections, the VPN seems to be able to connect!

So what I'm on the hunt for are suggestions on some obvious gotchas?
It feels like it could be routing, but the route table looks okay too!
I am fortunate enough that I can simply restore from snapshot after a failed reboot, but the day will come where that won't be possible and I'm not sure where or how to troubleshoot this any more.

[ctuchik]

Maybe some routes and traceroutes will help?

The traceroute is to another host on LAN, no VLAN. It tries to route to WAN instead for that?

[stephenw10]

What is 10.1.1.10, vtnet0?

You have no subnet for that shown locally so it's trying to reach that device via the default gateway. But it looks like that's probably a local subnet given your other addresses.

What does ifconfig vtnet0 show

[ctuchik]

@stephenw10 10.1.1.10 is the LAN interface for the firewall/pfsense:

I have the machine running and am writing this message over it, but I will run ifconfig vtnet0 after work and a reboot.

I can show now (in it's working state) what that looks like:

[2.7.2-RELEASE][admin@ishtar.i]/root: ifconfig vtnet0
vtnet0: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        description: LAN
        options=800b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,LINKSTATE>
        ether a6:b3:9c:fa:9b:64
        inet 10.1.1.10 netmask 0xffffff00 broadcast 10.1.1.255
        inet 99.99.99.99 netmask 0xffffffff broadcast 99.99.99.99
        inet6 fe80::a4b3:9cff:fefa:9b64%vtnet0 prefixlen 64 scopeid 0x1
        media: Ethernet autoselect (10Gbase-T <full-duplex>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
[2.7.2-RELEASE][admin@ishtar.i]/root:

As well as the same from above in a working state:

[2.7.2-RELEASE][admin@ishtar.i]/root: traceroute -w 1 -m 5 10.1.1.20
traceroute to 10.1.1.20 (10.1.1.20), 5 hops max, 40 byte packets
 1  t (10.1.1.20)  0.827 ms  0.639 ms  0.618 ms
[2.7.2-RELEASE][admin@ishtar.i]/root:

[2.7.2-RELEASE][admin@ishtar.i]/root: netstat -rn -4
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            192.168.50.1       UGS      vtnet1
10.1.1.0/24        link#1             U        vtnet0
10.1.1.10          link#9             UHS         lo0
10.20.20.0/24      link#4             U        vtnet3
10.20.20.1         link#9             UHS         lo0
10.30.30.0/24      link#5             U        vtnet4
10.30.30.1         link#9             UHS         lo0
10.50.50.0/24      link#7             U        vtnet6
10.50.50.1         link#9             UHS         lo0
10.72.231.175      link#9             UH          lo0
99.99.99.99        link#9             UH          lo0
127.0.0.1          link#9             UH          lo0
192.168.50.0/24    link#2             U        vtnet1
192.168.50.10      link#9             UHS         lo0
[2.7.2-RELEASE][admin@ishtar.i]/root:

[stephenw10]

That gateway is wrong. It looks to have the same IP as the LAN interface which is a conflict.

You probably shouldn't have a gateway on LAN at all. You would only need one if you have some downstream router there with other subnets behind it. And you don't since they're not in the routing table.

Otherwise remove that conflicting gateway.

[ctuchik]

@stephenw10 Yeah - right on.

That sorted it man thank you, the way you explained it, made it obvious that was the problem :-)

I wonder how it functioned before with that logical hurdle?!

[stephenw10]

Good question! Probably some small change to the bootup order.