Update 2.7.2 to 2.8.0. Single compter lost pfsense connection
-
I suspect my pfsense configuration has become corrupted in updating pfsense but not sure how to localise / fix the issue.
The system worked fine prior to upgrading pfsense.
Symptoms
-
Stand alone windows PC (192.168.11.32) can ping all local hosts I tried including pfsense (192.168.11.1) and Proxmox (192.168.11.50)
-
Proxmox v8.4.1 (192.168.11.50) gets no response from pfsense (192.168.11.1) but can ping all other local host I have tried including the above PC.
-
pfsense v2.8.0 (192.168.11.1) gets no response from Proxmox (192.168.11.50) but can ping all other local host I have tried
In addition
-
Proxmox has lost internet access. It uses pfsense (192.168.11.1) as a gateway.
-
In pfsense Enabling "Log packet" on the pass ICMP rule -> shows ICMP from Proxmox (192.168.11.50) is passed when I attempt to ping pfsense (192.168.11.1)
Other information which I'm not sure is relevant
-
I updated pfsense v2.7.2 -> v2.8.0 without uninstalling the packages pfBlockerNG-dev, System_Patches, nmap.
-
I uses Proxmox as the local time server (Chrony is more accurate than FreeBSD native implementation). In pfsense I have port forwarded requests for other servers to Promox. I disabled the port forward to simply the debugging. Re enabling it gave and error (overlapping range) but later after uninstalling pfblocker the port forward could be re-enabled without causing an error. After reinstalling pfBlockerNG rule could still be enabled and disabled without error. The error was similar to described in this recent forum post How to redirect IPv4 and IPv6 NTP traffic
-
pfsense actually runs on a Proxmox VM with all NIC passed through. Proxmox runs on a MiniPC with 6 NIC (Intel i201/i211). As a result the data path failing is Proxmox Hypervisor (192.168.11.50 -> Linux Bridge -> Intel NIC on MiniPC -> Netgear JGS524PE programmable switch untagged VLAN 11 Port to trunk port -> Intel NIC on miniPC passed through to pfsense LAN NIC / VLAN 11 = OPT1 (Main)
Suggestions??
-
-
@Patch I’d start with Ping and DNS on the Diagnostics menu. Seems more like a routing or rule problem if you can get to the pfSense GUI.
-
@SteveITS said in Update 2.7.2 to 2.8.0. Single computer lost pfsense connection:
I’d start with Ping and DNS on the Diagnostics menu
Good call. Which is what I did to get:
@Patch said in Update 2.7.2 to 2.8.0. Single computer lost pfsense connection:
pfsense v2.8.0 (192.168.11.1) gets no response from Proxmox (192.168.11.50) but can ping all other local host I have tried
I have normal access to pfsense as well as the Proxmox GUI.
DNS lookups appear to function as normal. If there is an address of interest to you I can check that.
-
@Patch does pfSense have only one gateway? Try floating state policy? https://forum.netgate.com/topic/190658/firewall-state-policy-floating-states-needed-but-why/5
Traceroute from Proxmox?
Check routing table on pfSense?
-
I would also check the states in pfSense when running a ping from Proxmox.
Also the ARP tables on both. Though since you see the request passed in the logs ARP should be there.
Are those rules in your screenshot floating or applied to an interface group? I agree the interface state policy change seems more likely here.
-
@SteveITS said in Update 2.7.2 to 2.8.0. Single compter lost pfsense connection:
does pfSense have only one gateway?
Yes
@stephenw10 said in Update 2.7.2 to 2.8.0. Single compter lost pfsense connection:
Are those rules in your screenshot floating or applied to an interface group?
Rules shown are for the interface Main (opt1) -> igb1.11 -> v4: 192.168.11.1/24
The only Floating rule is disabled.
The only NAT rules on the Main interface are:- Enabled DNS on ! Main addresses redirected to 127.0.0.1
- Disabled NTP on !Main addresses redirected to then problem Proxmox computer
Given DNS communication to pfsense works (when pfsense had a working NTP and access works from other computer, suggests corrupted rules or similar to me but I'm not sure how to test or correct that theory.
@SteveITS said in Update 2.7.2 to 2.8.0. Single compter lost pfsense connection:
Traceroute from Proxmox?
root@pve1:~# traceroute www.google.com traceroute to www.google.com (142.250.70.196), 30 hops max, 60 byte packets 1 * * * 2 * * * 3 * * *In comparison another site with almost identical set up (Later MiniPC)
root@pve4:~# traceroute www.google.com traceroute to www.google.com (142.250.70.228), 30 hops max, 60 byte packets 1 192.168.11.1 (192.168.11.1) 0.259 ms 0.239 ms 0.226 ms 2 Deleted-Text.net (Deleted IP) 8.750 ms 9.100 ms 8.924 msRouting table on Proxmox (computer partly lost connection to)
root@pve1:~# ip r default via 192.168.11.1 dev vmbr0 proto kernel onlink 192.168.11.0/24 dev vmbr0 proto kernel scope link src 192.168.11.50Which is very similar to the working site other than host IP
root@pve4:~# ip r default via 192.168.11.1 dev vmbr0 proto kernel onlink 192.168.11.0/24 dev vmbr0 proto kernel scope link src 192.168.11.54 root@pve4:~#On pfsense arp -a
includespve1.work.arpa (192.168.11.50) at 5a:07:9b:13:b1:09 on igb1.11 permanent [vlan]On Proxmox it appear the net-tools package is not installed
-
That arp entry implies the IP address is used by a local interface in pfSense dircetly. So it looks like you have an IP conflict. It ignores traffic from that address from Proxmox. A VIP perhaps?
-
@stephenw10 Now you are really confusing me
The only VIP I have is the entry created by pfBlockerNGI had assumed the above arp entry was from
pfsense -> Services -> DHCP Server -> Main (which is on igb1.11) -> includes
For completeness Routing table on pfsense but nothing stood out to me
netstat -rn Routing tables Internet: Destination Gateway Flags Netif Expire 0.0.0.0 redacted.1 UGS igb0 10.10.10.1 link#6 UH lo0 redacted.0/22 link#1 U igb0 redacted.1 link#1 UHS igb0 redacted.255 link#6 UHS lo0 127.0.0.1 link#6 UH lo0 192.168.1.0/24 link#3 U igb2 192.168.1.1 link#6 UHS lo0 192.168.10.0/24 link#2 U igb1 192.168.10.1 link#6 UHS lo0 192.168.11.0/24 link#9 U igb1.11 192.168.11.1 link#6 UHS lo0 192.168.12.0/24 link#10 U igb1.12 192.168.12.1 link#6 UHS lo0 192.168.30.0/24 link#4 U igb3 192.168.30.1 link#6 UHS lo0 AlsoRedacted.5 redacted.1 UGHS igb0 Internet6: Destination Gateway Flags Netif Expire ::1 link#6 UHS lo0 fe80::%igb0/64 link#1 U igb0 fe80::2f4:21ff:fe68:274f%lo0 link#6 UHS lo0 fe80::%igb1/64 link#2 U igb1 fe80::1:1%lo0 link#6 UHS lo0 fe80::2f4:21ff:fe68:2754%lo0 link#6 UHS lo0 fe80::%igb2/64 link#3 U igb2 fe80::2f4:21ff:fe68:2753%lo0 link#6 UHS lo0 fe80::%igb3/64 link#4 U igb3 fe80::2f4:21ff:fe68:2752%lo0 link#6 UHS lo0 fe80::%lo0/64 link#6 U lo0 fe80::1%lo0 link#6 UHS lo0 fe80::%igb1.11/64 link#9 U igb1.11 fe80::2f4:21ff:fe68:2754%lo0 link#6 UHS lo0 fe80::%igb1.12/64 link#10 U igb1.12 fe80::2f4:21ff:fe68:2754%lo0 link#6 UHS lo0 -
Ah, you have added the static DHCP entry as static ARP also?
That's almost always a bad idea IMO! But if that MAC address is correct then it's not the cause of the problem here.
If you run a ping from proxmox to pfSense what states are created in the pfSense state table?
Try running a pcap in pfSense for that. Is the source MAC what you expect it to be?
-
@stephenw10 said in Update 2.7.2 to 2.8.0. Single compter lost pfsense connection:
Ah, you have added the static DHCP entry as static ARP also?
That's almost always a bad idea IMO!
Mmm, so there is a better way I have missed.
What is the recommended way of setting client computers IP address from pfsense?@stephenw10 said in Update 2.7.2 to 2.8.0. Single compter lost pfsense connection:
But if that MAC address is correct then it's not the cause of the problem here.
Oops Bingo
Looks like I had an old incorrect MAC.
Proxmox GUI requires a static IP so two MAC addresses for one IP is not good.
Updated the MAC in pfsense DHCP rectified the communication fault.Thank you for your help in correcting my error.
-
You can set a static DHCP lease but not set static ARP. Normally pfSense would just ARP for the IP and update the MAC in the table but if you mark it static things like this can happen.
