Upgrade from pfSense CE from 2.7.2 to 2.8.1 breaks most Synology services and packages

homebrewuser

Hi all, somewhat IT/network fluent home user here who’s been using pfSense CE on a Protectli FW2B box for a few years now, in an extremely stable environment.

This is the 2nd time I’ve tried to upgrade from 2.7.2 to 2.8.x. The first time I tried it (whenever 2.8.0 came out) it did the same thing and I got frustrated and rolled the hardware back to 2.7.2 and have been living peacefully since then. When 2.8.2 came out, I thought I’d try again…not for any specific reason, just that I felt like I should generally be running the latest OS to ensure I’m fully up-to-date.

I have made no changes to my environment as part of the upgrade.

As soon as the upgraded completed, my Synology DS224+ (running latest DSM) experiences the following symptoms:

The NAS continues to function properly as a file server on my LAN, and all LAN hosts can access it.
Surveillance Station continues to function properly, and my cellphone app can access it via the LAN.
Cannot ping the NAS from pfSense (but can ping all other LAN hosts).
Synology Account will not connect.
NTP will not sync.
Package Center will not connect to show available packages.
Plex Media Server will not connect.

My environment feels pretty simple:

Router/Firewall is connected to L3 Switch on LAN interface and ISP gateway on WAN interface. All LAN/VLAN hosts are connected via the switch.
Router/Firewall
2a. Running pfSense at 192.168.0.1
2b. LAN is 192.168.0.1/24. All hosts except one are on this subnet.
2c. VLAN10 is 192.168.10.1/24. Only 1 client on this VLAN, restricted to Internet use only; purposely restricted from accessing 192.168.0.1 subnet.
2d. DNS Resolver running
2e. Snort is running in Legacy Mode on LAN and Inline IPS Mode on WAN. Paid snort ruleset and free ET ruleset in use.
2f. pfBlockerNG is running. I have a variety of free feeds activated.
2g. I’ve added all the recommended Synology sites in pfBlocker NG DNSBL Whitelist
Both NAS Ethernet ports are bonded together in DSM Network/Network Interface (each at 1Gb/s) and with LACP on the switch.

I’m pulling my hair out and would prefer not to rollback to 2.7.2 again if possible. Any thoughts welcome, thanks in advance!

tinfoilmatt

So post-upgrade, the entire network is stable and with good internet connectivity—except for the Synology NAS?

What kind of firewall rules are in place specific to the NAS?

Anything relevant in firewall logs? Snort alerts? pfBlockerNG alerts?

Have you rebooted the NAS following the upgrade? Power-cycled the switch?

When you say you can't ping it from pfSense, that makes me think of two follow-up questions: 1.) But you can successfully ping it from other network hosts? And 2.) although it should be able to unless intentionally filtered, why does it matter if pfSense can ping it? Is that just the primary diagnostic you're relying on to tell you there's a connectivity issue?

homebrewuser

@tinfoilmatt, yes that's correct. Everything working as before the upgrade except the noted NAS symptoms.

I don't think I have any NAS-specific FW rules (it's 192.168.0.9). Here's the WAN rules:

Here's the LAN rules:

I have rebooted the NAS and the switch, with no change.

The filtered System Firewall Log shows only:

...but that doesn't show up consistently when I try to re-sync from the NAS.

Nothing in the snort WAN or LAN alerts. There's a bunch of different pfBlockerNG logs, I've looked through the ones that imply blocked/dropped connections and couldn't find any entries associated with the NAS IP address...in fact most of the logs only seem to include lists of IP addresses.

Yes, I can ping the NAS server from other LAN hosts and it continues to function properly as a file server. I was reading through similar support threads on the web, and someone asked a similar question and it seemed like a relevant troubleshooting step so I tried it just to see. While I can't ping the NAS from the FW utility, I can ping all other LAN hosts from it.

I wonder if the NAS needs to be tagged in VLAN0?

tinfoilmatt

Does the ISP device connected to WAN interface provide a firewall?

homebrewuser

@tinfoilmatt , it's set to bridge mode...so even though it offers one when used as a router, it's not possible for it to be active currently is it?

johnpoz

@homebrewuser said in Upgrade from pfSense CE from 2.7.2 to 2.8.1 breaks most Synology services and packages:

so even though it offers one when used as a router

so you have a any any rule on your public wan of pfsense.. Yeah not a good idea that is for sure.

What are you trying to sync from to the nas - pfsense has nothing to do with conversations between devices on this 192.168.0 network talking to each other. Unless you have a bridge setup on pfsense where some devices would be on different sides of the bridge you have on pfsense.

homebrewuser

@johnpoz no bridge set up on pfsense that I know of. I'm trying to get the NAS to sync its services with the various synology.com sites it has to connect to for Package Center, NTP, etc.

johnpoz

@homebrewuser so your wanting to talk to something on the internet.

I have not gone through all your rules - you have a lot.. And not sure what your trying to talk to for port, etc. Or ip out on the internet.

Your last rule there is an any for .4, why not make a rule a rule for your .9 - log it, so you know what its talking to and then you can tweak your rules.

But those blocks to broadcast 255.255.255.255 on port 65001 sure wouldn't be anything out on the internet that is for damn sure, and pfsense not going to do anything with such traffic. Only thing that would have anything to do with such traffic would be other things on that same 192.168.0 network.

65001 is used by hdhomerun, which also uses these ports

Discovery Protocol - UDP 65001
Control Protocol - TCP 65001
Video stream - UDP 5002 and UDP 5004
LIRC for IR on HD Homerun - UDP 5000

stephenw10

I assume the Synology also can't ping pfSense?

Do you have anything setup with static ARP?

Do you have any floating firewall rules? Port forwards?

One of the biggest changes in 2.8.0 was the switch from floating firewall states to interface bound. That could cause connectivity issues for anything relying on an asymmetric connection. However it's hard to see how that could be the case here unless the Synology has interfaces in both subnets perhaps?

homebrewuser

@stephenw10 thanks, the NAS can ping all hosts on the LAN except the FW. I haven't set up any static ARP. I don't have any port forwards in place, but did try creating one based on the known port that Plex Media Server uses...without any luck, so I removed it.

here's the only floating rule in place:

I don't believe the NAS has an interface in the .10 subnet (VLAN10).

I wonder if I should try to config the NAS to be in VLAN1? But it worked just fine before the upgrade without being configured as part of a VLAN.

homebrewuser

@johnpoz I copied the .4 rule and changed to .9 but no luck.

I do have a hdhomerun unit on the LAN (used by Plex for live broadcast TV), so should I permit that broadcast traffic?

johnpoz

@homebrewuser said in Upgrade from pfSense CE from 2.7.2 to 2.8.1 breaks most Synology services and packages:

so should I permit that broadcast traffic?

Pointless if you allow it or block - pfsense isn't going to do anything with it. And pfsense has no control what other devices on that same network are going to see when talking to each other be it unicast, broadcast or multicast. Only reason you might want a rule that does something with it so it's not logged. I personally don't log default deny. I have specific rules to deny and log what I am interested in.. tcp syn hits, and specific common udp ports.

This keeps my log spam down.. If need to for troubleshooting, can re-enable default deny logging with a click of a button.

As to your issue, you might want to remove one of the connections on your bond you setup in your nas.. Just use 1 interface on your nas.. You can try just unplugging a connection, if still having issue break the bond you setup. Unless you are on a work sort of network with lots and lots of clients talking to the nas, that bond gets you nothing other than complication.. Guess it could get you redundancy on a failed port.. But its not making your connection to your nas from a client faster.

What exactly are you connected to with the nas - a switch, pfsense interfaces directly.. This 192.168.0 is that setup as a vlan in pfsense - or is it just your native (untagged) lan network.. Are you doing any sort of vlaning on switch your pfsense lan interface is connected to. Do you have any sort of bond/lacp setup on pfsense - ie multiple interface on pfsense for the "lan" network?

tinfoilmatt

As to your issue, you might want to remove one of the connections on your bond you setup in your nas.. Just use 1 interface on your nas.. You can try just unplugging a connection, if still having issue break the bond you setup. Unless you are on a work sort of network with lots and lots of clients talking to the nas, that bond gets you nothing other than complication.. Guess it could get you redundancy on a failed port.. But its not making your connection to your nas from a client faster.

This. And close your firewall. In reverse order of importance.

homebrewuser

@johnpoz I unplugged the Lan2 NAS connection and broke the bond in its config and kept same static IP (it was using .9 and .10 in the bond). Also removed the LACP config from my core switch. No change in symptoms...no ping from FW to NAS or from NAS to FW.

The NAS is connected to my core switch (older HP1910 PoE). Same switch my FW is directly connected to on its LAN interface. 192.168.0 is not set up as a VLAN in pfsense, only VLAN10 is identified as a VLAN there.

VLAN10 is also set up on the core switch, but only on the (trunked?) ports where the FW is connected and the port for a sub-fed switch (VLAN10 is only tagged on one port on the sub-switch, the other ports are untagged).

I do not have any interfaces setup in pfSense other than WAN, LAN, and VLAN10. In the attached pic, none of the other sub-menus have any entries other than the VLAN tab for VLAN10:

homebrewuser

@tinfoilmatt I have tried shutting down the fw previously, but run into challenges, no doubt because I didn't do it correctly. Any suggestions on correct procedure most welcome. I tried closing processes from the 'Services Status' widget on the Dashboard, but got a number of errors.

Should snort go first, or pfBlockerNG? Or something else?

tinfoilmatt

Downstream switch won't be relevant to anything.

Is that "ecobee" rule the final rule in your LAN ruleset? Or are there any others beneath it?

tinfoilmatt

You basically need to just delete all three rules you've configured, and that will 'close' the firewall completely.

homebrewuser

it was the last one, I shrank the screen so I could fit all of them

johnpoz

@homebrewuser so here is what I would do - look in the nas arp table.. do you see an arp for pfsense IP, ie your gateway? If not then no your never going to be able to talk to pfsense IP on the 192.168.0/24 network.

Try a ping to pfsense IP 192.168.0.1 right? Now look in your nas arp table

ash-4.4# arp -a
? (172.25.0.2) at 02:42:ac:19:00:02 [ether] on docker-97cefc8c
? (192.168.10.9) at 34:c8:d6:b4:01:55 [ether] on ovs_eth2
? (172.26.0.2) at 02:42:ac:1a:00:02 [ether] on docker-d646a7fd
i9-win.home.arpa (192.168.9.100) at b0:4f:13:0b:fd:16 [ether] on ovs_eth0
? (172.17.0.5) at 02:42:ac:11:00:05 [ether] on docker0
sg300-28.home.arpa (192.168.9.99) at 70:6e:6d:f3:11:93 [ether] on ovs_eth0
sg300-10.home.arpa (192.168.9.98) at c0:7b:bc:65:4f:13 [ether] on ovs_eth0
? (172.17.0.3) at 02:42:ac:11:00:03 [ether] on docker0
sg250-08.home.arpa (192.168.9.97) at a4:9b:cd:f6:58:09 [ether] on ovs_eth0
? (172.17.0.2) at 02:42:ac:11:00:02 [ether] on docker0
sg4860.home.arpa (192.168.9.253) at 00:08:a2:0c:e6:24 [ether] on ovs_eth0
ash-4.4#

So 192.168.9.253 is pfsense for me on my lan 192.168.9.0/24

If your nas does not get back the mac or its wrong - then no its never going to work... If its correct and you can not ping pfsense IP - then its a firewall rule on pfsense most likely. Or maybe static arp issue on pfsense.

One of the first steps in troubleshooting devices that can not talk to each other - is validate they correctly arp for the other device IP. Do you have any sort of private vlan setup on your switches?

Look in pfsense arp table do you see your nases correct mac for 192.168.0.9

tinfoilmatt

pfSense and the NAS should both be able to ping each other. But it feels like it should be troubleshooted from the 'NAS not being able to ping its gateway' angle, not the other way around.

Obviously if the NAS can't ping its Internet gateway (and ICMP isn't intentionally filtered on the LAN), then it probably doesn't have Internet connectivity. I'm assuming you won't be able to ping an external host, like 8.8.8.8, from the NAS either.