Firewall suddenly blocking RPC traffic from a specific server
-
Hello all. First-timer here. Our pfSense box supports two subnets; one for regular office traffic, the other for backup jobs/traffic. The server in question is a Hyper-V host with two NICs; office subnet and backup subnet. I'm kicking off a Hyper-V backup job from a server in the backup subnet and the Hyper-V host is responding back on port 135, but pfSense is blocking it via the following rule: @3 block drop in log inet all label "Default deny rule IPv4". Backups have been working fine previous to this.
Nothing has changed in our environment except for a power outage early Tuesday morning. I've tried specifically adding a rule for the backup subnet to pass all traffic on any port from the Hyper-V host to the backup server, but it still gets blocked. Any ideas you may have are appreciated. I'm not even sure where the "Default deny rule IPv4" rule is coming into play here. -
I'd like to add that if I tell the backup server to contact the Hyper-V host via it's address on the office subnet, everything works fine.
-
I just found a workaround, but I'd really like to know why I need to do this. I created a floating rule to pass all protocols/ports traffic from the Hyper-V host to the backup server, statetype=none.
If anyone has an idea why this is happening, or where else I should look, I'd be thankful.
-
You should be able to simply add a rule on the incoming interface.
-
You should be able to simply add a rule on the incoming interface.
Thanks for your reply. Yep, I tried that, but it didn't help. It only started working after I created the floating rule. I'm going to try rebooting pfSense this evening and see if it makes any difference.
-
Could you post your interface-specific rule as well as your floating rule? If it's hitting the default deny rule, your specific rule on the interface is not actually matching the traffic.
-
Could you post your interface-specific rule as well as your floating rule? If it's hitting the default deny rule, your specific rule on the interface is not actually matching the traffic.
Sure.
-
Here's the floating rule I added that made everything start working again. The BackupSubnet is 192.168.2.0/24. The backup server is 192.168.2.2. The Hyper-V host I was trying to back up has an IP on the BackupSubnet of 192.168.2.3. I had to include statetype=none in the floating rule for it to work.
-
Interesting, that definitely should be working. Did you try with no state tracking on your interface rule as well?
-
Interesting, that definitely should be working. Did you try with no state tracking on your interface rule as well?
Yes, I did set no state tracking on the interface rule. So strange. I also found out this morning that the backup report email that the server sends out after backups are completed got blocked, and even the floating rule isn't allowing it through.
I do a nightly backup of the pfSense config, so I'm considering restoring the config from this past Sunday, before everything started blowing up. crosses fingers
-
Remember that you can also go back through your revisions via Backup/Restore -> Config History.
-
I restored our pfSense config from the previous Sunday, before things started hosing up after a power outage on Tuesday, 4/22. Unfortunately, this didn't fix the problem, so I tried resetting pfSense to the factory config, ran the basic setup wizards and then did another restore of Sunday's config. Still no joy, so I re-applied my floating rule band-aid and everything works again. Wondering if I should just totally re-install pfSense. Maybe something's corrupted in the file system? Our other Hyper-V host, Server-Host2, has an address on the backup subnet, 192.168.2.0/24, and is not getting blocked. It's only traffic from Server-Host1.
-
Have you had snort installed and active at some point? Could be a residual block that just hasn't gotten cleared yet.
-
What else do you have in the rule that you get the "a" flag on it?
So you mention this hyper-v that has interface in both lan and backuplan?
The BackupSubnet is 192.168.2.0/24
That floating rule makes no sense then – your saying souce 192.168.2.0/24 to dest in that same network?
What is the path of the traffic.. Your lan is the source? What is its network? What is the lan rules? And you have something else in the rule if your getting "a" flag on the rule - so what is that?
-
Have you had snort installed and active at some point? Could be a residual block that just hasn't gotten cleared yet.
Nope, no Snort. I did have pfBlocker installed, but I removed that as part of my troubleshooting process. Removing it didn't make the problem go away and I haven't bothered to re-install it. The only other package I have installed is OpenVPN Client Export Utility.
-
What else do you have in the rule that you get the "a" flag on it?
So you mention this hyper-v that has interface in both lan and backuplan?
The BackupSubnet is 192.168.2.0/24
That floating rule makes no sense then – your saying souce 192.168.2.0/24 to dest in that same network?
What is the path of the traffic.. Your lan is the source? What is its network? What is the lan rules? And you have something else in the rule if your getting "a" flag on the rule - so what is that?
Thanks for your reply, johnpoz. To your questions:
Yes, both of our Hyper-V servers, Server-Host1 and Server-Host2, have two NICs each in 192.168.1.0/24 and 192.168.2.0/24.
BackupSubnet is 192.168.2.0/24 as you stated. Server-Host1 is the problem child; 192.168.2.3. Server-Host-2, 192.168.2.4, is not getting blocked when talking to Server-Backup, 192.168.2.2.
Since I restored the earlier pfSense config, I had to re-establish the floating rule band-aid that fixes the problem with Server-Host1 being blocked when talking to Server-Backup. I made the rule simpler, per the following pic. The path of the traffic is BackupSubnet to BackupSubnet, and I've verified this by logging packets handled by the floating rule.
-
As it has been stated here - and why-so-ever in terms of what is blocking the original settings - the floating rule workaround works.
We had the same problem, out of the sudden, pfSense refused connections between our DCs which are connected via IPSec. It did work perfect with our ruleset and neither there was a change in terms of update nor in terms of SNORT or other configuration - at least not to our knowledge (and we own/administratet he systems).
Hence @micky dam, thank you for your work around…
Maybe I can get even some more strange surroundings into the topic:
In our case-
we are running pfSense on both ends, however, our local office is running it on real HW and the DC is running within Hyper-V (and, yes, it's not our favorite situation but not changeable by now).
-
while the Hyper-V pfSense now need the floating rules, our similar configuration in the HW firewall didn't… similar does mean except for HW and IP adresses, both systems have the 'same' rules (except for opposite directions), packages etc.
-
Timezone and Time is also same on both systems
Our pfSense version is btw 2.2.4
-
