Super strange NAT problem that occurred for no reason at random
-
Yesterday at around 2PM I got a call from one of our offices saying they are no longer able to receive inbound calls. Our system is a Mitel Office 250 that has been working fine for the past several years with no issues on a pfSense firewall made from a Dell Optiplex. None of the rules have changed in the firewall, our SIP provider confirmed it is our firewall blocking their traffic and I'm out of answers. I have tried rekeying all the rules from scratch, including the IP aliases (which I have confirmed several times match our providers suggest list of IPs to allow) and changed how the NAT rule is configured, even just enabling the "pass" option in the rule creation field and attempted to manually create a matching rule myself. All packets that are supposed to be allowed are being blocked by default deny rule. I'm about at the point where I'm going to reflash pfSense or replace the unit entirely. Screens below, any suggestions? Thanks a lot!
-
@uncleasbestos First thing I would try it to take the aliases out of the picture entirely. Rejig your rules to allow those explicit IPs instead of the alias. Post a redacted screen of your WAN rules. Are you using pfBlocker, or an IDS like Snort/Suricata? If yes to any, temporarily disable them.
-
fact-check the ISP (and any equipment upstream) : Go here : DiagnosticsPacket Capture
and fill in the greens :
Instead of "1234" use your SIP destination port, and hit go ...
As soon as SIP traffic traffic arrives at the WAN interface, you will see an output on the packet capture page. This is the "10 seconds check incoming NAT traffic" test.
If nothing arrives at the pfSense WAN then ... undo everything, pfSense is fine, the issue is upstream.The 5 seconds test is even simpler :
Look at your WAN firewall rules :
See the green counter in front of them ? That means the rules have found a match and are getting 'used'. If the counters stay zero, then they didn't match .... and finally the default firewall action (rule) kicks in : the "Default deny rule IPv4" message gets logged.
-
That SIP traffic should be passed there if the source IP matches the hosts in the alias. But we can't see that in the redacted screenshots.
Since it's not matching and hitting the default rule I have to assume the source IPs do not match the alias.
-
Hey, I really appreciate all the responses. pfBlocker and Snort are disabled in the sample I uploaded. I ran a packet capture as suggested and the SIP requests were definitely getting through to the firewall and being blocked by it. Also, no states were being created and the traffic counter stayed at 0 the whole time. I double checked the aliases and they were correct, but I took the step suggested above and rekeyed the rules using the actual addresses instead of an alias, traffic passed successfully.
I have no idea at all why the alias would all of a sudden stop working correctly as I've never had that happen, but I'm glad we figured it out regardless.
Thank you everyone and may your status lights stay bright and green.
UncleAsbestos
-
Also, thank you for being kind in your responses.
-
@uncleasbestos Happy to help. Glad you got it working.
-
@uncleasbestos said in Super strange NAT problem that occurred for no reason at random:
I double checked the aliases and they were correct
So it looks like the firewall using rules with aliases, doesn't match these alases.
First, a less know secret : a firewall doesn't work or understand hosts names or aliases. A firewall can only work with IP addresses, when it concerns a source or destination.
Have a look at this file : /tmp/rules.debug.
This is the actual file, a list woth rules instructions, loaded into 'pf' the pfSense firewall.
Search for BCM_SIP.
You should find this :.... table <BCM_SIP> { a.b.c.d ...... } BCM_SIP= "<BCM_SIP>" ....The a.b.c.d is (are) the IP addresses that were resolved from teh alias, if the alias contained a host name.
When I create a alias called :
Note : you saw :
FQDN hostnames are periodically re-resolved and updated. If multiple IPs are returned by a DNS query, all are used
and use it in a WAN firewall rule like :
I can find this in my /tmp/rules.debug :
.... table <BCM_SIP> { 1.2.3.4 } BCM_SIP = "<BCM_SIP>" ....if the "BCM_SIP" was a FQDN, or something like "www.whatever-fqdn.tld" then this hos name is resolved every 5 minutes or so (see system settings for the frequency) and the resolved IP is entered into the firewall rule table.
Read : Alias Features and Limitations as it mentions important info.
There are still people out there that make an alias with 'www.facebook.com' and then find out it doesn't work as intended
Anyway, now you have even more tools to check.
