Two firewalls: pfSense blocks name resolutions for my second FW (IPfire)?

Maba79

Hi guys,
I'm a keen security/network hobbyist and always interested to try out different FW/router distros.

My home network is pretty decent size and my network edge consists of the following devices:
(Internet) -> cable modem/bridge (stripped out of all the other features) -> first firewall: pfSense -> second (upcoming) firewall: e.g. IPfire (haven't decided yet) -> router/WAP: DD-WRT (with LAN DHCP) -> switch -> followed by rest of the LAN devices (SIEM, servers, desktops etc.).

I just bought new hardware (Protectli FW2B) as my second firewall and I would like to place it right after my first FW (pfSense). I've been trying to install various FW distros into it and I get them all installed fine, but I encounter the same problem every time when trying them out:
My DNS gets broken in my new FW and it simply can't make any name resolutions. Pinging external IP addresses (e.g. 8.8.8.8) works fine and all services in my LAN which doesn't require any DNS resolutions works fine.

I've set static IPs with /29 bit mask between my pfSense and the new (IPfire) FW. No DHCP service between these two FWs and I haven't configured anything else in my new IPfire FW. I can make name resolutions from pfSense, but not with any other device behind it (e.g. from my freshly installed IPfire). Therefore I believe the culprit must be my pfSense FW which seems to somehow block or not forward any name resolutions.

I've tried various solutions e.g. I tried to make DNS NAT rule (port 53) from pfSense to IPfire, tried setting up various third-party resolvers in IPfire (e.g. 9.9.9.9, 1.1.1.1 etc.), tried enabling DNSsec in both firewalls, tried enabling "DNS forwarder" from pfSense (which I'm not even sure how it works), tried disabling/enabling DNS resolver from pfSense (in hope that my IPfire would start resolving names) etc. etc.

Nothing has worked and I just can't figure out what to do next. To me it sounds like the my pfSense is somehow blocking all the name resolutions but it's very puzzling, because this blocking happens only after I install my second FW after pfSense. I have had my DD-WRT router behind my pfSense for years without any problems (BTW: However, if I install DD-WRT in my new Protectl HW as a second FW/router, then this new DD-WRT faces the same DNS problem).

I'm desperate for any help. Any suggestions?? I'm not a DNS guru but I'm eager to learn. Thanks a lot!!

bmeeks

I assume you have your second firewall configured for some kind of NAT perhaps? Or is the second firewall acting as a pure router?

One thing that may be causing you trouble is the Access Permissions List used by the unbound DNS server on pfSense. By default only local addresses on the pfSense firewall (subnets, actually) that unbound knows about can make DNS queries of unbound. If you have networks (IP addresses) behind the other firewall whose values are not listed in the Access Control List for unbound, then any DNS requests from those clients will be ignored.

Maba79

@bmeeks Wow, awesome!! Thanks for the tip! I'll try to check this out. It'd be cool if you could point out where to fiddle this Access Permission List in GUI?

bmeeks

@Maba79 said in Two firewalls: pfSense blocks name resolutions for my second FW (IPfire)?:

@bmeeks Wow, awesome!! Thanks for the tip! I'll try to check this out. It'd be cool if you could point out where to fiddle this Access Permission List in GUI?

You would want to create a new one if you are using IP addresses that are not defined on pfSense. You can access it under SERVICES > DNS RESOLVER and then click Access Lists at the top of the DNS Resolver Settings page. Here is a link to the official pfSense documentation for this setting: https://docs.netgate.com/pfsense/en/latest/book/services/dns-resolver-acls.html.

Maba79

Excellent, thanks a lot friend!! I'll definitely check this one out.

akuma1x

Just curious, but why would you run 2 different routers/firewalls on the same network?

Maba79

@bmeeks I tried creating DNS Access Permission List where I allowed both subnets: subnet between the two firewalls and also the actual LAN subnet which is behind the second firewall. Unfortunately stil no luck. I also tried enabling/disabling the DNS forwarder, rebooting pfSense and setting up DNS NAT rule (port 53) from pfSense to IPfire. Nothing works and I'm all out of ideas.

Has anyone else managed to get IPfire (or any other FW) resolve names when it's behind pfSense?

bmeeks

@Maba79 said in Two firewalls: pfSense blocks name resolutions for my second FW (IPfire)?:

@bmeeks I tried creating DNS Access Permission List where I allowed both subnets: subnet between the two firewalls and also the actual LAN subnet which is behind the second firewall. Unfortunately stil no luck. I also tried enabling/disabling the DNS forwarder, rebooting pfSense and setting up DNS NAT rule (port 53) from pfSense to IPfire. Nothing works and I'm all out of ideas.

Has anyone else managed to get IPfire (or any other FW) resolve names when it's behind pfSense?

How do you have the IPFire device configured in terms of DNS? Is anything set there? Does it perhaps force some kind of DNS redirect to a non-existent server IP? If ping traffic to outside works from clients behind the IPFire, that would indicate basic routing is working. So the next step is to see if DNS requests are actually getting to pfSense.

Sniff the LAN interface on pfSense and see if you see arriving DNS requests from clients behind the IPFire device. If you do, then look in the DNS Resolver logs on pfSense and see if it is logging the request or what. You can enable verbose logging in the DNS Resolver. Use the previous Netgate documentation link I provided and backtrack from there to find the other DNS Resolver settings.

pfSense DNS operation just works out of the box now, but a lot of folks think they know a better way (or just don't read the docs) and proceed to immediately make changes. That frequently breaks it. Out of the box, you don't need to type anything anywhere for DNS to work. Don't enter any DNS server IP addresses anyplace during the pfSense set up process. The only reason to break this rule is when there is something upstream that will hinder the ability of the DNS Resolver to directly contact the root servers. Such a thing would be an upstream DNS redirect, for example.

Maba79

Thanks a lot for replying @bmeeks!

I haven't yet really fiddled with IPFire much, so it has pretty much all the default out-of-the-box settings. Only settings I've configured are setting the two name servers (9.9.9.9 and 1.1.1.1), which I've tried enabling and disabling in hope it would solve the issue. Problem with name resolution occurs no matter what secondary FW I place after pfSense, so therefore I assume that pfSense must the culprit.

NOTICE: My IPFire has currently automatically fired up a "recursor mode" in its DNS service. IPFire will change into recursor mode If there are no (working) DNS servers configured. In this mode, it will contact the root servers to resolve queries and recursively work its way down the DNS tree without using any third-party resolvers. This is realized currently in such way, that I'm able to resolve some names, but some (for some reason) others fail.

Here's my pfSense DNS settings:
I've set my pfSense's DNS server as 9.9.9.9 for DNS resolution, no additional DNS servers are set, following DNS setting are enabled: DNS resolver is enabled, DNSSEC is enabled and harden DNSSEC Data is enabled. Rest of the DNS settings are default (meaning disabled).

I enabled more verbose logging in my pfSense DNS resolver (Level 3: query level information), hooked up my Ubuntu laptop into my LAN interface back of my IPFire box and initiated packet capture via pfSense. I checked the results via Wireshark and it showed that some name resolutions did succeed, but others failed (e.g. http://dnssec-failed.org/ fails). I also saw few following DNS resolver log messages from pfSense system logs: "info: query response was ANSWER". To me this looks like name resolving was succesful. What kind of messages should I be looking for exactly?

I am no DNS expert. Could this be perhaps a DNSSEC problem because http://dnssec-failed.org/ fails? I can for example resolve pakfire.ipfire.org which is the address where my IPFire downloads the FW add-ons, but my IPFire can't update the add-on list and install any packages. I can't really cope with such an unstable DNS in my network. Name resolution has to work 100%, of course.

I haven't actually made much of DNS changes in pfSense when I first started using it. I only made sure that either Quad Nine (9.9.9.9) or Cloudfare (1.1.1.1) were my name resolvers and that DNSSEC was on.

If this starts to look too complicated issue, then I might consider changing my FWs order and put my IPFire in front and pfSense after IPFire and see if any name resolution errors occur then.

bmeeks

Out of the box pfSense allows any LAN traffic to go out to the Internet. That includes DNS traffic. And on the WAN, the out-of-the-box configuration allows all outbound traffic to pass (and establish states for any expected return traffic) while all unsolicited inbound traffic on the WAN is blocked. So an out-of-the-box pfSense configuration will not interfere with your IPFire box at all. It would appear as any other LAN client.

The only way the IPFire box would do anything with pfSense DNS (according to what you said are the out-of-the-box IPFire settings) is if the IPFire box is actually using DHCP on its WAN and it is allowing the DNS server specified by the pfSense DHCP server to override any other setting.

If you have altered the pfSense automatic NAT rules (which you mentioned earlier about "creating a port 53 DNS NAT rule" (not sure what you are actually describing there), then perhaps pfSense is intercepting port 53 traffic and causing grief. That would be a non-standard configuration and is not a normal out-of-the-box setup. Also, DNS uses both UDP and TCP, so if you set up a rule for just one protocol, that can cause issues as well.

Restore pfSense to a true out-of-the-box configuration by removing all rules (including any NAT rules you created) and test again.

From your initial post, it really sounds like you went and messed with the out-of-the-box DNS settings on pfSense. That is likely to be the cause of your grief now. You don't need any kind of NAT rules for DNS in a standard setup. And like I stated, you don't need to put any DNS servers anywhere in pfSense. It is ready to be a true resolver out of the box with no user action required at all. But if you made changes, then all bets are off. Hence my suggestion to return to a factory default setup and wipe out all rules you created and remove all DNS server IP addresses you entered.

Maba79

Thanks a lot for replying yet again @bmeeks!

My IPFire has now been in recursor mode behind pfSense for few days. In this mode, it will contact the root servers to resolve queries and recursively work its way down the DNS tree without using any third-party resolvers.

So far it seems that all name resolutions are successful for devices behind my two firewalls (pfSense and IPFire) in my LAN, but while I'm logged in IPFire all the name resolutions fail in IPFire. Interestingly IPFire itself is unable to resolve any names, but seems to successfully forward name resolutions to devices behind it (LAN). This unfortunately results in misbehaving IPFire box which can't e.g. update its packet list etc. Also IPFire is unable to execute any reverse lookups for any name servers I try to configure for it for testing purposes (e.g. 8.8.8.8, 1.1.1.1. etc.). Out-of-the-box and on front of pfSense IPFire is able to resolve names fine, therefore I'm still thinking pfSense is somehow the culprit.

From the beginning I have configured a static IP address with /29 bit mask in my IPFire's WAN interface (WAN interface is connected to my pfSense). DHCP is disabled in this interface on both sides (on IPFire and pfSense side).

The Port Forwarding DNS NAT rule (with both UDP/TCP traffic) I previously created was only for testing purposes. I have had it disabled since I noticed it didn't make any difference. I have now also deleted it.

I have also recently learned that the default behavior of the pfSense DNS Resolver will ignore manually configured DNS servers for client queries and query root DNS servers directly and to use the manually configured DNS servers for client queries, you'd have to visit Services > DNS Resolver and enable DNS Query Forwarding after completing the wizard. This is now something I have executed and the reason is following:

I have configured in pfSense's GUI "General settings" two DNS servers (first 9.9.9.9 and second 1.1.1.1) and enabled the DNS forwarder in pfSense (see previous explanation), because I wish to gain benefit from various Quad Nine (9.9.9.9) DNS services (e.g. anti-malware and DNS filtering). They are more beneficial resolvers than e.g. my ISP. If I delete them, then I believe my pfSense would also fall into recursor mode and contact the root DNS servers to resolve queries and recursively work its way down the DNS tree without using any third-party resolvers. Please correct me if you know better. I'm sure setting up pfSense without any third party DNS servers would be enough for many users, but like I said I'd like to benefit these additional benefits such third party resolvers can offer.

As all my LAN devices are now able to make name resolution except IPFire, then I guess I have few options left:

I can try to ask help from IPFire forum (though as I have explained earlier, I still assume the culprit is pfSense).
I could move IPFire in front of pfSense and see how my LAN and name resolutions would function then.
I could disable either of the FWs.

Maba79

For some reason I'm unable to edit my previous post so I just continue here.

For some reason it currently seems that I can now do successful name resolutions from only one device (Ubuntu 18.04) in my LAN behind my two firewalls (pfSense and IPFire). All the name resolutions are now failing from IPFire itself or from any other device in my LAN. This particular Ubuntu device has a static IP settings, but so has many other devices in my LAN also. I find this strange, because I haven't made any changes to DNS settings. IPFire itself is still unable to resolve any names, but seems to successfully forward name resolutions to this particular one Ubuntu device behind it. I haven't figured out why this particular Linux device succeeds when the rest of the devices fail (I have bunch of Windows, Linux and BSD machines in my network).

Oh well, this issue has turned more complicated than I expected and maybe it's now time to give up and find other ways to implement these two firewalls (e.g changing their order). I would still like to thank you for your help @bmeeks.