Strange DNS behaviour
-
Apologies, this is a slightly longer post and I want to give sufficient information of all the things I have tried already to resolve
Issue/Symptom:
DNS resolution fails for some domain after a period of time has passed. Things work fine after a restart only to have the same issue with some domains(not all) failing to resolve.Analysis and setup:
I have done packet captures (can post it here if needed) when it is working fine and when it doesn't. The DNS query request seem to go out fine at all time to external DNS servers but there is no response (no packets received) for some queries once the issue starts to manifest.I use pfsense DNS forwarder going to Cloudflare DNS on ipv4 and ipv6
The network is dual stack
The system memory usage seems to creep up. Memory usage start around 24% of 8gb available and slowly creeps up and the issue starts to appear when the usage is about 53 or 54%
What is making this weird is that the DNS packets seem to go out, but there is response from the server initial after a restart and no response for some domains as time progresses. I have tried lot of other public DNS as well cloudflare, google, 9.9.9.9 and all have the same behaviour leading me to think there is something wrong with the pfsense box.
Any pointers to further investigate and resolve this issue will be appreciated. Thanks.
Edit: System information screenshot attached
-
@jkmuk said in Strange DNS behaviour:
DNS packets seem to go out, but there is response from the server
How is that a pfsense problem? If your sniffing on your wan and you see the query go out and don't get a response - how is that anything to do with pfsense. Pfsense has no control over if something answers or not out on the internet.
Can you even ping who your trying to talk to for dns, for example 8.8.8.8 or 9.9.9.9
-
@johnpoz Logical question, which I have been grappling with myself. However the same server responds after a restart of pfsense. There is something probably in the request that is corrupted some how?
-
-
@michmoor said in Strange DNS behaviour:
@jkmuk @johnpoz asked if you could ping any dns server by IP when the trouble happens..
Yes I can ping the DNS servers and they also respond to some of the DNS queries without issues, but some queries seem to fail. Those that fail are consistent. e.g. bbc.co.uk always works sunnyvalley.cloud always fails when the issue happens.
-
@jkmuk are you running ZenArmor ?
-
@michmoor Yes, indeed
-
@jkmuk disable it OR uninstall and try again. Or see if there’s an update to it via the web managed console.
I have seen this type of inconsistent behavior exactly but in my case there was an update required.
Note: in pfsense 2.7 3rd party packages that aren’t signed cannot be installed so Zen will not work. Consider that before you upgrade.
Also be very weary of installing packages like this which bring dependencies that could leave your firewall in an insecure state. -
Additionally is this something that's always happened in pfSense? Or have you been running 2.6 for some time and it's just now started to fail some queries?
-
@stephenw10
The recent change is the upgrade to 2.6.Following @michmoor recomendation I did disable Zenarmor and the issue disappeared. There seems to be some in-compatibility introduced in v2.6 , also given the change to only allow signed packages in future I doubt the Zenarmor team will do anything about it. Sadly, I have decided to move to Opnsense, the zenarmor package is stable on it as has been running for a few days.
I liked pfsense and have been using it for more than a decade now, but sadly have to say good bye. Wish there was someone that developed a native pfsense package that made filtering simpler. Perhaps netgate might offer something in the future for their plus customers.
-
@jkmuk how exactly does a package running on your box prevent an answer - thought you said the dns was going out, but no return - and this was via a sniff?? Your saying this zenarmor is before your sniff for filtering?
-
@johnpoz said in Strange DNS behaviour:
@jkmuk how exactly does a package running on your box prevent an answer - thought you said the dns was going out, but no return - and this was via a sniff?? Your saying this zenarmor is before your sniff for filtering?
Zenarmor uses the netmap kernel device for blocking traffic. I've not investigated the details, but it is possible the netmap connection gets in front of the PCAP packet capture connection when netmap is enabled.
Netmap creates shared memory buffers between the kernel network stack and the physical NIC driver. The software that opened the netmap connection is then solely responsible for forwarding traffic between the kernel and the physical NIC driver. It could be the netmap path drops the packet before it gets a chance to be seen by the PCAP process used for packet capturing.
-
@bmeeks I for sure wouldn't be a fan of that - when I sniff I should be be seeing whats actually going on the wire, or what is coming in on the wire - before any "filtering" of it could happen.
-
@johnpoz said in Strange DNS behaviour:
@bmeeks I for sure wouldn't be a fan of that - when I sniff I should be be seeing whats actually going on the wire, or what is coming in on the wire - before any "filtering" of it could happen.
I'm not 100% positive that is the case in this instance, but I can see how theoretically it could happen. Depends on exactly how the packet path is altered when both PCAP and netmap are in use at the same time.
