Weird issue with DNS queries generated from pFsense and blocked by pFsense

[FECambot]

Hi all,

Context:
my company installed a pFsense to shield an internal web server accessible via internet, using HAProxy.
It runs smoothly, nothing fancy or exotic done/configured.

Issue:
Yesterday we saw in the dashboard GUI that Firewall was blocking queries originating from our local DNS and going to 45.125.66.17 or 45.125.66.11, port 53. Volume was by the thousand ...

After a freaking out stage, during which the web and the DNS servesr were subject to X-ray exam, i had an idea and changed the DNS server config of pFsense from the IP of our local DNS server to 8.8.8.8 .

Some packets were still issued/blocked for a short time (maybe just report lag), and as of now no more packets issued/blocked.

If my understanding is correct, a process in pFsense did these queries ? but which one ?
The blocked remote IP is the IP of a domaine name:
mail.serveroffer.lt

Absolutely no system is using internally pFsense as a gateway.

I'll take any hints or advices :)

[Gertjan]

@FECambot said in Weird issue with DNS queries generated from pFsense and blocked by pFsense:

Yesterday we saw in the dashboard GUI that Firewall was blocking queries originating from our local DNS and going to 45.125.66.17 or 45.125.66.11, port 53. Volume was by the thousand ...

After a freaking out stage, during which the web and the DNS servesr were subject to X-ray exam, i had an idea and changed the DNS server config of pFsense from the IP of our local DNS server to 8.8.8.8 .

Lets change (add) some items to the tactics ^^
Let me test these IPs :

root@ns311465:~# host 45.125.66.17
Host 17.66.125.45.in-addr.arpa not found: 2(SERVFAIL)

.....
Nothing, unknown. have to nmap it to find out something'.

root@ns311465:~# host 45.125.66.11
11.66.125.45.in-addr.arpa domain name pointer mail.serveroffer.lt.

Aha.
This https://serveroffer.lt/ is shows its a place where you can rent servers .... take note that is the IP of their web server, and their mail server btw. They probably don't allow this IP to be used by their clients. Only their commercial web site. And as they sell IT stuff, not pan cakes, they probably know what they are doing.

root@ns311465:~# host serveroffer.lt
serveroffer.lt has address 45.125.66.11
serveroffer.lt mail is handled by 10 mail.serveroffer.lt.

A minima, its a web server and mail server this mail.serveroffer.lt. 
For sending only, as I can't connect to their mail server (strange) :

root@ns311465:~# telnet 45.125.66.11 25
Trying 45.125.66.11...
telnet: Unable to connect to remote host: Connection refused

Who does the DNS for serveroffer.lt ?

root@ns311465:~# dig serveroffer.lt NS

.....;; ANSWER SECTION:
serveroffer.lt. 400 IN NS ns2.serveroffer.lt.
serveroffer.lt. 400 IN NS ns1.serveroffer.lt.

;; ADDITIONAL SECTION:
ns1.serveroffer.lt.     6564    IN      A       45.125.66.11
ns2.serveroffer.lt.     6564    IN      A       45.125.66.17

Ok, great. Experts, they do their own DNS - and 45.125.66.11 is one of the two domain name servers.
This DNS is for their internal usage, we (or just me ^^) can not use it :

root@ns311465:~# dig @45.125.66.11 google.com

No answer.

But I know it's alive, because, its a domain name server after all, so it has to answer to this :

root@ns311465:~# dig @45.125.66.11 serveroffer.lt A +short
45.125.66.11

did work out.
If not, their site would vanish from the Internet.

@FECambot said in Weird issue with DNS queries generated from pFsense and blocked by pFsense:

Firewall was blocking queries originating from our local DNS

Strange.
Outgoing DNS requests (Resolver) are not blocked by pfSense by default.
Why were 'you' hamering 'their' DNS "45.125.66.11" server (you saw it : it won't answer).

As pfSense doesn't care and doesn't know who "serveroffer.lt" is, the originating DNS request must come from one of the LANs.
edit : As I presume you don't have an open, freely accessible DNS (like, for example 8.8.8.8) on your networks.
Go check your LAN, and see who hammered your DNS Resolver ....

@FECambot said in Weird issue with DNS queries generated from pFsense and blocked by pFsense:

i had an idea and changed the DNS server config of pFsense from the IP of our local DNS server to 8.8.8.8 .

Ok, but tell me why (you think) that would change something.

Apply the KIS rule : activate the pfSense resolver, use the resolve mode, don't fight with the big ones, scan your LAN, check and control who is doing what, and you'll be fine.

@FECambot said in Weird issue with DNS queries generated from pFsense and blocked by pFsense:

If my understanding is correct, a process in pFsense did these queries ? but which one ?

Already answered.
unbound isn't some some kid that is doing nothing, gets bored, and start to haras some random IP.
It was asked to do so.
If it was asked nothing, it will execute the sleep(forever) system call == it does nothing.
And the order (the DNS request) came in by one of your LAN plugs. And so on.
I know, you said this :

@FECambot said in Weird issue with DNS queries generated from pFsense and blocked by pFsense:

Absolutely no system is using internally pFsense as a gateway.

so I say : connect yourself to the console access. This time : even SSH won't do.
But before, set unbound to log very verbosely, set it to the max.
And remove all LAN cables.

Now, connect, and :

tail -f /var/log/resolver.log

and tell us what you see.
Please do this for several days, so you proof me right ^^

[FECambot]

I don't know this remote IP, i have no business with them, all i wanted was to stop this flood of blocked packets...
pFsense was hammering my DNS resolver, i installed WireShark to check for it.
I'll think about it over this week-end, and on monday i'll have another brain (my fellow colleague) to process all your suggestions.

Thanks for the detailed feedback :).

[Gertjan]

@FECambot said in Weird issue with DNS queries generated from pFsense and blocked by pFsense:

pFsense was hammering my DNS resolver ...

The DNS resolver is a process running on pfSense.
As said : no joke neither humor, disconnected all LAN cables and it will stop.
pfSense is a router. Take this word to its basic meaning : it routes from LAN to WAN.
And from WAN to LAN.
It adds very little traffic ** on its own.
Why do you think it does ?

** example : ones in a while (twice a day ?) a task starts up that collects the list of all the software installed : the FreeBSD native packages, and the pfSense packages.
Then it loads the list from the Netgate server, but to do this, first, the Netgate package server host name must be resolved to an "IP" : so, yes, pfSense itself makes use of the build in DNS resolver.
A couple of packets per day ?
To Netgate servers of course. pfSense doesn't talk to any other server on the Internet by itself. Like never. And I mean it.

Edit ok, found another exception : if you have NTP activated, ones in a while the NTP server pool list will get questioned, a candidate is chosen, it's name is resolved and it will connect to this server to sync time. This server, granted, is not a Netgate server .... and adds another 10 packets a day ?

[FECambot]

I need a cooldown time to think straight :)
And by no mean i want to criticize pFsense.

[Gertjan]

@FECambot

No worries. I'm a user just like you.
Critics are not an issue at all. They are the roads to understanding.

Take your time.