PfSense blocking nameservers on Virtualmin?

eiger3970

The www.domain.com is resolvable when this pfSense router is removed and I use my old router.

The name servers are on my Virtualmin server 192.168.1.163.

The pfSense WAN IP is a DHCP received from the cable modem's LAN IP.

Both of the name servers on the Virtualmin server point to the 1 IP? The name servers point to the ISP's WAN IP.
The domain registrar also has the name servers pointing to the ISP's WAN IP.

The server has two name servers, namely ns1.domain.com and ns2.domain.com.

www.intodns.com is a test site to check if the website is working or what error are occurring.

I also tested shieldsup > https://www.grc.com/x/ne.dll?rh1dkyd2, which showed no ports open like 53. Weird as ports are forwarded.

johnpoz

"The name servers are on my Virtualmin server 192.168.1.163."

So lets forget about forwarding for second - you clearly got REFUSED from this server 192.168.1.180, but that is not 192.168.1.163.

As to your port showing open - you forwarded them, did you let the forward create the firewall rule for you? Pot up your wan firewall rules and your nats.

example see attached, notice how all my forwards show that they are linked to firewall rule, and on the firewall rules those ports are allowed. NTP for example that 192.168.1.40 box is a member of ntp.pool so it always has lots of outside ips getting the time.

nats.png_thumb
firewall.png_thumb

eiger3970

Yes, so 192.168.1.180 is the static IP of the old router which is now an AP for the Wi-Fi and uses the spare 4 Ethernet ports for LAN devices to access the pfSense router.
Not sure why 192.168.1.180 is rejecting any packets and it's an AP now and not a router. The ports are forwarded which worked with the Wi-Fi router working before being changed to an AP and using pfSense as the router.

Yes, pfSense has all the ports forwarded that the old Wi-Fi router used to allow all traffic to work.
I followed the standard install, so whatever the default settings pfSense sets up, this is then the case.

Yes, I have confirmed in pfSense that the port forwards (attachment Screenshot-1.png) have created firewall rules (attachment Screenshot-2.png).
Please see the attachments.

Screenshot-1.png_thumb

Screenshot-2.png_thumb

johnpoz

if .180 is just now an AP, why would you be doing a dns query to it?

;; SERVER: 192.168.1.180#53(192.168.1.180)

So are you having issues with any of the other forwards? I can tell from looking at the rules that your understanding of network protocols is lacking. Telnet for example is not a UDP protocol, nor is http or https, etc. But you have your forwards and rules listed for both tcp and udp.

For example does 22 ssh work from outside your network? If you PM my your domain I would be happy to look into if dns works for it or not, where its pointing, etc.

eiger3970

I didn't do a DNS query to the old router 192.168.1.180 on purpose.
I simply ssh'ed into the webserver and ran the dig command.
I guess the webserver is still using a default gateway of 192.168.1.180?

Maybe my port forwards aren't right? Should I have the IP address in Destination IP in the port forwards?

Thank you for the reply.
LAN devices cannot SSH pfSense.

I have two NATs and it's confusing.
Cable modem WAN IP from ISP xxx.xxx.xx.xx.
Cable modem WAN IP 192.168.0.2.
Cable modem LAN IP 192.168.0.50.
pfSense WAN DHCP 192.168.0.2.
pfSense LAN IP 192.168.1.155.

johnpoz

Dude lets schedule a time for me to teamviewer in again.

And you got some typo here - cuse how does your cable modem have the same wan IP as pfsense?

Cable modem WAN IP 192.168.0.2.
Cable modem LAN IP 192.168.0.50.
pfSense WAN DHCP 192.168.0.2.

Did you TURN off all the other forwards you had setup in your cable gateway? Next step is to yes double check your forwards, but need to verify that traffic actually gets to pfsense. Simple enough to do a sniff and send some traffic and verify that pfsense gets it.

Now we need to get your setup correct so you can forward stuff sure - but I highly suggest you Don't run your own DNS!!! And that your register let you point both your name servers to the same IP is beyond me!

eiger3970

Maybe I should have written:
Cable modem DMZ IP 192.168.0.2.

Yes, I turned off all the other forwards in the cable modem (there was only one to 192.168.0.2).

All port forwards in pfSense are the same as my old router.

I am now working out the sniffing which I haven't used on pfSense before.
pfSense 2.1 > Diagnostics > Packet Capture, shows packets


15:32:05.957029 IP 80.73.4.1.11974 > 192.168.0.2.53: UDP, length 48
15:32:05.990893 IP 80.73.4.1.20446 > 192.168.0.2.53: UDP, length 48
15:32:05.994305 IP 80.73.4.1.16531 > 192.168.0.2.53: UDP, length 48
15:32:06.020875 IP 192.168.0.2 > 192.168.0.50: ICMP echo request, id 60499, seq 57094, length 44
15:32:06.022187 IP 192.168.0.50 > 192.168.0.2: ICMP echo reply, id 60499, seq 57094, length 44
15:32:06.035289 IP 192.168.0.2.28288 > 8.8.8.8.53: UDP, length 46
15:32:06.208647 IP 8.8.8.8.53 > 192.168.0.2.28288: UDP, length 76
15:32:06.211065 IP 74.125.224.63.443 > 192.168.0.2.44342: tcp 65
15:32:06.212140 IP 74.125.224.63.443 > 192.168.0.2.44342: tcp 45
15:32:06.212276 IP 192.168.0.2.44342 > 74.125.224.63.443: tcp 0
15:32:06.212482 IP 192.168.0.2.44342 > 74.125.224.63.443: tcp 0
15:32:06.212520 IP 74.125.224.63.443 > 192.168.0.2.44342: tcp 0
15:32:06.212575 IP 192.168.0.2.44342 > 74.125.224.63.443: tcp 0
15:32:06.340727 IP 74.125.189.16.51332 > 192.168.0.2.53: UDP, length 37
15:32:06.378925 IP 74.125.224.63.443 > 192.168.0.2.44342: tcp 0
15:32:06.685704 IP 80.73.4.1.9241 > 192.168.0.2.53: UDP, length 48
15:32:06.713699 IP 80.73.4.1.14926 > 192.168.0.2.53: UDP, length 48
15:32:06.716480 IP 80.73.4.1.47377 > 192.168.0.2.53: UDP, length 48
15:32:06.719723 IP 80.73.4.1.62501 > 192.168.0.2.53: UDP, length 48
15:32:06.742865 IP 80.73.4.1.38217 > 192.168.0.2.53: UDP, length 48
15:32:06.745647 IP 80.73.4.1.50471 > 192.168.0.2.53: UDP, length 48
15:32:06.801138 IP 192.168.0.2.50950 > 74.125.129.84.443: tcp 0
15:32:06.857137 IP 192.168.0.2.49948 > 74.125.224.89.443: tcp 0
15:32:06.952081 IP 74.125.239.14.443 > 192.168.0.2.42534: tcp 65
15:32:06.952411 IP 74.125.239.14.443 > 192.168.0.2.42534: tcp 45
15:32:06.952612 IP 192.168.0.2.42534 > 74.125.239.14.443: tcp 0
15:32:06.952732 IP 74.125.239.14.443 > 192.168.0.2.42534: tcp 0
15:32:06.952848 IP 192.168.0.2.42534 > 74.125.239.14.443: tcp 0
15:32:06.997880 IP 74.125.129.84.443 > 192.168.0.2.50950: tcp 0
15:32:07.020887 IP 192.168.0.2 > 192.168.0.50: ICMP echo request, id 60499, seq 57350, length 44
15:32:07.022216 IP 192.168.0.50 > 192.168.0.2: ICMP echo reply, id 60499, seq 57350, length 44
15:32:07.022828 IP 74.125.224.89.443 > 192.168.0.2.49948: tcp 0
15:32:07.124205 IP 74.125.239.14.443 > 192.168.0.2.42534: tcp 0
15:32:08.020887 IP 192.168.0.2 > 192.168.0.50: ICMP echo request, id 60499, seq 57606, length 44
15:32:08.022235 IP 192.168.0.50 > 192.168.0.2: ICMP echo reply, id 60499, seq 57606, length 44
15:32:08.222384 IP 80.73.4.1.63144 > 192.168.0.2.53: UDP, length 48
15:32:08.248429 IP 80.73.4.1.25708 > 192.168.0.2.53: UDP, length 48
15:32:08.251729 IP 80.73.4.1.25328 > 192.168.0.2.53: UDP, length 48
15:32:08.256460 IP 80.73.4.1.21083 > 192.168.0.2.53: UDP, length 48
15:32:08.259361 IP 80.73.4.1.52333 > 192.168.0.2.53: UDP, length 48
15:32:08.281641 IP 80.73.4.1.63732 > 192.168.0.2.53: UDP, length 48
15:32:09.020886 IP 192.168.0.2 > 192.168.0.50: ICMP echo request, id 60499, seq 57862, length 44
15:32:09.022223 IP 192.168.0.50 > 192.168.0.2: ICMP echo reply, id 60499, seq 57862, length 44
15:32:09.069953 IP 192.168.0.2.51754 > 117.20.45.131.443: tcp 34
15:32:09.321877 IP 117.20.45.131.443 > 192.168.0.2.51754: tcp 34
15:32:09.322050 IP 192.168.0.2.51754 > 117.20.45.131.443: tcp 0
15:32:09.450342 IP 74.125.239.31.443 > 192.168.0.2.36462: tcp 65
15:32:09.453065 IP 74.125.239.31.443 > 192.168.0.2.36462: tcp 45
15:32:09.453207 IP 192.168.0.2.36462 > 74.125.239.31.443: tcp 0
15:32:09.453418 IP 192.168.0.2.36462 > 74.125.239.31.443: tcp 0
15:32:09.453450 IP 74.125.239.31.443 > 192.168.0.2.36462: tcp 0
15:32:09.453503 IP 192.168.0.2.36462 > 74.125.239.31.443: tcp 0
15:32:09.618327 IP 74.125.239.31.443 > 192.168.0.2.36462: tcp 0
15:32:09.709451 IP 80.73.4.1.39502 > 192.168.0.2.53: UDP, length 48
15:32:09.714635 IP 80.73.4.1.54104 > 192.168.0.2.53: UDP, length 48
15:32:09.731944 IP 80.73.4.1.33951 > 192.168.0.2.53: UDP, length 48
15:32:09.740454 IP 80.73.4.1.11774 > 192.168.0.2.53: UDP, length 48
15:32:09.759855 IP 80.73.4.1.14301 > 192.168.0.2.53: UDP, length 48
15:32:09.762640 IP 80.73.4.1.34165 > 192.168.0.2.53: UDP, length 48

I checked my website and the DNS still isn't found when going via pfSense. Works when old router is used, so you said pfSense won't stop DNS in my settings, so maybe the website needs some settings adjusted.

The TeamViewer ID is the same as yesterday.
The TeamViewer password is: 2046 today.

johnpoz

"so maybe the website needs some settings adjusted."

No it would not be your website - but yes the box/vm it runs on. What is the gateway of that machine.. From our session the other day before I had to run to work. It was clear that pfsense was forwarding traffic to the IP you say is your dns .163 - but there were no answers from that machine.

And from the vm I was on, I could not query that local IP for dns.. But since I could ping it?? Not sure if was pinging a NAT or not.. Since the VM was on a 10.x address and your dns is on 192.168.1 address

As I asked you in the chat from the last tv session - show me a dns query from something on your network that can talk to your nameserver. Cuz pfsense is clearly forwarding the traffic too it. But there were no answers from it.

eiger3970

So, I disconnected the Switch and connected pfSense LAN direct to the server.
I used another Internet connection to try to bring up the website (default gateway 192.168.1.180 to old router settings) and nothing. The VoIP phone (default gateway 192.168.1.155 to pfSense LAN) also on the server didn't work.

I connected pfSense LAN back into the Switch and the server's cable from the switch back into the server and my LAN was able to ping it and the VoIP phone (default gateway 192.168.1.155 to pfSense LAN) worked, but website (default gateway 192.168.1.180 to old router settings) still won't show via pfSense.

johnpoz

Dude I have been busy with RL last view days – this is NOT freaking rocket science.. When I was on your system your nameser, what you keep calling virtualmin was NOT answering a query from your own local network.. So nothing that we do on pfsense is going to fix that.

virtualmin is just some freaking software to help host websites for clients.. It dos NOT provide dns - your underlaying dns software would do that - I would assume BIND if running on linux

http://www.virtualmin.com/
Install your Operating System

Start with a freshly installed, Grade A supported Operating System on your server or VPS. CentOS and Ubuntu LTS are popular OS's for hosting..

So lets forget that, lets forget using it to toubleshoot basic network and applications. Does your your OS your running virtualmin on point to pfsense to for a gateway for starters? Is Bind Running for another - where is a query from a box on the 192.168 local network of yours doing a query to your nameserver??

Also - I am more than happy to help.. But give me something on the actual network your nameserver is on - not some vm behind a NAT on some 10.x network.

Lets schedule a time tmrw maybe? I am home all day and happy to teamviewer in and get this working.. But needs some basics to work.. So whatever OS we are going to use for TV - the mouse has to work.. Lets not do a vm running on a vm, running on some other vm software under a VM ;)

Do you not have a hard machine on the same network as your pfsense lan, and your nameserver - There should be no problem with it being linux based – but I think the problem is the nested vms I believe you were using. And whatever OS your virtualmin is runing on we are going to need access to that - be it ssh, be it remote desktop, but it another TV session - but I am not going to troubleshoot virtualmin in this situation since it has NOTHING to do with the actual problem,.

eiger3970

Hi, yes, CentOS Linux 6.4 is using BIND.

I checked CentOS and the network configuration is:
Address: 192.168.1.163.
Netmask: 255.255.255.0.
Gateway: 192.168.1.155.
DNS: 192.168.180.

I changed the DNS to 8.8.8.8.

So yes, CentOS is pointing automatically to pfSense's LAN 192.168.1.155.

Happy for some online work. I have an updated TV, so that might help with the mouse click problem, which was on a real machine. (only the 2nd time I went to a VM to try and fix the mouse click problem, which it did (but caused the network issue on 10.x.x.x)).

I'm online in about 30 minutes.

johnpoz

So why would a dns server running bind not point to itself for dns? Is bind not allow recursive? Why would you not point it to pfsense if that is the case - how are you going to resolve your on local domain pointing to 8.8.8.8?

So can you query your bind server and resolve your domain now? Because I couldn't last time I was on your network. So lets see your query - because if that works then it will work from the outside since pfsense port forwards are setup, and we saw the traffic being sent to your .163 address via the sniff on pfsense lan remember.

eiger3970

I'm getting a bit confused with all the settings now with pfSense router and Proxmox server with Virtualmin (and Virtualmin running off CentOS (CentOS then having its own DNS settings).

So I changed the CentOS DNS from 192.168.1.180 to 8.8.8.8 and now to 192.168.1.155.

Website still not showing.

johnpoz

ARGGH dude what that box uses for dns has NOTHING to do with your issue.

Your running BIND as you stated, this hosts up your domain.tld, this is not answering a simple query from computer on the same network as it.

Say 192.168.1.162 – so how does pfsense have anything to do with it?

So my local domain is local.lan -- if I ask my dns server for a simple A record, lets call it my printer I call brother.local.lan

C:>nslookup
Default Server: pfsense.local.lan
Address: 192.168.1.253

brother.local.lan
Server: pfsense.local.lan
Address: 192.168.1.253

Name: brother.local.lan
Address: 192.168.2.50

See how I get a response.. So on your network.. Do a simple nslookup for a record that should be there say www.yourdomain.tld

Do you get a response?? If NOT then nothing you do on pfsense or the rest of your network is going to fix that.. That is a problem with BIND running on your host, is it even running? Have you looked in its log? Does this centos box have a local host firewall? etc.. etc..

You need to fix that before we have to worry about people on the internet being able to resolve www.yourdomain.tld.

See attached - I am on my workstation on the 192.168.1.0/24 network, my dns (pfsense in this case) has a record for all my local devices in the local.lan domain. If I query it for a record - it answers. Lets see this from your workstation doing a query to your .163 server running bind. You can change the host you query via server command in nslookup. So make sure you change server to your .163 address and do a query for records you created in yourdomain

Let us see these queries!! Then if not working from the internet I will be happy to TV in again and take a look at your forwards. But they were working last time I was in.

If your using dig, you can do same sort of command with @serverIP fqdn

C:>dig @4.2.2.2 www.pfsense.org

; <<>> DiG 9.9.5-W1 <<>> @4.2.2.2 www.pfsense.org
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 56986
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;www.pfsense.org. IN A

;; ANSWER SECTION:
www.pfsense.org. 1800 IN A 192.207.126.26

;; Query time: 221 msec
;; SERVER: 4.2.2.2#53(4.2.2.2)
;; WHEN: Wed Feb 26 07:54:13 Central Standard Time 2014
;; MSG SIZE rcvd: 60

simplequery.png_thumb

changeserver.png_thumb

eiger3970

Yes, I have checked that the BIND server is running.

Here are the results:


192.168.1.120 > Terminal > nslookup www.domain.tld
Server:		8.8.8.8
Address:	8.8.8.8#53
** server can't find www.domain.tld: SERVFAIL

192.168.1.120 > Terminal > dig www.domain.tld
; <<>> DiG 9.9.3-rpz2+rl.13214.22-P2-Ubuntu-1:9.9.3.dfsg.P2-4ubuntu1.1 <<>> www.domain.tld
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 63678
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;www.domain.tld.		IN	A

;; Query time: 3177 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Sat Mar 01 15:37:10 EST 2014
;; MSG SIZE  rcvd: 48

192.168.1.163 > Terminal > nslookup www.sk8parks.org.au
Server:		192.168.1.155
Address:	192.168.1.155#53
** server can't find www.domain.tld: NXDOMAIN

192.168.1.163 > Terminal > dig www.domain.tld
; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.17.rc1.el6_4.6 <<>> www.domain.tld
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 52297
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;www.domain.tld.		IN	A

;; Query time: 3192 msec
;; SERVER: 192.168.1.155#53(192.168.1.155)
;; WHEN: Sat Mar  1 15:41:56 2014
;; MSG SIZE  rcvd: 37

johnpoz

ARRRGGHHHH!!!!!

Query your freaking bind server and does it return an answer??

How hard is that to understanding – I have stated like a million times already. You query google and pfsense.. WTF?? From the DNS box itself even??

Neither of those are going to work - because your BIND server is Not Answering!!

dig @192.168.1.163 www.sk8parks.org.au

or nslookup

server 192.168.1.163
www.sk8parks.org.au

If your BIND server does not respond, since that is where you point to for this sk8parks.org.au then no other dns server on the planet is going to resolve sk8parks.org.au.. And that has nothing to do with a port forwarding or pfsense.

eiger3970

Thank you for the clarification.
I think the results are showing that from my computer 192.168.1.120, I can connect to BIND.


192.168.1.120 ~ $ dig @192.168.1.163 www.domain.tld

; <<>> DiG 9.9.3-rpz2+rl.13214.22-P2-Ubuntu-1:9.9.3.dfsg.P2-4ubuntu1.1 <<>> @192.168.1.163 www.domain.tld
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 31480
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 1, ADDITIONAL: 3

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;www.domain.tld.		IN	A

;; ANSWER SECTION:
www.domain.tld.	38400	IN	A	xxx.xxx.xxx.xx

;; AUTHORITY SECTION:
domain.tld.	38400	IN	NS	localhost.localdomain.

;; ADDITIONAL SECTION:
localhost.localdomain.	86400	IN	A	127.0.0.1
localhost.localdomain.	86400	IN	AAAA	::1

;; Query time: 3 msec
;; SERVER: 192.168.1.163#53(192.168.1.163)
;; WHEN: Mon Mar 03 10:02:26 EST 2014
;; MSG SIZE  rcvd: 143

192.168.1.120 ~ $ nslookup
> server 192.168.1.163
Default server: 192.168.1.163
Address: 192.168.1.163#53
> www.domain.tld
Server:		192.168.1.163
Address:	192.168.1.163#53

Name:	www.domain.tld
Address: xxx.xxx.xxx.xx

johnpoz

And what is your bind config?

Because I can understand why you would change out the IP address of your record, but that stays your Nameserver is 127.0.0.1 localhost.localdomain?

And why was I not able to query it when I was teamviewered in and on your windows box. Does your bind config not allow answer to network outside of 192.168.1.0/24?

Please post your bind config.

should be named.conf.

eiger3970

Ok, here's the BIND configuration on the webserver.


//
// named.conf
//
// Provided by Red Hat bind package to configure the ISC BIND named(8) DNS
// server as a caching only nameserver (as a localhost DNS resolver only).
//
// See /usr/share/doc/bind*/sample/ for example named configuration files.
//

options {
        listen-on port 53 {
                any;
                };
        listen-on-v6 port 53 {
                any;
                };
        directory       "/var/named";
        dump-file       "/var/named/data/cache_dump.db";
        statistics-file "/var/named/data/named_stats.txt";
        memstatistics-file "/var/named/data/named_mem_stats.txt";
        recursion yes;

        dnssec-enable yes;
        dnssec-validation yes;
        dnssec-lookaside auto;

        /* Path to ISC DLV key */
        bindkeys-file "/etc/named.iscdlv.key";

        managed-keys-directory "/var/named/dynamic";
};

logging {
        channel default_debug {
                file "data/named.run";
                severity dynamic;
        };
};

zone "." IN {
        type hint;
file "named.ca";
};

include "/etc/named.rfc1912.zones";
include "/etc/named.root.key";

zone "domain.tld" {
        type master;
        file "/var/named/domain.tld.hosts";
        allow-transfer {
                127.0.0.1;
                localnets;
                };
        };

I think you weren't able to query the BIND server when in the Windows OS, as the Windows OS was on a network of 10.0.0.1, being different from BIND server's network 192.168.1.0/24.

I tested if the pfSense firewall is blocking port 53 with the following results:


192.168.1.163# lsof -ni tcp:53
COMMAND  PID  USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
named   1300 named   20u  IPv6   9350      0t0  TCP *:domain (LISTEN)
named   1300 named   21u  IPv4   9355      0t0  TCP 127.0.0.1:domain (LISTEN)
named   1300 named   25u  IPv4  10525      0t0  TCP 192.168.1.163:domain (LISTEN)

192.168.1.163# netstat -nat | grep :53
tcp        0      0 192.168.1.163:53            0.0.0.0:*                   LISTEN      
tcp        0      0 127.0.0.1:53                0.0.0.0:*                   LISTEN      
tcp        0      0 :::53                       :::*                        LISTEN

192.168.1.120# lsof -ni tcp:53

192.168.1.120# netstat -nat | grep :53

If I understand the results, then my LAN computer 192.168.1.120 can't get through pfSense 192.168.1.155 to the webserver 192.168.1.163's DNS port 53.

pfSense does have port 53 forwarded to 192.168.1.163.

I didn't know the Windows OS was on a different network…this must have happened by default with the Virtual Machine's bridge setting.
I am trying to fix this.

johnpoz

And do you have a firewall running on this server.. You must because that explains the problem. Windows was pinging the IP, so should of worked unless firewall on the .163 box blocking?

As to .120 can't get through - what??? You queried and got an answer.. And .120 does not NOT go through pfsense to get to .163 – they are on the same segment, pfsense is only used for on and off that 192.168.1.0/24 segment - boxes talking to each other on that network could give a shit if pfsense was even on.

Lets be clear you are changing out domain.tld for your actual domain?

And you think you did what with that netstat and lsof command? That has Nothing to do with what pfsense is or isn't doing, your just showing if that box is listening on tcp 53.. What about UDP, most queries use UDP not TCP.. tcp would be used for zone transfers and large queries.

And once you get queries working, you have to fix your zone file - you can not list localhost.localdomain as your NS with loopback as the IP and expect the zone to work ;)

Can you run

iptables --list

On the centos box, the .163 so we can see the firewall rules on it. Prob have to be root to run it.

example

root@ubuntu:/# iptables --list
Chain INPUT (policy ACCEPT)
target prot opt source destination
sshguard all -- anywhere anywhere

Chain FORWARD (policy ACCEPT)
target prot opt source destination

Chain OUTPUT (policy ACCEPT)
target prot opt source destination

Chain sshguard (1 references)
target prot opt source destination
root@ubuntu:/#