pfsense not using GoDaddy's domain forward, only for internal users

scottys · Oct 10, 2019, 2:12 PM

Hey so let me try to explain this.
We have domains we host for other people and set up forwarding so it goes to the site we made for them (which is a page under our main site, but that's how they want it). Externally, everything works fine. Internally, no one can reach it when going to the domain but they can reach the destination normally.

Also, it looks like pfsense crashed sometime last week, possibly have something to do with it?:

info (1).0

The main reason why I think there way be a relationship is if I go on our backup pfsense, everything works normally (after changing gateways on servers and workstations to ensure it is all going to the right place). So for some reason the main pfsense is not redirecting from the DNS like it should, however our backup (exact copy) works normally.

Any help would be greatly appreciated

Thank you!

Derelict · Oct 10, 2019, 8:23 PM

If it was an exact copy they would behave exactly the same.

Normally the issue you describe is NAT reflection.

scottys · Oct 14, 2019, 12:35 PM

Normally the issue you describe is NAT reflection.

Yes, I would agree however no NAT rules have changed (or really anything on pfsense). It used to work, and it doesn't anymore

scottys · Oct 15, 2019, 2:31 PM

bump

This is driving me crazy. It was working a few weeks ago and there have been no changes. It doesn't make sense

Basically this is what is going on:

(internal user)

go to test.example.com (this is setup in the DNS for a 301 redirect to one of our sites)
it times out

(external user)

go to test.example.com
it loads normally

Derelict · Oct 15, 2019, 3:56 PM

Great. Show your port forwards. What does the domain resolve to externally? What does the domain resolve to internally?

scottys · Oct 16, 2019, 12:34 PM

@Derelict said in pfsense not using GoDaddy's domain forward, only for internal users:

Great. Show your port forwards. What does the domain resolve to externally? What does the domain resolve to internally?

Fixed NAT reflection problems here about a hear and a half ago:

NAT Inbound:

NAT Outbound:

Ok so to explain this a bit. Traffic comes in and hits .50 which is a load balanced cluster of 4 servers (.51-.54). Outbound I put the fail-over IPs ahead of the main IPs, but it doesn't seem to matter as long as the gateway is correct and there is a rule for it. The main IP is the 8.18.xx.xx one

When going to order.castrolroadside.ca, it will DNS 301 redirect you to https://nsdmc.com/something
Externally, like if you click it, it works. However internally, like if I click it, we get time out errors.

scottys · Oct 16, 2019, 12:40 PM

@Derelict said in pfsense not using GoDaddy's domain forward, only for internal users:

What does the domain resolve to externally? What does the domain resolve to internally?

The domain that is being forwarded resolves to 184.168.131.241 which is GoDaddy's forwarding IP apparently. The domain it forward to resolves to the correct external IP of our servers.

For reference, I went into ever place pfsense could be blocking it and I do not see this IP anywhere. I ran a packet caputure and can see it sending packets to the IP [SYN] but not getting anything back in the TCP string

scottys · Oct 16, 2019, 12:56 PM

Side note: I have set up tests that have it redirect to google.com and other sites besides ours. The forwarding on those do not work either

stephenw10 · Oct 16, 2019, 4:47 PM

So it fails accessing both order.castrolroadside.ca and nsdmc.com/something from an internal client?

When you ran the packet capture which IP was it trying to reach? On the internal interface?

A 301 redirect is an http function rather than DNS. If it just resolved correctly with DNS I doubt this would be an issue.
The initial connection to 184.168.131.241 will not be caught by NAT reflection since that's obviously not forwarded locally. I would expect the redirected connection to your WAN to be caught though.

Steve

scottys · Oct 16, 2019, 4:27 PM

@stephenw10 said in pfsense not using GoDaddy's domain forward, only for internal users:

So it fails accessing both order.castrolroadside.ca and nsdmc.com/something from an internal client?

No. order.castrolroadside.ca fails from an internal user but works normally outside of pfsense. nsdmc.com/something is accessible from both internal and external users

When you ran the packet capture which IP was it trying to reach? On the internal interface?

It hit the internal interface (pfsense) and going to 184.168.131.241, but nothing seems to be returned while following the packet trail.

A 301 redirect is an http function rather than DNS. If it just resolved correctly with DNS I doubt this would be an issue.

I understand. Just trying to explain the setup we have as that 301 redirect is setup in the same place where the DNS is for that domain.

The initial connection to 184.168.131.241 will not be caught by NAT replaction since that's obviously not forwarded locally. I would expect the redirected connection to your WAN to be caught though.

Caught how? Could pfsense be "stripping" the SNI or other parts that would say "I am trying to go to order.castrolroadside.ca" and not "I am trying to go to 184.168.131.241"
The latter seems like what is going on, since that is the same IP for GoDaddy redirects in their DNS zone

Example:
nsd-test.com does not work for me, but if you go there it should redirect you to google

stephenw10 · Oct 16, 2019, 5:01 PM

Ok, not a NAT reflection issue if you can't see redirects to an external site either.

Is it even succeeding with the connection to 184.168.131.241?

Probably time for a packet capture of the connection or maybe check the browser console.

Steve

scottys · Oct 16, 2019, 5:33 PM

@stephenw10 said in pfsense not using GoDaddy's domain forward, only for internal users:

Is it even succeeding with the connection to 184.168.131.241?

No, it is not responding to pings where when I ping the IP from my phone it works. This would mean pfsense is blocking it, but I have checked everyplace it could be blocked (pfBlockerNG, Snort, Aliases, etc) and the IP not in there.

stephenw10 · Oct 16, 2019, 6:18 PM

Or it could be mis-routed somehow...

scottys · Oct 16, 2019, 7:25 PM

@stephenw10 said in pfsense not using GoDaddy's domain forward, only for internal users:

Or it could be mis-routed somehow...

I am checking every rule that is in place and it doesn't make sense.

I think it is a godaddy issue. I just tested on one of my personal domains using a different DNS provider (1and1) and it redirected me just fine...

scottys · Oct 16, 2019, 8:15 PM

Confirmed that it is something in the pfsense. Tested from my homelab, which uses pfsense, and all the redirects work perfectly. I am at a loss for word with all the frustration this is causing

stephenw10 · Oct 16, 2019, 9:54 PM

Ok, run a packet capture for that IP on WAN, is it even leaving?

Mystery disappearing packets... check IPSec tunnels.

Traffic that is blocked and not logged, check pfBlocker, Snort and Suricata if you have any of those installed.

Steve

scottys · Oct 16, 2019, 11:09 PM

@stephenw10 said in pfsense not using GoDaddy's domain forward, only for internal users:

Ok, run a packet capture for that IP on WAN, is it even leaving?

Mystery disappearing packets... check IPSec tunnels.

Traffic that is blocked and not logged, check pfBlocker, Snort and Suricata if you have any of those installed.

Steve

The packet capture shows it hitting pfsense main gateway to go out, but I dont think it is leaving. No IPSec, and the IP or my workstation IP does not show up in pfBlockerNG, snort, and custom aliases (as we have a rule for specific hosts/networks to block.)

It literally just disappears. I'll post a packet capture tomorrow when I get there

scottys · Oct 17, 2019, 3:29 PM

@stephenw10 said in pfsense not using GoDaddy's domain forward, only for internal users:

Ok, run a packet capture for that IP on WAN, is it even leaving?

Mystery disappearing packets... check IPSec tunnels.

Traffic that is blocked and not logged, check pfBlocker, Snort and Suricata if you have any of those installed.

Steve

Packet Capture on backup pfsense (that works)

11:12:48.920644 d8:9e:f3:12:77:6d > 00:1b:21:8b:15:80, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 128, id 60463, offset 0, flags [DF], proto TCP (6), length 52)
    90.0.0.95.5329 > 184.168.131.241.80: Flags [S], cksum 0xde49 (correct), seq 2187747910, win 65535, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
11:12:48.920646 d8:9e:f3:12:77:6d > 00:1b:21:8b:15:80, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 128, id 60464, offset 0, flags [DF], proto TCP (6), length 52)
    90.0.0.95.5330 > 184.168.131.241.80: Flags [S], cksum 0x2732 (correct), seq 509181290, win 65535, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
11:12:49.920854 d8:9e:f3:12:77:6d > 00:1b:21:8b:15:80, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 128, id 60465, offset 0, flags [DF], proto TCP (6), length 52)
    90.0.0.95.5330 > 184.168.131.241.80: Flags [S], cksum 0x2732 (correct), seq 509181290, win 65535, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
11:12:49.921617 d8:9e:f3:12:77:6d > 00:1b:21:8b:15:80, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 128, id 60466, offset 0, flags [DF], proto TCP (6), length 52)
    90.0.0.95.5329 > 184.168.131.241.80: Flags [S], cksum 0xde49 (correct), seq 2187747910, win 65535, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
11:12:50.004757 00:1b:21:8b:15:80 > d8:9e:f3:12:77:6d, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 52, id 0, offset 0, flags [DF], proto TCP (6), length 52)
    184.168.131.241.80 > 90.0.0.95.5329: Flags [S.], cksum 0xf48b (correct), seq 1184051546, ack 2187747911, win 29200, options [mss 1380,nop,nop,sackOK,nop,wscale 7], length 0
11:12:50.005159 d8:9e:f3:12:77:6d > 00:1b:21:8b:15:80, ethertype IPv4 (0x0800), length 60: (tos 0x0, ttl 128, id 60467, offset 0, flags [DF], proto TCP (6), length 40)
    90.0.0.95.5329 > 184.168.131.241.80: Flags [.], cksum 0xa31e (correct), seq 1, ack 1, win 1024, length 0
11:12:50.005824 d8:9e:f3:12:77:6d > 00:1b:21:8b:15:80, ethertype IPv4 (0x0800), length 345: (tos 0x0, ttl 128, id 60468, offset 0, flags [DF], proto TCP (6), length 331)
    90.0.0.95.5329 > 184.168.131.241.80: Flags [P.], cksum 0x3446 (correct), seq 1:292, ack 1, win 1024, length 291: HTTP, length: 291
	GET / HTTP/1.1
	Accept: text/html, application/xhtml+xml, image/jxr, */*
	Accept-Language: en-US
	User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; Trident/7.0; rv:11.0) like Gecko
	UA-CPU: AMD64
	Accept-Encoding: gzip, deflate
	Host: test2.castrolroadside.ca
	Connection: Keep-Alive
	
11:12:50.088793 00:1b:21:8b:15:80 > d8:9e:f3:12:77:6d, ethertype IPv4 (0x0800), length 54: (tos 0x0, ttl 52, id 33045, offset 0, flags [DF], proto TCP (6), length 40)
    184.168.131.241.80 > 90.0.0.95.5329: Flags [.], cksum 0xa50e (correct), seq 1, ack 292, win 237, length 0
11:12:50.110807 00:1b:21:8b:15:80 > d8:9e:f3:12:77:6d, ethertype IPv4 (0x0800), length 268: (tos 0x0, ttl 52, id 33046, offset 0, flags [DF], proto TCP (6), length 254)
    184.168.131.241.80 > 90.0.0.95.5329: Flags [P.], cksum 0xf2ec (correct), seq 1:215, ack 292, win 237, length 214: HTTP, length: 214
	HTTP/1.1 301 Moved Permanently
	Server: nginx/1.12.2
	Date: Thu, 17 Oct 2019 15:12:50 GMT
	Content-Type: text/html; charset=utf-8
	Transfer-Encoding: chunked
	Connection: close
	Location: https://nsdmc.org
	
	0
	
11:12:50.110821 00:1b:21:8b:15:80 > d8:9e:f3:12:77:6d, ethertype IPv4 (0x0800), length 54: (tos 0x0, ttl 52, id 33047, offset 0, flags [DF], proto TCP (6), length 40)
    184.168.131.241.80 > 90.0.0.95.5329: Flags [F.], cksum 0xa437 (correct), seq 215, ack 292, win 237, length 0
11:12:50.111038 d8:9e:f3:12:77:6d > 00:1b:21:8b:15:80, ethertype IPv4 (0x0800), length 60: (tos 0x0, ttl 128, id 60469, offset 0, flags [DF], proto TCP (6), length 40)
    90.0.0.95.5329 > 184.168.131.241.80: Flags [.], cksum 0xa126 (correct), seq 292, ack 215, win 1023, length 0
11:12:50.111277 d8:9e:f3:12:77:6d > 00:1b:21:8b:15:80, ethertype IPv4 (0x0800), length 60: (tos 0x0, ttl 128, id 60470, offset 0, flags [DF], proto TCP (6), length 40)
    90.0.0.95.5329 > 184.168.131.241.80: Flags [.], cksum 0xa125 (correct), seq 292, ack 216, win 1023, length 0
11:12:50.112451 d8:9e:f3:12:77:6d > 00:1b:21:8b:15:80, ethertype IPv4 (0x0800), length 60: (tos 0x0, ttl 128, id 60471, offset 0, flags [DF], proto TCP (6), length 40)
    90.0.0.95.5329 > 184.168.131.241.80: Flags [F.], cksum 0xa124 (correct), seq 292, ack 216, win 1023, length 0
11:12:50.195320 00:1b:21:8b:15:80 > d8:9e:f3:12:77:6d, ethertype IPv4 (0x0800), length 54: (tos 0x0, ttl 52, id 21203, offset 0, flags [DF], proto TCP (6), length 40)
    184.168.131.241.80 > 90.0.0.95.5329: Flags [.], cksum 0xa436 (correct), seq 216, ack 293, win 237, length 0
11:12:51.921420 d8:9e:f3:12:77:6d > 00:1b:21:8b:15:80, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 128, id 60472, offset 0, flags [DF], proto TCP (6), length 52)
    90.0.0.95.5330 > 184.168.131.241.80: Flags [S], cksum 0x2732 (correct), seq 509181290, win 65535, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
11:12:52.004288 00:1b:21:8b:15:80 > d8:9e:f3:12:77:6d, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 52, id 0, offset 0, flags [DF], proto TCP (6), length 52)
    184.168.131.241.80 > 90.0.0.95.5330: Flags [S.], cksum 0xa608 (correct), seq 4214035499, ack 509181291, win 29200, options [mss 1380,nop,nop,sackOK,nop,wscale 7], length 0
11:12:52.004520 d8:9e:f3:12:77:6d > 00:1b:21:8b:15:80, ethertype IPv4 (0x0800), length 60: (tos 0x0, ttl 128, id 60473, offset 0, flags [DF], proto TCP (6), length 40)
    90.0.0.95.5330 > 184.168.131.241.80: Flags [.], cksum 0x549b (correct), seq 1, ack 1, win 1024, length 0

Packet capture from main pfsense (which does not work)

11:17:45.491015 d8:9e:f3:12:77:6d > 00:1b:21:8b:1f:70, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 128, id 21202, offset 0, flags [DF], proto TCP (6), length 52)
    90.0.0.95.5425 > 184.168.131.241.80: Flags [S], cksum 0xa19f (correct), seq 712767099, win 65535, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
11:17:45.491016 d8:9e:f3:12:77:6d > 00:1b:21:8b:1f:70, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 128, id 21203, offset 0, flags [DF], proto TCP (6), length 52)
    90.0.0.95.5424 > 184.168.131.241.80: Flags [S], cksum 0x1674 (correct), seq 3340887297, win 65535, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
11:17:46.491139 d8:9e:f3:12:77:6d > 00:1b:21:8b:1f:70, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 128, id 21204, offset 0, flags [DF], proto TCP (6), length 52)
    90.0.0.95.5424 > 184.168.131.241.80: Flags [S], cksum 0x1674 (correct), seq 3340887297, win 65535, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
11:17:46.491143 d8:9e:f3:12:77:6d > 00:1b:21:8b:1f:70, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 128, id 21205, offset 0, flags [DF], proto TCP (6), length 52)
    90.0.0.95.5425 > 184.168.131.241.80: Flags [S], cksum 0xa19f (correct), seq 712767099, win 65535, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
11:17:48.491019 d8:9e:f3:12:77:6d > 00:1b:21:8b:1f:70, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 128, id 21207, offset 0, flags [DF], proto TCP (6), length 52)
    90.0.0.95.5425 > 184.168.131.241.80: Flags [S], cksum 0xa19f (correct), seq 712767099, win 65535, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
11:17:48.491020 d8:9e:f3:12:77:6d > 00:1b:21:8b:1f:70, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 128, id 21206, offset 0, flags [DF], proto TCP (6), length 52)
    90.0.0.95.5424 > 184.168.131.241.80: Flags [S], cksum 0x1674 (correct), seq 3340887297, win 65535, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
11:17:52.491185 d8:9e:f3:12:77:6d > 00:1b:21:8b:1f:70, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 128, id 21208, offset 0, flags [DF], proto TCP (6), length 52)
    90.0.0.95.5426 > 184.168.131.241.80: Flags [S], cksum 0xa1c9 (correct), seq 1107156686, win 65535, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
11:17:53.491645 d8:9e:f3:12:77:6d > 00:1b:21:8b:1f:70, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 128, id 21209, offset 0, flags [DF], proto TCP (6), length 52)
    90.0.0.95.5426 > 184.168.131.241.80: Flags [S], cksum 0xa1c9 (correct), seq 1107156686, win 65535, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
11:17:55.492233 d8:9e:f3:12:77:6d > 00:1b:21:8b:1f:70, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 128, id 21210, offset 0, flags [DF], proto TCP (6), length 52)
    90.0.0.95.5426 > 184.168.131.241.80: Flags [S], cksum 0xa1c9 (correct), seq 1107156686, win 65535, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0

scottys · Oct 21, 2019, 5:52 PM

bump.

I have gone through every setting I can think of from outing to NAT to firewall rules to where it might be getting blocked, and I can't find anything. Anyone run into issues like this?

stephenw10 · Oct 26, 2019, 3:49 PM

Where exactly did you take those captures?