Port forwarding failing on the return loop



  • Attempting to porf forward to DNS server on the LAN. So public ip > 172.17.16.40 is set to for TCP/UDP on the DNS port (53). I can ping 172.17.16.40 from pfSense. I can do successful DNS lookups to 172.17.16.40 from other systems that can reach it. But when I go to it via the public IP there's no response. Watching the traffic on 172.17.16.40 with iptraf shows that it's getting the requests and sending responses:

    UDP (72 bytes) from 207.136.236.70:56294 to 172.17.16.40:53 on eth0   
      UDP (227 bytes) from 172.17.16.40:53 to 207.136.236.70:56294 on eth0 
      UDP (72 bytes) from 207.136.236.70:56294 to 172.17.16.40:53 on eth0               
      UDP (227 bytes) from 172.17.16.40:53 to 207.136.236.70:56294 on eth0

    The FirewallNATPort Forward screen on pfSense shows:

    Rule on (checked), Linked rule, WAN, TCP/UDP, *, *, public address, 53 (DNS), 172.17.16.40, 53 (DNS)

    That's obviously set right, since the traffic does get to the internal system, which responds. But something's going wrong and causing the return traffic to not to reach the querying remote system. I can ping the querying system from the DNS server, and outbound traffic is generally allowed. There's no traffic from the public IP on pfSense reaching the querying system and getting blocked there – it's a Linux box with its own firewall and I log all drops. I can also confirm that if I telnet to the public IP, port 53, the response does not make it back. Yet if I telnet to port 53 on the internal IP (via an OpenVPN tunnel) it responds just fine.

    I tried setting it to Pass rather than Linked Rule, but that didn't help. Should I be debugging this from the command line, looking at the pf settings directly? I'm more used to Linux netfilter (directly or through FireHOL); the utilities for pf are new to me.


  • Netgate

    Diagnostics > Packet Capture on the firewall inside interface for the same traffic.

    What is the default gateway set to on 172.17.16.40

    Whatever the issue is, it is probably listed here:

    https://doc.pfsense.org/index.php/Port_Forward_Troubleshooting



  • @Derelict:

    Diagnostics > Packet Capture on the firewall inside interface for the same traffic.

    Thanks.

    What is the default gateway set to on 172.17.16.40

    It's set to an internal router. That router, in turn, has a default gateway which is not the pfSense box. Does PF have a way to make this work? If so, can that be configured through pfSense, or does it require going beyond it?

    Is pfSense's version of port forwarding dependent on the default gateway being the pfSense box that's attempting the NAT? We're trying to configure is a pair of pfSense boxes in a multi-WAN (2 now, may go to 3) setup with multiple IPs per WAN, where CARP has each box in normal operation handling a different WAN for incoming traffic, and each is using NAT for several of the IPs. There are a lot of different NAT variants, and reading Hansteen's The Book of PF makes clear that there are a variety of complex ways to hand NAT with PF and the BSDs. These are often not intuitively obvious, particularly in getting them to handle return traffic well, Hansteen says. Is there any documentation on how to explore the rules that the pfSense GUI creates, and perhaps how to customize them beyond the defaults supported by the interface? Where does pfSense store its settings? What are the command line tools for exploring the specifics of the firewall it creates?

    Looking at System > Advanced > Firewall & NAT > Network Address Translation, we currently have nothing set. Are any of these pertinent to getting NAT to work from a pfSense box which is not the default gateway for the LAN? Which proxy software exactly is on offer through this menu? It says it doesn't handle UDP well, so it wouldn't be suited for DNS. But can proxying of external traffic be achieved in, and if so through this menu (similarly to how xinetd can be a port-forwarding proxy for TCP in Linux)? Are there other BSD proxies which wouldn't have the TCP-only limitation?

    Whatever the issue is, it is probably listed here:

    https://doc.pfsense.org/index.php/Port_Forward_Troubleshooting

    A number of those suggestions refer to the "reply-to" setting. Where might I find a technical explanation of how that's implemented, and specifically what it's capable of facilitating?

    Thanks again!

    Whit


  • Netgate

    Yeah if you port forward from pfSense, the reply traffic needs to go back to pfSense.

    This is not something that has to change on pfSense but on the target host, since it's the one (properly) sending reply traffic according to the its routing table.

    You can make the traffic related to the port forward appear to come from something on the same subnet by using outbound NAT on the inside pfSense interface - the one that has the target host on it. Limit the outbound NAT to traffic sourced from any destined for the target host on the target port.

    That host will lose the ability to see that actual source address of the connection - they will all appear to come from pfSense.

    replies will then be same-subnet so the routing table on the host will not be used.



  • Thanks for advice. I've simplified the network so that the inside server is on a subnet shared with pfSense. Still having a problem.

    pfSense sees only the traffic coming in, 3 tries like this, the external interface shows:

    16:47:04.259561 IP 207.136.236.70.45347 > <public ip="">: UDP, length 44

    and the internal interface shows:

    16:57:58.367616 IP 207.136.236.70.40273 > 172.17.19.54.53: UDP, length 44

    And with pfSense set to log the NAT rule, it does show that traffic as accepted of course.

    It's getting NAT'd into the DNS server behind that, which replies to pfSense (now on same subnet), again 3 sets of these

    UDP (72 bytes) from 207.136.236.70:45347 to 172.17.19.54:53 on eth1
      UDP (72 bytes) from 172.17.19.54:53 to 207.136.236.70:45347 on eth1

    So the DNS server gets the request, and sends a reply back on the same interface. I know that port 53 is available from pfSense's POV, as the test for that shows it responding (as indeed the lines above show it responding to the NAT'd request).

    Now, there are two interfaces and routing tables on the Linux DNS server in question here. As the above shows it's responding on eth1, as it should, since the routing table in question says:

    default via 172.17.19.1 dev eth1
      172.17.19.0/24 dev eth1  scope link  src 172.17.19.54

    and the rule for that is:

    32764: from all to 172.17.19.54 lookup dmz
      32765: from 172.17.19.54 lookup dmz

    Now, the pfSense firewall this is coming in on is on 172.17.19.2 in this case, not on the default 172.17.19.1. With 2 pfSense firewalls, I want the ability to NAT from both of them, from different public interfaces, at the same time. And I need them to each be able to fail over to the other. Is there any way to handle that on the pfSense side – assuming the problem now is I need to have it route from pfSense on 172.17.19.1 to 172.17.19.2 -- but not have that happen if 172.17.19.2 is down and the pfSense box on 172.17.19.1 has taken over the public interface from it? (172.17.19.1 in this case is also a CARP IP.)

    I'm thinking I probably need yet more distinct routing tables in the DNS server to get the default return address right. I was hoping that would follow MAC rather than IP with these on the same subnet now. Evidently not.

    Just in passing, I was disappointed to see that FreeBSD's multiple routing tables aren't supported in pfSense.</public>



  • Think what I'll have to do is have CARP interfaces in both directions on the DMZ side of the pfSenses, then have a second IP on the Linux DNS server, and have a separate routing table for each of its IPs, to go back to the proper pfSense box.

    Seems like that should work.


  • Netgate

    UDP (72 bytes) from 207.136.236.70:45347 to 172.17.19.54:53 on eth1
      UDP (72 bytes) from 172.17.19.54:53 to 207.136.236.70:45347 on eth1

    Look at the source MAC addresses of the inbound traffic and the dest MAC address of the reply traffic there.

    I would create a transit network between the two routers instead of putting the other router on LAN.