*Solved* Multi-wan and port-forwards *Subject Edited*



  • –Edit -- See my last post as to what was my problem.
    Devnull22

    Hello all, just launched myself in the world of pfsense, and am having some troubles achieving what I set out to do. First off, I'm a jack of all trades, master of none due to the broad range of things I do at work, so I do know a fair bit about networking and firewalls, just not a master at any of it!

    I'll first try and depict what I want to do, so it'll be clearer when I explain my problem:

    Internet
                             /          
                           /              
                         /                  
      lab  ---  pix 515 (adsl)       asa5505 (100 mpbs fiber)
                     |                          |
              10.17.17.0/24         10.110.0.0/16
                     |                          | (10 mbps lan extension)
                     |                          |
            pfsense opt1  -sync-    pfsense wan
                     \                         /
                       \                     /
                         \                 /
                            pfsense lan
                                  |
                                  |
                services, like OpenVPN AS , web services (wiki) and others than need to be reached from either the 10.110.0.0/16 range or the Internet

    The pfsense boxes (two of them in failover with carp VIPs) work fine, and I've got a load balancing set up so that the link with the fiber lan extension is used first, then fails over to the second link. This is intentional as the other link on the pix is a 100 mpbs link, and we have a 10 mpbs lan extension to that network, where most of our work servers are.

    The pfsense boxes are in our main office, the servers we host are in hosting sites. We have some services we need open on both links, (like openvpn as) so that if one link fails, we can still reach them through the other link (dsl for example). This is where it doesn't seem to be working as I expect. I've set up port forwards, in the NAT section, and the corresponding firewall rules, for both WAN and opt1. It's working fine for WAN, but times-out for opt1.

    I imagine that what's causing me these problems is the outbound rules I have that forces traffic out on WAN first if available, and opt1 only if it failed. Hence, these questions that I hope you guys can help answer!

    • If a port forward is set up on both wan and opt1, will pfsense keep states of which IP connected on which interface and answer from that interface? (we have a peplink device that did just that, which I'm trying to replace)
    • If a load balancer is installed to prioritize traffic on WAN, will it mean port forwards on opt1 won't work if the WAN is up, because of routing on outgoing?
    • Proxy Arps VIPs don't seem to propagate through sync, and are not good for failover, as I've read before, but other VIPs than CARP just don't seem to work for port forwards, even though I can select them, it doesn't work until I changed it to CARP or PARP, but I've got a limit of 256 CARP. My next setup will need possibly more (I want to be able to scale to about 500 VIPs, available from 2 internet connections. To be clearer... citrix servers or rdp servers that will need to be reached from 2 different internet connections, but can each only have one gateway defined.)

    I can clarify things if you guys need me to, I'm just a bit new so don't exactly know what info to give or how to get it, but with a bit of help (and patience ;-) I can find it out and give it to ya!

    Thanks in advance,

    Devnull



  • I've managed to confirm what I was thinking, when the packets from the internet enter the pix, then the opt1 interface, they go back out through the WAN interface (tcpdump on em1, the wan interface, confirms this).

    connection sent to opt1, or em2, but tcpdump on em1, removed some info (external ip address)

    tcpdump -i em1 | grep 96

    tcpdump: verbose output suppressed, use -v or -vv for full protocol decode                                                                                     
    listening on em1, link-type EN10MB (Ethernet), capture size 96 bytes                                                                                           
    –- cut for brevity ---                                                                 
    15:27:17.758355 IP 10.1.17.15.https > modemcable*.--96....59891: S 392713312:392713312(0) ack 11253854 win 5792 <mss 5="" 312338118="" 1460,sackok,timestamp="" 60861812,nop,wscale="">                                                                                                                                 
    15:27:17.759573 IP modemcable
    .--96...*.59891 > 10.1.17.15.https: R 1:1(0) ack 1 win 5792                                                   
    --- cut again for brevity---
    113 packets received by filter                                                                                                                                 
    0 packets dropped by kernel

    Aside from static routes to ensure that packets for our vpn networks (ipsec for customer support) the only outbound route on the lan rules use the load balancer. I've tried inverting this to use the dsl link before the other link, to no avail. Packets still go out on the WAN link. I imagine this is due to the default gateway pointing on my gateway on em1 (WAN)

    Is this normal behaviour or have I possibly put something bad in my config? Anyone have a clue as to how I can determine where this is going wrong?

    Thanks in advance,

    Devnull</mss>



  • Okay, done a lot more testing, and here's more details:

    I have removed the load-balancers to isolate the problems. The setup is now like this:

    Internet
                             /          
                           /              
                         /                  
      lab  –-  pix 515 (adsl)       asa5505 (100 mpbs fiber)
                     |                          |
              10.17.17.0/24         10.110.0.0/16
                     |                          | (10 mbps lan extension)
                     |                          |
            pfsense opt1  -sync-    pfsense wan
                     \                         /
                       \                     /
                         \                 /
                            pfsense lan
                                  |
                                  |
                        OpenVPN AS

    The pfsense router is connected to 2 networks, one local, the other a lan extension (10 mbps fiber), both secured by a firewall, that maps ips 1 to 1 (cisco pix and asa 5510, both of which I know are configured correctly since our current peplink works fine now with the same setup)

    So the internet passes through one firewall, reaches the pfsense router, which has virtual IP defined as CARP ip addresses for services, and port forwards configured along with firewall rules to let this pass. I can confirm the rules for the wan work just as planned. The one for OPT1 (pix firewall + static dsl) also,with one big but:

    • The port forward on OPT1 is passed to the internal IP, but then, the traffic goes back to pfsense, and is routed through the WAN link.

    TCPdumps here, the attached images are my setup on pfsense:

    PFSense: em2 (pixdsl, or OPT1) command used: tcpdump -n -i em2 src or dst 96.XX.XX.XX (my home ip address)
    11:42:27.885064 IP 96.XX.XX.XX.45501 > 10.17.17.15.443: S 1525146743:1525146743(0) win 5840 <mss 7="" 2630964="" 1380,sackok,timestamp="" 0,nop,wscale="">11:42:30.885730 IP 96.XX.XX.XX.45501 > 10.17.17.15.443: S 1525146743:1525146743(0) win 5840 <mss 7="" 2631714="" 1380,sackok,timestamp="" 0,nop,wscale="">11:42:36.886230 IP 96.XX.XX.XX.45501 > 10.17.17.15.443: S 1525146743:1525146743(0) win 5840 <mss 7="" 2633214="" 1380,sackok,timestamp="" 0,nop,wscale="">11:42:48.890129 IP 96.XX.XX.XX.45501 > 10.17.17.15.443: S 1525146743:1525146743(0) win 5840 <mss 7="" 2636214="" 1380,sackok,timestamp="" 0,nop,wscale="">PFSense: em1 (wan) command used: tcpdump -n -i em1 src or dst 96.XX.XX.XX.XX (my home ip address)
    11:42:27.885858 IP 10.1.17.15.443 > 96.XX.XX.XX.45501: S 3402657138:3402657138  (0) ack 1525146744 win 5792 <mss 5="" 345635797="" 1460,sackok,timestamp="" 2630964,nop,wsc=""  ale="">11:42:27.887211 IP 96.XX.XX.XX.45501 > 10.1.17.15.443: R 1:1(0) ack 1 win 5792
    11:42:30.886198 IP 10.1.17.15.443 > 96.XX.XX.XX.45501: S 3402657138:3402657138  (0) ack 1525146744 win 5792 <mss 5="" 345636097="" 1460,sackok,timestamp="" 2630964,nop,wsc=""  ale="">11:42:30.887551 IP 96.XX.XX.XX.45501 > 10.1.17.15.443: R 1:1(0) ack 1 win 5792
    11:42:31.682849 IP 10.1.17.15.443 > 96.XX.XX.XX.45501: S 3402657138:3402657138  (0) ack 1525146744 win 5792 <mss 5="" 345636177="" 1460,sackok,timestamp="" 2630964,nop,wsc=""  ale="">11:42:31.684130 IP 96.XX.XX.XX.45501 > 10.1.17.15.443: R 1:1(0) ack 1 win 5792
    11:42:36.886795 IP 10.1.17.15.443 > 96.XX.XX.XX.45501: S 3402657138:3402657138  (0) ack 1525146744 win 5792 <mss 5="" 345636697="" 1460,sackok,timestamp="" 2630964,nop,wsc=""  ale="">11:42:36.888240 IP 96.XX.XX.XX.45501 > 10.1.17.15.443: R 1:1(0) ack 1 win 5792
    11:42:38.084969 IP 10.1.17.15.443 > 96.XX.XX.XX.45501: S 3402657138:3402657138  (0) ack 1525146744 win 5792 <mss 5="" 345636817="" 1460,sackok,timestamp="" 2630964,nop,wsc=""  ale="">11:42:38.086213 IP 96.XX.XX.XX.45501 > 10.1.17.15.443: R 1:1(0) ack 1 win 5792
    11:42:48.890580 IP 10.1.17.15.443 > 96.XX.XX.XX.45501: S 3402657138:3402657138  (0) ack 1525146744 win 5792 <mss 5="" 345637897="" 1460,sackok,timestamp="" 2630964,nop,wsc=""  ale="">11:42:48.891904 IP 96.XX.XX.XX.45501 > 10.1.17.15.443: R 1:1(0) ack 1 win 5792
    11:42:50.889730 IP 10.1.17.15.443 > 96.XX.XX.XX.45501: S 3402657138:3402657138  (0) ack 1525146744 win 5792 <mss 5="" 345638097="" 1460,sackok,timestamp="" 2630964,nop,wsc=""  ale="">11:42:50.891008 IP 96.XX.XX.XX.45501 > 10.1.17.15.443: R 1:1(0) ack 1 win 5792
    11:43:14.898576 IP 10.1.17.15.443 > 96.XX.XX.XX.45501: S 3402657138:3402657138  (0) ack 1525146744 win 5792 <mss 5="" 345640497="" 1460,sackok,timestamp="" 2630964,nop,wsc=""  ale="">11:43:14.899764 IP 96.XX.XX.XX.45501 > 10.1.17.15.443: R 1:1(0) ack 1 win 5792

    OpenVPN AS server tcpdump: command : tcpdump -n -i eth0 src or dst 96.XX.XX.XX
    11:40:28.795269 IP 96.XX.XX.XX.45501 > 10.1.17.15.443: S 1525146743:1525146743   (0) win 5840 <mss 7="" 2630964="" 1380,sackok,timestamp="" 0,nop,wscale="">11:40:28.795472 IP 10.1.17.15.443 > 96.XX.XX.XX.45501: S 3402657138:3402657138   (0) ack 1525146744 win 5792 <mss 5="" 345635797="" 1460,sackok,timestamp="" 2630964,nop,wsc=""  ="" ale="">11:40:31.794452 IP 96.XX.XX.XX.45501 > 10.1.17.15.443: S 1525146743:1525146743   (0) win 5840 <mss 7="" 2631714="" 1380,sackok,timestamp="" 0,nop,wscale="">11:40:31.794511 IP 10.1.17.15.443 > 96.XX.XX.XX.45501: S 3402657138:3402657138   (0) ack 1525146744 win 5792 <mss 5="" 345636097="" 1460,sackok,timestamp="" 2630964,nop,wsc=""  ="" ale="">11:40:32.590793 IP 10.1.17.15.443 > 96.XX.XX.XX.45501: S 3402657138:3402657138   (0) ack 1525146744 win 5792 <mss 5="" 345636177="" 1460,sackok,timestamp="" 2630964,nop,wsc=""  ="" ale="">11:40:37.792879 IP 96.XX.XX.XX.45501 > 10.1.17.15.443: S 1525146743:1525146743   (0) win 5840 <mss 7="" 2633214="" 1380,sackok,timestamp="" 0,nop,wscale="">11:40:37.792961 IP 10.1.17.15.443 > 96.XX.XX.XX.45501: S 3402657138:3402657138   (0) ack 1525146744 win 5792 <mss 5="" 345636697="" 1460,sackok,timestamp="" 2630964,nop,wsc=""  ="" ale="">11:40:38.990661 IP 10.1.17.15.443 > 96.XX.XX.XX.45501: S 3402657138:3402657138   (0) ack 1525146744 win 5792 <mss 5="" 345636817="" 1460,sackok,timestamp="" 2630964,nop,wsc=""  ="" ale="">11:40:49.792240 IP 96.XX.XX.XX.45501 > 10.1.17.15.443: S 1525146743:1525146743   (0) win 5840 <mss 7="" 2636214="" 1380,sackok,timestamp="" 0,nop,wscale="">11:40:49.792318 IP 10.1.17.15.443 > 96.XX.XX.XX.45501: S 3402657138:3402657138   (0) ack 1525146744 win 5792 <mss 5="" 345637897="" 1460,sackok,timestamp="" 2630964,nop,wsc=""  ="" ale="">11:40:51.790597 IP 10.1.17.15.443 > 96.XX.XX.XX.45501: S 3402657138:3402657138   (0) ack 1525146744 win 5792 <mss 5="" 345638097="" 1460,sackok,timestamp="" 2630964,nop,wsc=""  ="" ale="">11:41:15.790590 IP 10.1.17.15.443 > 96.XX.XX.XX.45501: S 3402657138:3402657138   (0) ack 1525146744 win 5792 <mss 5="" 345640497="" 1460,sackok,timestamp="" 2630964,nop,wsc=""  ="" ale="">11:42:03.790760 IP 10.1.17.15.443 > 96.XX.XX.XX.45501: S 3402657138:3402657138   (0) ack 1525146744 win 5792 <mss 5="" 345645297="" 1460,sackok,timestamp="" 2630964,nop,wsc=""  ="" ale="">edit: forgot to mask one public ip



    ![firewall lan.png](/public/imported_attachments/1/firewall lan.png)
    ![firewall lan.png_thumb](/public/imported_attachments/1/firewall lan.png_thumb)
    ![firewall pixdsl.png](/public/imported_attachments/1/firewall pixdsl.png)
    ![firewall pixdsl.png_thumb](/public/imported_attachments/1/firewall pixdsl.png_thumb)
    ![firewall wan.png](/public/imported_attachments/1/firewall wan.png)
    ![firewall wan.png_thumb](/public/imported_attachments/1/firewall wan.png_thumb)
    ![outbound nat.png](/public/imported_attachments/1/outbound nat.png)
    ![outbound nat.png_thumb](/public/imported_attachments/1/outbound nat.png_thumb)
    ![port forwards.png](/public/imported_attachments/1/port forwards.png)
    ![port forwards.png_thumb](/public/imported_attachments/1/port forwards.png_thumb)
    ![static routes.png](/public/imported_attachments/1/static routes.png)
    ![static routes.png_thumb](/public/imported_attachments/1/static routes.png_thumb)</mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss>



  • New network graphic as attachment




  • If you have > 255 public IPs/VIPs, and are using CARP, you need to route the additional subnets to one of your CARP IPs and add them as Other VIPs.

    Traffic coming in on a port forward will get routed back out the same WAN (assuming you're using 1.2.1 or newer) with the reply-to that gets added to the rules. If you have static routes on the WAN or OPT WAN side pointing to something other than the gateway on that interface, that complicates things, as you'll want the reply-to for most traffic, but in the case of the static route that will cause the reply traffic to go to the gateway on that interface. At a quick glance, I don't think that's an issue here.

    Unless you changed IPs in your tcpdump, the SYN ACKs are coming from a different source IP than the ACKs were destined to. That would make the traffic not match the state and hence not follow the correct path.



  • as far as I can tell, everything should work, as the routes I have put in are mostly to ensure traffic meant for an IPSEC VPN managed by the pix (gateway of OPT1) gets sent there, and for a separate subnet on WAN also.

    I've had someone on the IRC channel comment on this:

    there is a special case when the reply-to will not work in pfSense and that is only when it tries to reply-to to a traffic local to pfSense

    I have not managed to find out more about this, but he mentionned I probably have fallen on this problem. I have no idea what to do next as anything I try results in the same issue, and getting rid of the pix and ASA is not an issue due to the setup at work, and that fact that the peplink worked quite well, so making major chances will be refused by management since we had something working first with that setup…

    Pastebin of rules.debug: http://pastebin.com/m704e80c1

    As for the tcpdumps, I only changed every instance of my personal home ip to 96.XX.XX.XX, the other private ips have not been changed, since they're not public. Which one was not okay, making the states invalid, if I may ask?

    I will forget the problem of 255 carp ips max so far as it's not as urgent as getting port forwards to work correctly (route in accordance to reply-to rules that are okay in rules.debug)

    Thanks,



  • Pretty sure Ermal was the one who commented on that in IRC, he sent an email to our developer list with a link to this thread making a comment that this scenario is a problem, but I don't see at a glance what he's talking about, I think he's possibly confusing this with something else. He didn't reply yet when I asked how exactly it's a problem. We bypass reply-to when traffic is sourced from a local subnet, which is what he was referring to, but your 96.x.x.x IP is nowhere near a local subnet. (speaking of 96.x.x.x, you forgot the X's in this line: "OpenVPN AS server tcpdump: command : tcpdump -n -i eth0 src or dst 96", might want to edit your post)

    Scratch my earlier comment on the SYN ACK being sourced from a different IP than the SYN was sent to, I misread your captures at a glance previously.

    I'm bumping this thread on our developer list to see what Ermal was referring to.



  • We have a special provison that adds negate rules for rules that have a gateway item set.

    It will however only trigger for traffic that has a destination of any. For example:
    allow from lannet to any gateway dsl
    will create a number of negate rules that look like this
    allow from lannet to <localnetworks>Which are added above the original rule so that you can actually reach your own external network and services on pfSense.

    This however does not apply to traffic coming in from the internet to the host.

    This kind of surprises me though. My setup is not so wildly different.

    I have a external pfSense carp cluster, with 2 connections and webservices hitting either connection. I have no issue terminating any traffic, be that smtp, http or https.
    Heck, even udp dns works fine.

    What I do have though is that the DMZ host has multiple addresses under a 1:1 NAT.
    The DNS is advertised with both x.x.3.195 and x.x.4.99, the DMZ DNS server is a private range subnet and has both a 10.0.6.99 as well as a 10.0.6.195.
    I then have 2 1:1 NAT rules, one connecting the 195 address on WAN and another 1:1 for connecting the 99 to the opt1 address.

    So far this has proved to work reliably of the past few years. Maybe this information is valuable to you.

    Internet
                              /         
                            /             
                          /                 
                  cisco1811_2          cisco1811_1
                      |                          |
              x.x.4.97/27            x.x.3.193/27
                      |                          |
                      |                          |
            pfsense ext opt1  -sync-    pfsense ext wan
                      \                        /
                        \                    /
                          \                /
                            pfsense ext carp –-- DMZ Web Server
                                  |
                                  |
        Internal to External network bridge (single link)</localnetworks>



  • cmb: Thanks, I'll be waiting for a bit, as this is for our internal needs but might be used for a bigger setup with a lot of citrix servers behind them (need 2 WANs for link failover, as well as increased possible incoming bandwith. Outgoing bandwith will be of less concerns, so failover will be okay instead of complete load-balancing for outgoing)

    databeesje: Thanks for the comment, I'll see if I can try it, but unfortunately, I doubt it will help as the same server needs to be reached from the two wan links, mapped to the same internal IP (citrix servers don't play well in a farm with multiple IPs in our setup) so 1:1 nat will be hard to implement as I doubt I can have the same mapping on both wan and opt1. It does give me food for tought though, and I'll try and improvise to see if it works.



  • It works fine for port forwards on both WANs too, I've set that up countless times and run it on my own network, not sure how in your scenario is any different. No response from Ermal yet on what he was thinking about re: his earlier comment.



  • I'll try installing a new server with a single server set up, make sure port-forward works, save config, and go from here adding each feature we need and see if that works.

    I'll save config each time and go a step at a time till either it works, or it breaks ;-)
    Thanks for your time on this, I'll keep this thread updated if I can find out what didn't work, or what caused this…



  • Okay, I managed to have enough time to a) reinstall the server completely, and start from scratch and b) have enough downtime on a couple of servers to be able to change their default gateway to the new install to test.

    After having some problems with opt1 not working at all, a reset to defaults and a re-configuration gave me a working config on both links, albeit without failover and load-balancing. But now, I managed to set up port-forwarding and it's working okay, people are actually working from home on the vpn, some on wan, some on opt1.

    I'll keep this thread updated once I manage to get some time to reinstall the load-balancers and the failover server (the CARP ips are all set up already and being used, so I doubt failover was the problem, or will cause me problems). We'll see if any of those two will give me problems, or if I'll have a complete working setup!



  • Just a quick update To state that I was stuck because of this bug back then:
    http://redmine.pfsense.org/issues/show/111

    Solution listed by cmb
    http://forum.pfsense.org/index.php/topic,19763.msg101819.html#msg101819

    My home IP used to end in .225, it changed to a new range when I had my cable modem changed due to phone issues (voip at home from our provider) That coincided with the reinstall of the setup.

    Funny how things turn out =)


Log in to reply