Load balance Nat port forward not working

billiho · Dec 16, 2010, 8:18 PM

Hi all,

1 I am running ver 1.2.3. I have it set up for a hotel. I have set up load balancing, captive portal, and I need access to 3 AP wireless units, A separate wireless router, a vnc computer, a network DVR, a network camera and other stuff.

I set up load balancing and it looks like it's working but nothing goes out WAN2. I have WAN1 with a ISP supplied static address of xxx.xxx.1.79/22, with a gateway of xxx.xxxx.0.1, and when I add WAN1 to load balancing it shows up as xxx.xxx.0.1, not xxx.xxx.1.79?.

I have WAN2 setup using a dlink router so the 2 WAN ports don't have the same subnet and I have some other stuff plugged into the lan ports of the dlink router so I can get to those things without having to access the "guest" network. WAn2 of the pfsense router gets a reserved dynamic address of 192.168.3.2 from the dlink. The dlink router in front of the WAN2 connection has a dynamic ip address of xx.xx.12.54. Both connections are working and I can access the pfsense router from either wan.

I set up the load balance according to the page I saw here:

http://www.tomschaefer.org/web/wordpress/?p=538

I set up the captive portal from this tutorial:

http://doc.pfsense.org/smiller/Captive_Portal.htm

I set up the systems I want to remote access as NAT port forwards. some seem to work, and others don't although they are set up for the correct protocol, correct port, and ip address. the AP units are 3com and I can't get access remotely, but the wireless router I get access no problem, and I have a vnc connection that works. I removed 2 of the 3 AP units in rules to see if I could get one working.

Typical setup for the wireless and the 3com AP's is

192.168.1.60:8000 access to a wireless router. I setup
interface=WAN
External address=Interface Address
protocol=TCP,
Port= other (8080) or whatever port
NAT ip=192.168.1.60, or whatever address
local port=8080, or whatever port
description=router

So....in summary, WAN1 seems to take almost all, if not all the traffic, NAT seems to sort of work but not for everything, and I'm feeling a bit frustrated! When I do a speedtest I get about 22mb second but that's excluding my laptop from Captive Portal using a MAC exclusion. I get 1mb/sec on a captive portal client, which is what I set up for bandwidth for CP clients. Any computer sees only the WAN1 address if I do a whatsmyip

Any help would be great. I hope I've supplied enough info.

Thanks
Bill

billiho · Dec 16, 2010, 9:35 PM

It looks like ver 1.2.3 can't do load balancing on captive portal clients. So that's one problem resolved. On to the next….

jimp · Dec 17, 2010, 4:43 PM

Sure it can, just add the IP of the internal servers in as Captive Portal IP bypasses, two entries each: one 'to' and one 'from' and then it should work.

billiho · Dec 17, 2010, 5:06 PM

I used pass thru mac on the Captive Portal page to set up those clients that will bypass Captive Portal and get full load balancing, and that seems to work fine. I can get on a bypass client and go to whatsmyip and if I hit refresh a bunch of times it flips between the two WANs, so that's fine for them.

What I originally wanted to do was to have the Captive Portal clients use the load balancing, but it appears that captive portal doesn't work with load balancing on ver 1.2.3, but will work on 2.0 when it's ready. I could be wrong, and quite frankly I'd like to be wrong. I removed a linksys RV042 that did load balancing so that I could implement captive portal and throttle captive portal clients, but that's going to have to wait until 2.0 is really stable, I guess. I'm debating whether it's better to have clients using one WAN port with Captive portal using the pfsense, or use the old RV042 with load balancing and no throttling, or maybe an combination if I can figure it out and it works reliably.

My biggest issue right now seems to be using the NAT forwarding. I have tried a whole bunch of stuff to get remote access to more then one device using https. The first device, a router on port 8080 is set up for https access and I can access it no problem. I am trying to set up access for 3 more devices that also use https. I have tried all sorts of stuff and nothing seems to work. I'm starting to wonder if a protocol can be used only once, even if I use different port numbers. The device that works is at 8080, the pfsense router itself is at 9000, the other devices are at 9100-9103, everything is https, and all are tcp. I can access all devices no problem from inside the network, and I could access them all on the old Linksys rv042 from outside no problem. The NAT, and rules should be the same as the existing working connection except using different internal ip addresses and port numbers.

So why the (*&)( can't I access these other devices outside the network??

billiho · Dec 17, 2010, 6:51 PM

I don't think the pfsense is going to work in this application. A simple problem can't be solved by the look of it. NATting should be a pretty simple matter, but not only does it not seem to work right, except for one vague response no one seems to be able to tell me why or how to fix it. Captive portal works, but not with load balancing, unless I wait for version 2.0 to stabilize.

All I see on these forums are a bunch of questions with vague or no answers, and that doesn't help anyone. I guess the real gurus are busy fixing their own problems.

I don't know how anyone could suggest using this in a business environment unless their needs are simple or they want to have a geek on staff to take care of the firewall full time. If I have a problem there's no one to turn to unless I want to pay someone for support, and even with support it may not work the way I want.

It's too bad, because I like the concept of open source, it's the execution that falls short. Those of you that think otherwise don't understand that time=money and time wasted=lost money. I have probably spent 15-20 hours to get nowhere, plus the cost of buying a netgate router. This ends up being an expensive waste of time.

The "crappy" Linksys RV042 router that I replaced worked perfectly, except it didn't have a captive portal. It took a couple hours to completely set up and tweak, and it has been in place for about 3 years without a hiccup. So why do I want the pfsense? Because it's difficult to setup, doesn't work quite right, and there's no support. Gee, those are great reasons.

I would love to see people respond and tell me how wonderful open source is, how it can do anything and be customized, etc. The only thing wrong with that school of thought is the basic problem that no one has yet been able to answer a simple question.

jimp · Dec 17, 2010, 7:25 PM

Patience and a good attitude go a long way toward getting help from a volunteer forum. Lots of people use it in high-end environments with much success.

My answer earlier was a bit vague, but I was referring to the port forward portion. A side effect of skimming, which is my fault. With port forwards you need IP bypass not MAC bypass. MAC bypass works like a portal auth, it requires a periodic web access to renew the 'session' - an IP bypass does not suffer from this.

Captive Portal and Multi-WAN is a bit trickier but I haven't tried it, as you said it's likely only possible on 2.0 (which many people are also using in production… with caution)

Why not just let the RV042 handle the multi-wan bit and let pfSense do the portal? You should be able to setup port forwards/1:1 nat in the RV042 to pass your forwarded ports back to pfSense and then inside. Or disable NAT on pfSense if the RV042 can handle multiple internal subnets with static routes. Or just use 2.0 (though captive portal is broken in current snapshots a fix is going in as I type this).

Tens of thousands of people use pfSense, everything from home users, businesses of all sizes, enterprise class, you name it. Commercial support is available for people who need immediate professional consultation (or who just want to donate to the project and get something for their money) and we have no shortage of customers.

Just because it didn't do exactly what you wanted in one specific scenario in a -release (and the beta can) is no reason to talk trash about the project as a whole for the needs of everyone else.

billiho · Dec 17, 2010, 8:26 PM

yes, I am dumping, and I apologise, but that's because I'm rapidly going nowhere and feeling frustrated.

I tried your suggestion with no luck. Does the router need a reboot every time I make a change? That means I have to kick everyone off it, and that's going to land me in hot water pretty quick. I just tried a reset but it made no difference.

Here is the settings I changed. What I don't understand is why one works and the other doesn't. If I'm going to 192.168.1.60 via https port 8080 it works. If I go to 192.168.1.100 https port 9100 it doesn't? Setups are the same, I can access from the lan np, the rules should be set to allow access. I ping it and get a reply. I know it's working because there's a bunch of clients going thru it, but I can't get to it? WTF?????

jimp · Dec 17, 2010, 8:28 PM

Settings should take effect right away in most cases, especially when it comes to port forwards.

I noticed in your port forward definitions the 9100 is set for tcp/udp. Have you tried setting that to just tcp?

billiho · Dec 17, 2010, 8:29 PM

tried tcp and tcp/udp, no difference.

jimp · Dec 17, 2010, 8:36 PM

Have you gone over all of the usual troubleshooting steps?

http://doc.pfsense.org/index.php/Port_Forward_Troubleshooting

Usually if one works and the other doesn't, and the setup is identical, it ends up being something on that machine (not using pfSense as its gateway, local software firewall, etc)

Some packet captures on the WAN and LAN sides would help narrow it down, you could see how the traffic is (or isn't) flowing to that server.

billiho · Dec 17, 2010, 9:09 PM

Just set Wan1 to turn on logs. Sure enough it's blocking. It says:

@62 block drop in log quick all label "Default block all just to be sure."

turned on logs for the porting that worked. It said:

@49 pass in log quick on vr1 inet proto tcp from any to 192.168.1.60 port = 8080 flags S/SA keep state label "USER_RULE: NAT cannery Router"

Cative portal ip exclusions look like:

![captive-Portal_3_revenge_of _the_pfsense.png](/public/imported_attachments/1/captive-Portal_3_revenge_of _the_pfsense.png)
![captive-Portal_3_revenge_of _the_pfsense.png_thumb](/public/imported_attachments/1/captive-Portal_3_revenge_of _the_pfsense.png_thumb)

billiho · Dec 17, 2010, 9:21 PM

more detailed log info

bad response:
pf: 3. 008448 rule 62/0(match): block in on vr1: (tos 0x0, ttl 124, id 279, offset 0, flags [DF], proto TCP (6), length 48) xxx.xxx.53.7.2039 > xxx.xxx.1.79.9100: tcp 16 [bad hdr length 12 - too short, < 20]

Good response:
pf: 068366 rule 49/0(match): pass in on vr1: (tos 0x0, ttl 124, id 24514, offset 0, flags [DF], proto TCP (6), length 48) xxx.xxx.53.7.2071 > 192.168.1.60.8080: tcp 16 [bad hdr length 12 - too short, < 20]

It seems that the bad one has the extenal IP adderss, the good one has the internal IP address. Is that because it has passed thru?

Could it be because 192.168.1.60 gets dhcp and 192.168.1.100 doesn't? That would be weird.

billiho · Dec 17, 2010, 9:39 PM

I have been through the troubleshooting a few times. I have tried a bunch of different settings.

I dunno.

billiho · Dec 18, 2010, 1:35 AM

I finally figured out everything and it works! Yay!

The key was that I had to set up NAT port forwards, which sets up the rules, AND I had to set up captive portal allowed IP addresses. I had thought I didn't need the NAT forwards because I had the Captive portal allowed IP addresses and the rules set up. So MAC=bad, IP=good when setting up Captive portal NAT forwards.

So, I need to apologize for dumping on the pfsense, and thanks to jimp. I was getting really frustrated at not getting anywhere after spending about 3 days trying to get the port forwards working. What was screwing me up was that some of the forwards worked, and some didn't. The first 3 rules worked fine with Pass Thru MAC, but the other 3-4 rules didn't work, even though they were set up the same. I guess that's just one of those oddities with the system. Once you know these things it starts to make sense.

I don't know if anyone can update the tutorials, but that sort of key info makes life much, much easier.

jimp · Dec 20, 2010, 6:46 PM

Unfortunately both of the howtos you linked are contributed and aren't on our Wiki or I could update them rather easily.

I did add a note here:
http://doc.pfsense.org/index.php/Port_Forward_Troubleshooting#Common_Problems

About needing an IP bypass for a port forward to a server behind a captive portal.