Assigning multiple WAN IP's to multiple interfaces

[dcol]

Hi, all,
I am new to PFSense but not to routing/firewalls.
Trying, with no success, to assign multiple WAN IP's to multiple interfaces.
My ISP assigned four IP's to me - example 101.102.103.114,117,124,125  GW:100.101.102.113/28
My system has 5 NIC ports labeled WAN, LAN, SERVER1, SERVER2, SERVER3
The goal is to connect the ISP to the WAN port, then each remaining interface with be assigned its own IP.
The interface gateways are setup as follows
LAN - 192.168.1.1 - used as local LAN for internal computers
SERVER1 - 192.168.10.1 - Web server 1
SERVER2 - 192.168.20.1 - Web server 2
SERVER3 - 192.168.30.1 - Email server
SERVER interfaces plug directly into the servers and use subnet #2 as their local IP.

I should not need VLAN's since I have physical NIC ports for each WAN IP.
The reason for the multiple interfaces is to have more firewall control and bandwidth.
I have tried many NAT 1:1 and outbound configuration with no luck.
I setup the WAN and LAN first and that seems to work fine. Just can't get the rest of the IP's assigned to their respective SERVER interfaces.

Can anyone help?
Thanks
Dan

[tim.mcmanus]

If I understand you correctly, you're only using one NIC as a WAN port, and the other NICs are essentially LANs (because of the 192.168.x.x addressing).

You want to assign a single, static address to your WAN interface. Then use virtual IPs to allow that WAN NIC to use multiple addresses. Then you'd create firewall/NAT rules that say something like "when a packet comes in to address 101.102.103.117, route to SERVER1 LAN" so forth and so on.

I'm not onsite, but that's the way I have it set up.

[dcol]

I have multiple IP's via one WAN port. and want to assign each assigned WAN IP to it's own NIC interface.

I did try to setup a manual outbound NAT using the WAN IP to the SERVER interface NIC.
What happened was I could go outbound, ie. internet connection. But could not access that server from the LAN.
On the internal network I could not ping the server. From an outside location, I got the "DNS Rebind attack" error, which I will follow up on. Probably need to setup a DNS forwarder rule, I hear.

So how do I get the other interfaces to talk to each other, or the LAN to access the SERVER interfaces.

[Derelict]

Yeah, you can't really.  Interfaces need a subnet of at least a /31 in pfSense. (and every other layer 3 IP router).

You can 1:1 NAT with VIPs for each server on the outside interface forwarded to inside addresses.

Or you can port forward ports 80 and 443 on different outside VIPs to different inside servers, which will leave other ports on all the VIPs free to do other things.

Unless you really need the servers isolated from each other I think you are adding unnecessary complexity (to start with) by doing all the different interfaces.

And unless you really need the 1:1, I would put all the servers on one interface (using a switch) and port forward to them.

And test your NAT from outside, not inside.

https://doc.pfsense.org/index.php/Port_Forward_Troubleshooting

[dcol]

All the interfaces are setup as /24. Then I use an address within the subnet for the server.
Seems to work fine for outgoing/browsing. Just can't access a website on the server internally or externally.
DNS Forwarder errors with 'The DNS Resolver is enabled using this port. Choose a non-conflicting port, or disable DNS Resolver' when I try to enable it.
Am I missing something?

[Derelict]

Sounds like you're missing about everything. I really don't know where to start.

For the last error you posted only one DNS server can listen on port 53 on the firewall's IP addresses.  Either the Forwarder or the Resolver.  Choose which one you want to use and stick with it.

And I still think you ware WAY over-complicating things.  Just use one interface for your servers and put them all on different IP addresses.  One NAT config, one set of rules, etc.  If, in the future, you want to move them, it ought to be easy.

[dcol]

I do not want to run 4 servers off one NIC. The reason I am switching from my old firewall appliance is because of bandwidth issues and that device had 8 assignable ports which I only used 4.

I simply want to assign my 4 WAN IP's to the 4 NICs. I have a block of 8 IPs from the ISP on one WAN line.(Only use 4)
I am using Manual Outbound NAT with 4 Virtual IP's. All 4 servers can browse the internet but if you try to view a webpage on any of the servers, you get the pfsense webconfigurator. This is using the IP and not the domain name from a remote browser. If I use DNS, I get that DNS Rebind error.

So I am almost there, just need to tweak something and I tried every setting I can think of.

By the way, using NAT 1:1 does not work at all incoming or outgoing. Can't use port forwarding because I will be using the same port on multiple servers.

Temporarily I set the firewall on each interfaces to pass all traffic to eliminate any firewall blockage.

Really appreciate the help

[tim.mcmanus]

I've posted screen shots of my Port Forward, NAT, 1:1, Outbound NAT, and Rules.  As well as my LAN2 Interface.

Each of your Interfaces needs firewall rules created to communicate to other interfaces.  See my LAN2 Rules.  Port Forwarding, NAT, 1:1, and Outbound NAT are all more for external communications.  You should try first to get your server onto one of your SERVERx interfaces and then get that to go to the Internet.  So reset your router and get one server on one interface set up to at least communicate with the Internet.  That is low-hanging fruit.  If you can't do that, the rest doesn't matter.

The only interface that by default can communicate to the Internet is the first LAN interface.  All of the other interfaces lack rules to communicate anywhere else.  I suggest taking the LAN rules and copying them to SERVER1 and see if that gets your server to communicate to the Internet.

You can use the same port with multiple servers using virtual IPs.  See my screen shots to see how it's done.


[dcol]

Thanks for you response I believe it will be a great help when I try it tonight.

It looks like your configuration has 2 ISP WAN connections. One for the local network and one for your servers. WAN, WAN2
If you only have one ISP, then tell me what is plugged into the outer WAN interface
I assume you setup 3 Virtual IP's 96.57.99.139,140, and 141

using your IPs, here is my setup
10.0.1.1 - LAN - 96.57.99.138 - my 4th IP
10.0.2.1 - SERVER1 - 96.57.99.139 - Your LAN2
10.0.3.1 - SERVER2 - 96.57.99.140
10.0.4.1 - SERVER3 - 96.57.99.141
WAN - ISP connection with multi IP's

My NAT: Outbound would have only WAN to each subnet source and NAT address as WAN address
My SERVERx Firewall: Rules would be the same for each interface

[jahonix]

@dcol:

I do not want to run 4 servers off one NIC … because of bandwidth issues...

But they are all connected through one bottleneck WAN interface, right?
Do you use lots of local traffic to your servers? (Exchange Server with some Outlook clients does count.  ;-)

[tim.mcmanus]

96.57.99.138-141 are all on the same WAN2 interface.  They route to machines on my LAN and LAN2 (mostly LAN2).

WAN is a completely different WAN interface.  Yes, WAN and WAN2 are two independent and different WAN connections.

So if you follow the screen shots, a connection coming into WAN2 for address 96.57.99.140 would route (in your case) to SERVER2 (10.0.3.1).  You'll see this in the NAT screen where it comes into a public IP and then routes to a private IP.

You'll see a subsequent firewall rule to the NAT (they can be created at the same time, and I recommend this) for the WAN2 interface (which is the 96.57.99.138-141 interface) and routes that traffic to the destination server in LAN2.

LAN2 has firewall rules allowing any-to-any, so traffic can go in and out of that interface.  All of your SERVERx interfaces should be any-any to allow traffic into and out of the devices on that subnet.

However, as I stated in my initial post, start with the SERVERx rules and create the initial any-any rule FIRST.  Then see if the server can reach the Internet.  It should.  Once you've solved that issue, create the virtual IPs and then NATs, and it should just start working.

[dcol]

Thanks it is now working as expected, except the download speeds are 50% lower than when I was on the firewall appliance.
My system is a Dell with i5-4690 8GB memory, Intel i340-T4 quad NIC, and SSD drive. Should be faster internet speeds.

Is there any way to bring up the performance? Maybe there is some limiting setting somewhere.

[tim.mcmanus]

If you're not running any additional packages (and even if you were), you shouldn't see any impact to performance. Your specs seems. Rey good for the task at hand and then some.

How are you measuring download speeds and from where to where?

[dcol]

My normal bandwidth is 20Mbps Upload and 150Mbps Download. Since using pfsense I still get 20Mbps Up but only 40Mbps down. I figured it is some throttling on the downloads.

UPDATE
The servers bandwidth, clocking at 100Mbps down, is much higher than the LAN. I'd be one happy camper if the LAN did that well

I was hopping after I add a bunch of packages, which I have not done so far, I can maintain decent speeds.
I am going to post my settings shortly and lets see if anyone can spot any mistakes I may have made.

Thanks to you all, you have been a great help and frankly makes pfsense a better product.

[dcol]

Here are screenshots of my setting. Public IP's are partially masked for security.
Every setting not shown would be the default setting.

[tim.mcmanus]

Your MBUF usage is very high for a computer with your specs. Not sure why and not entirely sure it's at all related to your issue of speed. Everything else looks okay.

[dcol]

MBUF was high because of the Intel Quad NIC. I added kern.ipc.nmbclusters="1000000" to the loader.conf.local file and now the MBUF is down to 2%

Thanks for that catch.