Traffic blocked even with any/any rules on both interfaces

[podilarius]

It looks like from the picture that the 130 VLAN is for the web servers to access the databases for dynamic content or whatever. It does need the opt interface if you don't want the traffic going all the way outside of pfsense and coming in via the CiscoASA. In does not matter who handles the VLAN (ESX or physical), the VLAN is assigned to the OPT interface anyway. You can open all ports if you like and it would act like a routed solution, but the interface would still be required. the l3 cisco switch could route dmz vlan 190 subnet to 130.2 though. all other traffic out the ASA if that is what you prefer.

[johnpoz]

So you only have 1 cisco switch then.  And I assume since you mention you have 14 other vlans your trunking the connection to the cisco that connected to ??  But seems you show the pfsense having interface in the vlan your lan is in 130?  And you point boxes in this vlan 130 to cisco as their gateway.

What is the default gateway on the cisco when you want to access some network that is not a part of your vlans?  In your previous drawing you show this ASA that is in a 192.168.234 network?

Wouldn't it be cleaner if your connection to your cisco from pfsense was its own interconnect vlan.  And do the same for your ASA connection to the cisco to just keep it cleaner.  Then create a dmz vlan for pfsense for that segment?

Now the dmz vlan on pfsense would be connected to your isolated dmz you would just need to create the appropriate routing and firewall rules to get to your other vlans via the interconnect vlan

Something like the attached.

you would use a interconnect vlan that ties your pfsense to all your other vlans on the cisco switch.  You would create a dmz vlan to distinguish that as isolated, etc.  This could share address space and be the same vlan as your servers you want to put in this vlan, or could be different if you so desired.

But I would think this would be cleaner

So in for example dmz interface pfsense dmz.1 servers on this vlan would be dmz.2, .14, .??  And use the pfsense dmz.1 IP as their gateway.

Again I would create a NEW vlan to use as interconnect to your other vlans.   Lets call - so pfsense would have IC.1 as IP and cisco would have interface in this interconnect vlan IC.2

And the cisco would also have asa vlan so asa would have asa.1 IP and cisco would have asa.2 in this vlan.  Now your cisco would use default gateway of asa.1 for all traffic it needs to route for your other vlans to get to the internet.

For routing between your dmz and other vlans you would route through the interconnect vlan.  So if box say in vlan A needed to get to dmz.14 address it would use the cisco ip in vlan A, say vlanA.1 cisco would say oh you want to go to dmz.14 send the traffic to pfsense at IC.1

Now depending on what routes and rules you put in place on pfsense would determine what kind of traffic you would allow between your dmz vlan off of pfsense and your other vlans off of your cisco.  On the cisco would not not allow routing between vlans A,B,C etc and the DMZ vlan - all this traffic would have to go through pfsense.  your just going to be using some ports on the cisco as connections in an isolated vlan - just like breaking that vlan out on a different switch.

Now i just did a quick read over of the thread and that is how I would configure what I understand your trying to do.

[rjensen]

Thank you very much to everyone who helped solve this for me! It was clearly the routing and now i have build it like the drawing johnpoz posted (created a dedicated VLAN for the "interconnect VLAN") and firewall rules etc. not works.
WHich means im on to the next issue :)
Ill post this in another thread but wanted to try it out real quick:

I have 5 external IPs (each via a dedicated OPT/WAN interface via DHCP) pointing to the same ISP gateway. My NAT rules works on/off depending on which interface i define as having the default gateway and i see this in the logs:

Mar 18 17:45:49 routed[2891]: em3 (90.184.xxx.xx1 (mask 0xfffffe00)) is duplicated by em1 (90.184.xxx.xxx (mask 0xfffffe00))
Mar 18 17:45:49 routed[2891]: em4 (90.184.xxx.xx2 (mask 0xfffffe00)) is duplicated by em1 (90.184.xxx.xxx (mask 0xfffffe00))
Mar 18 17:45:49 routed[2891]: em4 (90.184.xxx.xx3 (mask 0xfffffe00)) is duplicated by em1 (90.184.xxx.xxx (mask 0xfffffe00))
Mar 18 17:45:49 routed[2891]: em5 (90.184.xxx.xx4 (mask 0xfffffe00)) is duplicated by em1 (90.184.xxx.xxx (mask 0xfffffe00))
Mar 18 17:45:49 routed[2891]: em5 (90.184.xxx.xx5 (mask 0xfffffe00)) is duplicated by em1 (90.184.xxx.xxx (mask 0xfffffe00))

if i turn the pfSense off, release all DHCP requests against my ISP and power it back on NAT rules only works against the interface that holds the default gateway role.
any advice - ill post in another thread if this deserves its own subject.

Thank you all for making me understand this and get it up and running!!

[johnpoz]

Why are you using so many interfaces for the same connection?  Why don't you just create virtual IPs for your other IPs on the same interface?

Yeah if you want to nat off those other Ips you would have to create specific rules for that.

[rjensen]

afaik virtual IPs can only be fixed - i can only get to those external WAN IPs using DHCP unfortunately.
ive read that it requires that i "load balance" all the WAN interfaces against the same GW IP but im pretty sure i had this working some time ago without doing that?
What do you mean by "specific rules for that" when doing NAT?

Thank you!

[johnpoz]

I would assume you would have to switch over to manual nat rules for your outbound if you want say specific IPs to use specific interface when it talks back to internet..

So for example traffic comes in interface 3 publicIP, gets natted to box 192.168.1.3, he answers back to pfsense lan IP, pfsense needs to know to answer that back using public IP3 vs the default gateway.

if you change your outbound nats to manual you will see, not sure what happens with auto mappings when you have multiple wan interfaces?

[rjensen]

i dont care which IPs by DMZ based servers use for internet access (its blocked anyway…) unless it has an impact on my incoming NAT rules which it seems to have?

If i do a regular HTTPS NAT rule that only works against the external WAN interface that holds the "default gateway"

[johnpoz]

huh?  So you have no inbound traffic on these other IPs?  And you don't use them for internet access?  Then why do you have them setup?  Im confused with your last post.

You might not care, but the server talking to publicIP 3, is going to care if traffic comes back from publicIP 1 – normally not going to like that ;)

[rjensen]

Sorry, i have incoming traffic (public IP1 > NAT (HTTPS) > server in DMZ) but i dont care if that server in DMZ goes to the internet using public IP1, 2 3 etc. (its blocked to get to the internet)
however if i build these 2 NAT rules only the one public IP (WAN interface) holding "default gateway" works:

public IP1 > NAT (HTTPS) > server01 in DMZ
public IP2 > NAT (HTTPS) > server02 in DMZ

hopes that make sense - it should be no surprise to anyone im new at this :)

[johnpoz]

"public IP2 > NAT (HTTPS) > server02 in DMZ"

This is my point, so server02 is going to answer that client talking to public IP2 from public IP1 – that client wanting to talk to public IP2 is not going to accept traffic from public IP1 as an answer..

You need to look at your outbound nat rules to make sure that traffic coming in on public IP2 goes back out public IP2..

And not understanding this statement
"but i dont care if that server in DMZ goes to the internet using public IP1, 2 3 etc. (its blocked to get to the internet)"

If you going to want server in the DMZ to answer something from the internet, how could its internet traffic be blocked?

[rjensen]

OK, i see the Firewall > NAT > Outbound but thats beyond me…

Not sure what source and destination would be? Im looking to achieve being able to nat say port 443 against 1 server and 80 against another server on the same public IP if that makes sense.

I might be wrong but i think i had these multiple NAT rules working before without making any changes.
Under System > Advanced > firewall/NAT i have the options disabled as shown on the attached screenshot: is that part of my issue?

sorry - the servers in DMZ do have internet access, i am just planning on disallowing the servers to browse the internet by blocking port 80/443 outbound to secure the environment. but no such rules at the moment.


[johnpoz]

You need to look at your outbound nats, change them from automatic to manual so you can see what rules are in place currently.

[rjensen]

Ive been trying to figure out how to make this work and im not sure if this is a "normal network issues" or related to pfSense? Im at a place now where im thinking going with 5 x pfSense servers - 1 for each external IP as i cant seem to figure out how this is configured in general…

I have switched to manual outbound NAT rules which creates a pre-defined set of rules it seems.
Im also seeing issues when i have multiple WAN interfaces enabled: none of my servers in the DMZ can get to the internet. Before if i only have 1 WAN address set with a default GW that works fine as they can hit the internet and my NAT rules works.

With these 5 WAN IPs (DHCP - Same gateway) how would i configure the outbound NAT rules?
I dont understand if this is set for each host, subnet, interface etc. I understand inbound NAT but as i cant grasp the concepts of outbound NAT i have no idea what to configure.

Is it actually achievable having 1 pfSense with 5 WAN IPs (same gateway) and have regular NAT rules on say HTTPS for each of those 5 WAN IPs to different DMZ targets?

[johnpoz]

Switch to manual outbound and we can walk through why your having an issue.

So example - now I only have 1 public IP, and here are my outbound nats after switching to manual so you can see them.

So you see the rule I highlighted - this says hey your coming from network 192.168.1.0/24 use the WAN address as public and dynamically do the ports.  This is the auto created nat rule..

Lets see yours!

So I created a virtual IP, just example 1.2.3.4 - then I created an outbound nat that says hey if your coming from 192.168.1.42 you use the IP 1.2.3.4 when you go outbound.

[rjensen]

Attached. Only have 2 of the WAN interfaces active now - thats enough to confuse me so ill add the other 3 in later :)

Right now none of my servers in 192.168.190.0/24 can get to the internet. traceroute etc. does not work.
even the pfSense itself cant do traceroute.
i have WAN1 set with default gateway.


[rjensen]

I tried the same as you did (i think…) defining that 192.168.190.0/24 has to go through WAN01 but still - even the pfSense cant get to the internet. Im just trying a basic traceroute.
With this approach would i need 5 different subnets/VLANs to make this work? As i have 4 webservers each requiring HTTPS 443 to be available would i need an outbound NAT rule that is mirroring my inbound NAT rule?

[johnpoz]

Well if you can not even get to the internet your rules are not going to work…  If you say you can not get to the internet from pfsense - how is an outbound route going to come into play?

[rjensen]

agreed - but how do i figure out what the issue is? With a single WAN interface it works - as soon as i add the 2nd all outgoing traffic fails.

[podilarius]

The rules you have posted for manual outbound NAT looks wrong.  You have multiple subnets using multiple interfaces. Simplify the rules a bit by putting different subnets on different external ip addresses. That is unless you are more worried about inbound. Then 1:1 nat would be advisable with the default outbound for all sinners on only one address.

[rjensen]

i did just try that and the result is that i still cant get to the internet now from any machines in the subnets i have changed outbound NAT for.
pfSense i also tried enabling a second WAN interface and changed all the outbound NAT rules to use that interface, but still no luck.

one thing i keep seeing is that when i add a second interface it monitors the gateway and it gets no status:

Name Gateway Monitor Status         Description
GW_WAN xx.xx.xx.xx Gathering data Gathering data

the first interface i have added (default gateway is set on this) works fine and it reports RTT and online status.
can this have an impact?