Traffic blocked even with any/any rules on both interfaces

[rjensen]

I have 2 interfaces where 2 VMs are connected with each other. VMware View connection and security servers.
There are any/any rules created to allow everything on both these interfaces but even with that TCP port 4001 gets blocked. If i disable the firewall under settings (turning pfSense into a router) traffic flows OK.

See attached pictures of the rules and a full packet capture.
I have also tried to create "easy rules" based on the blocked traffic but its still getting blocked.

Is there some kind of intelligent filtering going on even with a full allow rule? TCP port 4001 in this case is JMS traffic.

Capture from both interfaces below:

17:15:21.155100 00:50:56:a7:64:1a > 00:50:56:a7:60:81, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 127, id 9400, offset 0, flags [none], proto TCP (6), length 52)
    192.168.190.31.54528 > 192.168.130.31.4001: Flags ~~, cksum 0x2730 (correct), seq 2642674736, win 65535, options [mss 1460,nop,wscale 2,nop,nop,sackOK], length 0
17:15:21.155528 00:50:56:a7:64:1a > 00:50:56:a7:60:81, ethertype IPv4 (0x0800), length 54: (tos 0x0, ttl 127, id 9401, offset 0, flags [none], proto TCP (6), length 40)
    192.168.190.31.54528 > 192.168.130.31.4001: Flags [.], cksum 0x81d3 (correct), seq 2642674737, ack 352866577, win 32768, length 0
17:15:21.156121 00:50:56:a7:64:1a > 00:50:56:a7:60:81, ethertype IPv4 (0x0800), length 87: (tos 0x0, ttl 127, id 9402, offset 0, flags [none], proto TCP (6), length 73)
    192.168.190.31.54528 > 192.168.130.31.4001: Flags [P.], cksum 0x928a (correct), seq 0:33, ack 1, win 32768, length 33
17:15:21.257534 00:50:56:a7:64:1a > 00:50:56:a7:60:81, ethertype IPv4 (0x0800), length 94: (tos 0x0, ttl 127, id 9403, offset 0, flags [none], proto TCP (6), length 80)
    192.168.190.31.54528 > 192.168.130.31.4001: Flags [P.], cksum 0xb694 (correct), seq 33:73, ack 16, win 32764, length 40
17:15:21.258222 00:50:56:a7:64:1a > 00:50:56:a7:60:81, ethertype IPv4 (0x0800), length 103: (tos 0x0, ttl 127, id 9404, offset 0, flags [none], proto TCP (6), length 89)
    192.168.190.31.54528 > 192.168.130.31.4001: Flags [P.], cksum 0xdad0 (correct), seq 73:122, ack 92, win 32745, length 49
17:15:21.259936 00:50:56:a7:64:1a > 00:50:56:a7:60:81, ethertype IPv4 (0x0800), length 83: (tos 0x0, ttl 127, id 9405, offset 0, flags [none], proto TCP (6), length 69)
    192.168.190.31.54528 > 192.168.130.31.4001: Flags [P.], cksum 0x7f36 (correct), seq 122:151, ack 107, win 32741, length 29
17:15:21.261107 00:50:56:a7:64:1a > 00:50:56:a7:60:81, ethertype IPv4 (0x0800), length 83: (tos 0x0, ttl 127, id 9406, offset 0, flags [none], proto TCP (6), length 69)
    192.168.190.31.54528 > 192.168.130.31.4001: Flags [P.], cksum 0x9aa3 (correct), seq 151:180, ack 198, win 32718, length 29
17:15:21.261658 00:50:56:a7:64:1a > 00:50:56:a7:60:81, ethertype IPv4 (0x0800), length 96: (tos 0x0, ttl 127, id 9407, offset 0, flags [none], proto TCP (6), length 82)
    192.168.190.31.54528 > 192.168.130.31.4001: Flags [P.], cksum 0x9a65 (correct), seq 180:222, ack 230, win 32710, length 42

int 190

17:17:43.888218 00:50:56:a7:60:82 > 00:50:56:a7:4a:52, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 128, id 9644, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.190.31.54536 > 192.168.130.31.4001: Flags [.], cksum 0x4141 (correct), seq 3925529778, ack 391294809, win 32686, options [nop,nop,sack 1 {4294967268:1}], length 0
17:17:43.939968 00:50:56:a7:60:82 > 00:50:56:a7:4a:52, ethertype IPv4 (0x0800), length 60: (tos 0x0, ttl 128, id 9645, offset 0, flags [DF], proto TCP (6), length 40)
    192.168.190.31.54537 > 192.168.130.31.4001: Flags [.], cksum 0x52a6 (correct), seq 4283120382, ack 791727554, win 32694, length 0
17:17:44.970371 00:50:56:a7:0b:24 > 00:50:56:a7:4a:52, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 128, id 12812, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.190.132.51438 > 192.168.130.122.4001: Flags [.], cksum 0x20a8 (correct), seq 2796161125, ack 3540584317, win 32686, options [nop,nop,sack 1 {4294967268:1}], length 0
17:17:46.931406 00:50:56:a7:60:82 > 00:50:56:a7:4a:52, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 128, id 9646, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.190.31.54536 > 192.168.130.31.4001: Flags [.], cksum 0x4141 (correct), seq 0, ack 1, win 32686, options [nop,nop,sack 1 {4294967268:1}], length 0
17:17:49.739942 00:50:56:a7:0b:24 > 00:50:56:a7:4a:52, ethertype IPv4 (0x0800), length 83: (tos 0x0, ttl 128, id 12813, offset 0, flags [DF], proto TCP (6), length 69)
    192.168.190.132.51436 > 192.168.130.122.4001: Flags [P.], cksum 0x378e (correct), seq 431059994:431060023, ack 1619160734, win 32678, length 29
17:17:50.051842 00:50:56:a7:0b:24 > 00:50:56:a7:4a:52, ethertype IPv4 (0x0800), length 83: (tos 0x0, ttl 128, id 12814, offset 0, flags [DF], proto TCP (6), length 69)
    192.168.190.132.51436 > 192.168.130.122.4001: Flags [P.], cksum 0x378e (correct), seq 0:29, ack 1, win 32678, length 29
17:17:50.180117 00:50:56:a7:60:82 > 00:50:56:a7:4a:52, ethertype IPv4 (0x0800), length 60: (tos 0x0, ttl 128, id 9647, offset 0, flags [DF], proto TCP (6), length 40)
    192.168.190.31.54538 > 192.168.130.31.4001: Flags [.], cksum 0x39b4 (correct), seq 3649408172, ack 4289260114, win 32694, length 0
17:17:50.660289 00:50:56:a7:0b:24 > 00:50:56:a7:4a:52, ethertype IPv4 (0x0800), length 83: (tos 0x0, ttl 128, id 12815, offset 0, flags [DF], proto TCP (6), length 69)
    192.168.190.132.51436 > 192.168.130.122.4001: Flags [P.], cksum 0x378e (correct), seq 0:29, ack 1, win 32678, length 29
17:17:50.707027 00:50:56:a7:0b:24 > 00:50:56:a7:4a:52, ethertype IPv4 (0x0800), length 60: (tos 0x0, ttl 128, id 12816, offset 0, flags [DF], proto TCP (6), length 40)
    192.168.190.132.51440 > 192.168.130.122.4001: Flags [.], cksum 0x672b (correct), seq 760600741, ack 1663419776, win 32550, length 0
17:17:51.525904 00:50:56:a7:0b:24 > 00:50:56:a7:4a:52, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 128, id 12817, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.190.132.51438 > 192.168.130.122.4001: Flags [.], cksum 0x20a8 (correct), seq 0, ack 1, win 32686, options [nop,nop,sack 1 {4294967268:1}], length 0
17:17:51.861567 00:50:56:a7:0b:24 > 00:50:56:a7:4a:52, ethertype IPv4 (0x0800), length 83: (tos 0x0, ttl 128, id 12818, offset 0, flags [DF], proto TCP (6), length 69)
    192.168.190.132.51436 > 192.168.130.122.4001: Flags [P.], cksum 0x378e (correct), seq 0:29, ack 1, win 32678, length 29

~~

[podilarius]

What are the interface types? a WAN and a LAN or a LAN and an OPT? It is possible that you might have WAN option to block private network on. There could be unwanted NAT on return trip. The wide open rules should allow the traffic to pass.

[rjensen]

Its 2 OPT interfaces but being used for LAN routing if that makes sense. I just added 2 additional vmnics (im running this on ESXi) and they came up as OPT# but i renamed to fit my network labels.
Bogus and private networks are not blocked as per the screenshots.

[podilarius]

You could have a routing problem. Are both subnets behind each of the OPT interfaces using pfsense as its gateway? If not, is there a router that is doing that?

[rjensen]

podilarius,
Thanks for helping with this!

Only one of the subnets are using pfsense as a gateway - all other machines are connected to a cisco switch that does the routing.

If i change both servers to use the pfsense as the gateway for the 2 subnets that seems to work - but then it seems like my traffic is blocked from other VLANs. eg. i try to access something on a subnet thats behind the any/any Pass rule and it gets blocked (tested with accessing the webinterface on the application server on 443 and comes up as blocked)

Basically im trying to come up with a setup where pfsense only routes 2 of my subnets and the rest is done my my cisco switch which has inter vlan routing on.

[podilarius]

That would make sense. Basically you are going to have to tell the Cisco to route traffic for the other side of your pfsense box to the IP address on its side of the pfsense box.
Hope that makes sense, I don't have examples from your setup to personalize.

basically it would go:

computer1 -> cisco -> pfsense opt1 -> pfsense opt2 -> computer 2

computer 1 has the cisco as its default gateway. Cisco says that anything going to computer 2's subnet goes to pfsense opt1 ipaddress. computer 2 uses pfsense as its gateway.
The cisco must have the route to make it all work.

[rjensen]

Attached an example of how it its setup.
Basically my requirement would be for the servers in DMZ to be able to communicate with the LAN via firewall rules in the pfSense.
The Cisco ASA might confuse the picture - its there for other reasons :)

Even with that routing setup you proposed i still get errors - i might be configuring it wrong?

Thank you!

[rjensen]

Also its not really clear to me if i need an OPT interface for VLAN 130 (LAN) on the pfSense for it to communicate with those VMs or if i can achieve that via routing (the cisco switch)?

[Klaws]

The point of using VLANs is to share backbones (and NICs). So you don't really need a physical OPT1 interface.

The picture of your current setup looks like your Cisco switches connect to pfSense via trunk ports already. If you connect to pfSense voa access ports, you'd need to setup VLAN interfaces in pfSense.

Whatever: from the configuration point of view, removing the physical OPT1 NIC doesn't significanly reduce complexity.

Of course, if your OPT1 NIC is an old Realtek device which you pulled from the junk bin…than you might really want to remove it ;)

[rjensen]

im running pfSense virtually but my ESXi hosts have  multiple VLANs configured as trunks

my pfSense just has a virtual NIC in one or more of those VLANs (eg. 190, 130)

basically what i dont get is how i get traffic from DMZ > LAN.
Do i configure my server in DMZ with the pfsense as default gateway on VLAN 190 and my LAN hosts with the cisco as the default gateway on VLAN 130?
how does the pfSense know about VLAN 130 - do i need to add a static route on the pfSense or have an OPT interface in VLAN 130 for it to access that VLAN?

[Klaws]

Sorry, I know nothing about ESXi VLAN support.

In pfSense, VLANs are treated just like physical NICs. Click on Interfaces > (assign) > VLAN and start adding VLANs. Then, on the tab "Interface Assignments", add interfaces and assign the VLANs to them. You did that probably already.

In every LAN (this includes the DMZ), pfSense should be the default gateway. It appears that the DB servers sit in the 192.168.130.* subnet, right? So they won't work with the Cisco ASA as default gateway, as it already sits outside this range.

Of course, this has abot nothing to do with your traffic getting blocked. I once observed a stange issue with an OPT1 interface (which was the "physical" interface underneath some VLANs, aka the default VLAN, aka the untagged LAN) where traffic wouldn't get through. I had to define and assign a gateway to that interface, which was simply pfSense itself (with the IP address in that interfaces range).

[rjensen]

I have no idea on how to progress with this…

i have attached a drawing of how its setup now but still traffic to/from those interfaces are blocked even though there is any/any rules.

i can traceroute and ping between the 2 hosts no problem - but other forms of traffic gets blocked.

if anyone wants to spend time with me going through this on a webex/teamviewer - much appreciated! :)

[Klaws]

Why do you use the Cisco switch's IP address as the default gateway? As far as I can tell, it doesn't route but switch. The default gateway should be pfSense…right?

[rjensen]

i have inter vlan routing enabled on the cisco switch as it routes all my other 14 VLANs configured on that switch.

I only want the pfSense to route the DMZ VLAN if possible but it sounds like that is my issue?

[Klaws]

I'd try to assign pfSense (192.168.130.2) as the gateway in the 192.168.130.* subnet (on the pfSense side).

Well - I wrote "default gateway" earlier. That was wrong for the clients in the LAN. It would probably break inter-VLAN routing.

On the clients, use the "route" command to add a route to the 192.168.190.* subnet, with pfSense (192.168.130.2) as the gateway for this route. On Windows boyes, this would be something like

route ADD 192.168.190.1 MASK 255.255.255.0 192.168.130.2

• I think.

Edit: sorry if I happen to explain things you may have already done. With the vast amount of possible network configuration items available, it's hard to tell what might have been implemented and what not.

[podilarius]

It looks like from the picture that the 130 VLAN is for the web servers to access the databases for dynamic content or whatever. It does need the opt interface if you don't want the traffic going all the way outside of pfsense and coming in via the CiscoASA. In does not matter who handles the VLAN (ESX or physical), the VLAN is assigned to the OPT interface anyway. You can open all ports if you like and it would act like a routed solution, but the interface would still be required. the l3 cisco switch could route dmz vlan 190 subnet to 130.2 though. all other traffic out the ASA if that is what you prefer.

[johnpoz]

So you only have 1 cisco switch then.  And I assume since you mention you have 14 other vlans your trunking the connection to the cisco that connected to ??  But seems you show the pfsense having interface in the vlan your lan is in 130?  And you point boxes in this vlan 130 to cisco as their gateway.

What is the default gateway on the cisco when you want to access some network that is not a part of your vlans?  In your previous drawing you show this ASA that is in a 192.168.234 network?

Wouldn't it be cleaner if your connection to your cisco from pfsense was its own interconnect vlan.  And do the same for your ASA connection to the cisco to just keep it cleaner.  Then create a dmz vlan for pfsense for that segment?

Now the dmz vlan on pfsense would be connected to your isolated dmz you would just need to create the appropriate routing and firewall rules to get to your other vlans via the interconnect vlan

Something like the attached.

you would use a interconnect vlan that ties your pfsense to all your other vlans on the cisco switch.  You would create a dmz vlan to distinguish that as isolated, etc.  This could share address space and be the same vlan as your servers you want to put in this vlan, or could be different if you so desired.

But I would think this would be cleaner

So in for example dmz interface pfsense dmz.1 servers on this vlan would be dmz.2, .14, .??  And use the pfsense dmz.1 IP as their gateway.

Again I would create a NEW vlan to use as interconnect to your other vlans.   Lets call - so pfsense would have IC.1 as IP and cisco would have interface in this interconnect vlan IC.2

And the cisco would also have asa vlan so asa would have asa.1 IP and cisco would have asa.2 in this vlan.  Now your cisco would use default gateway of asa.1 for all traffic it needs to route for your other vlans to get to the internet.

For routing between your dmz and other vlans you would route through the interconnect vlan.  So if box say in vlan A needed to get to dmz.14 address it would use the cisco ip in vlan A, say vlanA.1 cisco would say oh you want to go to dmz.14 send the traffic to pfsense at IC.1

Now depending on what routes and rules you put in place on pfsense would determine what kind of traffic you would allow between your dmz vlan off of pfsense and your other vlans off of your cisco.  On the cisco would not not allow routing between vlans A,B,C etc and the DMZ vlan - all this traffic would have to go through pfsense.  your just going to be using some ports on the cisco as connections in an isolated vlan - just like breaking that vlan out on a different switch.

Now i just did a quick read over of the thread and that is how I would configure what I understand your trying to do.

[rjensen]

Thank you very much to everyone who helped solve this for me! It was clearly the routing and now i have build it like the drawing johnpoz posted (created a dedicated VLAN for the "interconnect VLAN") and firewall rules etc. not works.
WHich means im on to the next issue :)
Ill post this in another thread but wanted to try it out real quick:

I have 5 external IPs (each via a dedicated OPT/WAN interface via DHCP) pointing to the same ISP gateway. My NAT rules works on/off depending on which interface i define as having the default gateway and i see this in the logs:

Mar 18 17:45:49 routed[2891]: em3 (90.184.xxx.xx1 (mask 0xfffffe00)) is duplicated by em1 (90.184.xxx.xxx (mask 0xfffffe00))
Mar 18 17:45:49 routed[2891]: em4 (90.184.xxx.xx2 (mask 0xfffffe00)) is duplicated by em1 (90.184.xxx.xxx (mask 0xfffffe00))
Mar 18 17:45:49 routed[2891]: em4 (90.184.xxx.xx3 (mask 0xfffffe00)) is duplicated by em1 (90.184.xxx.xxx (mask 0xfffffe00))
Mar 18 17:45:49 routed[2891]: em5 (90.184.xxx.xx4 (mask 0xfffffe00)) is duplicated by em1 (90.184.xxx.xxx (mask 0xfffffe00))
Mar 18 17:45:49 routed[2891]: em5 (90.184.xxx.xx5 (mask 0xfffffe00)) is duplicated by em1 (90.184.xxx.xxx (mask 0xfffffe00))

if i turn the pfSense off, release all DHCP requests against my ISP and power it back on NAT rules only works against the interface that holds the default gateway role.
any advice - ill post in another thread if this deserves its own subject.

Thank you all for making me understand this and get it up and running!!

[johnpoz]

Why are you using so many interfaces for the same connection?  Why don't you just create virtual IPs for your other IPs on the same interface?

Yeah if you want to nat off those other Ips you would have to create specific rules for that.

[rjensen]

afaik virtual IPs can only be fixed - i can only get to those external WAN IPs using DHCP unfortunately.
ive read that it requires that i "load balance" all the WAN interfaces against the same GW IP but im pretty sure i had this working some time ago without doing that?
What do you mean by "specific rules for that" when doing NAT?

Thank you!