Can't block vlan to vlan traffic even with explicit IP blocks

[terahz]

Hello smart people!

My setup is the following:

2.3.1-RELEASE-p5 (amd64) on a small HTPC style box.

WAN - em0
LAN - VLAN10 on em1 - 192.168.1.0/24
GUEST - VLAN99 on em1 - 192.168.0.0/24

IPv6 is disabled

DNS Forwarder is configured and enabled for both internal interfaces with strict binding
Snort  with Barnyard2 is enabled on WAN

Switch port connected to em1 is a trunk with 10,99 tagged vlans only.

My problem is that I can't seem to prevent pfsense from routing traffic between the vlans.

LAN Rules:
allow from all to LAN address ports 443,80 - Anti-Lockout
block from all IPv4 to GUEST net all
block from IPv4 GUEST net to all
allow from all IPv4 to all

GUEST Rules:
block from all IPv4 to LAN net all
block from IPv4 LAN net to all
allow all IPv4 to all

Yet I'm able to ping any 192.168.0.0 client from any 192.168.1.0  client and vice-versa.

I tried adding explicit IP blocks, both manually and from the quick rules in the logs - didn't work
I tried doing the rfc1918 alias setup recommended in posts here - didn't work

Packet capture on GUEST shows the packets flowing right trough.

I realize it's 3AM here and I'm probably missing something obvious, so please help.

[Derelict]

Yeah. Post your rule screen shots instead of a summary of what you think you've done.

You haven't done something silly like System > Advanced, Firewall & NAT Tab, Disable Firewall right?

Any floating rules?

LAN Rules:
allow from all to LAN address ports 443,80 - Anti-Lockout
block from all IPv4 to GUEST net all
block from IPv4 GUEST net to all
allow from all IPv4 to all

GUEST Rules:
block from all IPv4 to LAN net all
block from IPv4 LAN net to all
allow all IPv4 to all

The italicized rules are unnecessary and will do nothing.

https://doc.pfsense.org/index.php/Firewall_Rule_Troubleshooting

https://forum.pfsense.org/index.php?topic=113842.msg632969#msg632969

[terahz]

@Derelict:

Yeah. Post your rule screen shots instead of a summary of what you think you've done.

LAN

GUEST

@Derelict:

You haven't done something silly like System > Advanced, Firewall & NAT Tab, Disable Firewall right?

Nope. Firewall is active.

@Derelict:

Any floating rules?

None.

@Derelict:

The italicized rules are unnecessary and will do nothing.

That's what I figured too, but because it wasn't working I kept adding more rules. Will remove these.


[johnpoz]

so your sure the traffic is flowing through pfsense, and not just a problem with vlan settings on your switch where you have these networks in the same layer 2?

" trunk with 10,99 tagged vlans only."

So your lan is a vlan as well?  And tagged.. I normally don't do that and run untagged in a native vlan on the actual interface.  So what is your port setup on the devices in lan and the other in guest?  The ports are in pvid 10 and other is pvid 99?

Also there is checkbox on firewall advanced where you can tell pfsense not to check firewall rules for networks on the same interface.  You don't have that checked do you?

[Harvy66]

Layer 3 switch?

[terahz]

@johnpoz:

so your sure the traffic is flowing through pfsense, and not just a problem with vlan settings on your switch where you have these networks in the same layer 2?

Yes, if I disable the GUEST interface, the bridge is broken. I also see the traffic in the firewall logs and in packet capture on the GUEST interface.

@johnpoz:

" trunk with 10,99 tagged vlans only."
So your lan is a vlan as well?  And tagged.. I normally don't do that and run untagged in a native vlan on the actual interface.  So what is your port setup on the devices in lan and the other in guest?  The ports are in pvid 10 and other is pvid 99?

Yes, everything is tagged. It's a bad idea to have untagged 1 and tagged <x>since 1 is the default vlan everywhere. The switch port is configured as trunk, passing only tagged traffic, member of tagged vlan 10 and tagged vlan 99.

@johnpoz:

Also there is checkbox on firewall advanced where you can tell pfsense not to check firewall rules for networks on the same interface.  You don't have that checked do you?

I don't have that checked.

@Harvy66:

Layer 3 switch?

Yes, the switch has Layer 3 functionality.

One interesting data point might be that the pings between the two vlans are very very slow and irregular. They can sometime take up to a second to return and on average take few hundred ms.</x>

[Derelict]

Traceroute I guess. You have something else going on.

[johnpoz]

"untagged 1 and tagged <x>since 1 is the default vlan everywhere"

Where did I say it was untagged 1??  While I do have 1 of my networks as vlan 1..  My other network is not vlan 1.  While yes it common practice for an enterprise to not use vlan 1.  In home or small setup where you don't have to worry about someone putting in some switch and leaving ports in the default vlan its not an issue..  Does not matter if you use untagged 1 or 100 or whatever if your environment is under control.  I don't have any concerns of some switch getting plugged into my network where having ports on vlan 1 would be of any concern at all.

"The switch port is configured as trunk, passing only tagged traffic, "

Says who??  what is your port configuration??  What is the configuration of your other ports.  What is the make of your switch, in cisco you could tag the native vlan with
vlan dot1q tag native

But I don't believe its actually possible to remove the native vlan from the port.  Which is why the practice of setting ports to some unused vlan ID when not being used, etc.

So your switch is layer 3??  So who says the traffic is even routing over pfsense - why are you not routing that traffic at your layer 3 switch?

What I can tell you for sure is pfsense filters traffic just fine between vlans on the same physical interface..  I have multiple vlans on a physical interface and it blocks traffic just fine..</x>

[terahz]

@johnpoz:

Says who??  what is your port configuration??  What is the configuration of your other ports.

Says me, the person who configured the switch. For a third time, the port is configured as a trunk with only tagged traffic for vlan 10 and 99. Here is the output of show vlan on my switch. port 1/0/1 is where pfsense is connected.

#show vlan 

 VLAN 1 
   Name : default
   Tagged Member Ports   :                     
   Untagged Member Ports :                     

 VLAN 10 
   Name : Main
   Tagged Member Ports   : 1/0/1               
   Untagged Member Ports : 1/0/2-1/0/22,1/0/24-1/0/28

 VLAN 99 
   Name : Guest
   Tagged Member Ports   : 1/0/1               
   Untagged Member Ports : 1/0/23              

 Total Entries : 3

@johnpoz:

What is the make of your switch

D-Link DGS-1510-28X

@johnpoz:

So your switch is layer 3??  So who says the traffic is even routing over pfsense - why are you not routing that traffic at your layer 3 switch?

Yes, the switch has layer 3 functionality but there are no gateways or routes defined. And I am saying the traffic is routing over pfsense, because if I disable GUEST or LAN on the pfsense box, traffic stops routing. I can also capture the traffic from pfsense and I can see it in the pfsense firewall logs. I guess the only extra proof that I can generate is to show screenshots of the mac addresses of the interfaces and post a pcap captured from the pfsense box…

@Derelict:

Traceroute I guess. You have something else going on.

I'll post a screenshot of it later. From what I remember from last night(this morning?) it had one hop before reaching the destination - the corresponding pfsense IP of the source network.

[Derelict]

What is the output of show ip route on the switch?

[terahz]

Here is traceroute:

$ traceroute to 192.168.0.100 (192.168.0.100), 64 hops max, 52 byte packets
 1  gw (192.168.1.1)  1036.060 ms  0.161 ms  0.138 ms
 2  * 192.168.0.100 (192.168.0.100)  437.802 ms  2.809 ms

From this machine:

$ ifconfig en1
en1: flags=8863 <up,broadcast,smart,running,simplex,multicast>mtu 1500
	options=67 <rxcsum,txcsum,vlan_mtu,tso4,tso6>ether 00:0f:53:09:11:d4 
	inet6 fe80::20f:53ff:fe09:11d4%en1 prefixlen 64 scopeid 0x4 
	inet 192.168.1.99 netmask 0xffffff00 broadcast 192.168.1.255
	nd6 options=1 <performnud>media: autoselect (10GbaseSR <full-duplex,flow-control>)
	status: active</full-duplex,flow-control></performnud></rxcsum,txcsum,vlan_mtu,tso4,tso6></up,broadcast,smart,running,simplex,multicast> 

and show ip route:

#show ip route
Code: C - connected, S - static
      * - candidate default

Gateway of last resort is not set

C    192.168.0.0/24 is directly connected, vlan99
C    192.168.1.0/24 is directly connected, vlan10

Total Entries: 2 

[Derelict]

C    192.168.0.0/24 is directly connected, vlan99
C    192.168.1.0/24 is directly connected, vlan10

Your switch is routing between the VLANs, not the firewall.

You have to configure your hosts so the pfSense interface address is their default gateway.

Do something like this on the switch:

no interface vlan99
no interface vlan10

Don't have one so that's a guess.

[terahz]

@Derelict:

Your switch is routing between the VLANs, not the firewall.

I don't know why everyone keeps claiming that. As I've already said a few times, if I disable just one of the two vlan interfaces in the firewall,  routing stops working. I can see the traffic in the firewall logs and I can capture it there.

@Derelict:

You have to configure your hosts so the pfSense interface address is their default gateway.

That's what I've done, as visible by the traceroute above. Here is what a typical host's routing table looks like:

#route -nv
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
192.168.1.0     0.0.0.0         255.255.255.0   U     0      0        0 eth0
0.0.0.0         192.168.1.1     0.0.0.0         UG    0      0        0 eth0

@Derelict:

Do something like this on the switch:

no interface vlan99
no interface vlan10

If I do this I will not be able to manage the switch since these are the only two interfaces currently configured and I don't have the hardware at home to connect to the console. I just disabled vlan99 using no interface vlan99 and it didn't make a difference.

#conf t
(config)#no interface vlan99
(config)#show ip route
Code: C - connected, S - static
      * - candidate default

Gateway of last resort is not set

C    192.168.1.0/24 is directly connected, vlan10

Total Entries: 1 

…

Unfortunately, I just fixed it. I'm saying unfortunately, because the fix doesn't tell me what was wrong or how it got stuck in that state. I decided to shutdown the pfsense box just to prove that I can't route between the two vlans if it is out of the picture. The moment it went down, I could no longer access one vlan from the other. Unfortunately when I booted it up, the two vlans remained isolated and now I can see the blocks in the firewall logs (where earlier I was seeing the pass entries). Disabling the deny from any to GUEST net rule on the LAN immediately enables access. Enabling it, blocks it. Everything seems to be as expected now.

Oh well. Thanks everyone for the help.

[johnpoz]

So you never flushed your states then is what it sounds like..