Devices on different VLANs do not seem to be able to talk across firewall
-
@dlponladd said in Devices on different VLANs do not seem to be able to talk across firewall:
Port 8 - VLAN 1, 10-12 tagged
Port 10 - VLAN 10 taggedThe idea being that the WAN would be plugged in at port 10 and routed through the pfSense computer to the rest of the network.
Huh.. if your wan comes into pfsense - why is is tagged, and ok even if tagged - why would be be also taggged on port 8? If vlan 10 is the wan?
Maybe a basic drawing of what is plugged in where..
-
RE : pfSense laptop multiple NICs?
The pfSense laptop does not have multiple NICs. I based the initial configuration on this youtube video.
RE : Post interface assignments
Interface Assignments - pfSense
Interface / Network Port LAN / em0 WAN / VLAN10 on em0 -lan OPT1 / VLAN11 on em0 -lan OPT2 / VLAN12 on em0 -lanInterface Assignments - Cisco CB250 switch
Interface / Network Port / Mode / VLAN GE1 / 1 / Access / 1U GE2 / 2 / Disabled GE3 / 3 / Disabled GE4 / 4 / Access / 11U GE5 / 5 / Access / 12U GE6 / 6 / Disabled GE7 / 7 / Disabled GE8 / 8 / Trunk / 1U, 10-12T GE9 / 9 / Disabled GE10 / 10 / Trunk / 10TRE : Firewall Rules
All VLANs have the same rules : Allow All Traffic, based on the Default Allow Lan to Any rule. reference
RE : Links
(please note that any linked references are as much for me as for anyone else... I'm trying to keep notes that I can go back and reference later)
RE : WAN
Due to the hardware limitations of the laptop, I can only connect a single ethernet cable to it. The configuration of ports 8 and 10 are based on the video I linked above. The cliff notes version is that the goal is to force all traffic traveling to and from the WAN through the pfSense box using VLAN configuration.
That's what I should have written instead of "to the rest of the network".
I do not have the WAN hooked up for the testing I performed. I, er, wanted to try to wrap my head around what was going on before adding that piece into the puzzle.
RE : Network Diagrams
The following are diagrams of the two setups I was working with before posting here.
-
@dlponladd said in Devices on different VLANs do not seem to be able to talk across firewall:
Due to the hardware limitations of the laptop
There is nothing wrong with a router on a stick setup.. I wouldn't be trying to tag vlan 1 that is for sure.
But your test setup 2 should work without problem - forgetting wan.. with your 11 and 12 being access, ie untagged 11 and 12 and them with them tagged to pfsense.
There should be no reason that shouldn't work as long as your devices are pointing to pfsense as their gateway. And your rules are correct for the interfaces they are on.. Are they really any any, you didn't source network that might be wrong? Or set a gateway on rules?
Or set a gateway on the enterface where pfsense thinks its a "wan" connection.
-
RE : Rules
I just went back through and confirmed Permit Any-Any on all interfaces.
Just for kicks and giggles, I went and reset and retested the two setups to see if anything changed. Test results are the same.
RE : Gateway on Rules
I don't know what setting a gateway on rules does... so I just left it at the default. It says "*", which I think means "Any."
RE : Gateway on Interface
I have the interfaces set up as Static IPv4. There is an option for a gateway but it is currently blank.
-
@dlponladd well something is borked for sure.. This is clickly clicky work.
Pfsense is not a vm running on this laptop?
Your saying your 2 test boxes in both vlans get the correct IPs from dhcp on pfsense?
What IP ranges are you using? Each test box can ping its gateway, ie pfsense IP address on each vlan
-
RE : pfSense in VM
I had wanted to run pfSense in something akin to a dual-boot configuration but the installer had other ideas. And/or I didn't quite understand what was going on. Call it user error.
pfSense is not running inside a VM.
RE : IP Ranges
VLAN 1 is assigned 192.168.9.10 to 254
VLAN 11 is assigned 192.168.11.10 to 254
VLAN 12 is assigned 192.168.12.10 to 254[edit] Fixed the IP ranges for 11 and 12 to not be wrong.
Based on my observations, the firewall usually grabs the .1 address. The managed switch usually grabs the .10 address. The test computers grab the .11 address, typically. There is occasionally some weirdness where a computer will hold onto an IP address when I swap what port it is connected to on the switch. Sometimes it happens, sometimes not.
Both test boxes get correct IP addresses from DHCP on pfSense. I think it's from pfSense. I think the Managed Switch can do DHCP, but I also think I have that turned off. In any case, I don't see unexpected IPs.
RE : Pinging
I get normal pings from both the switch port and from the pfSense firewall/gateway. I am a little fuzzy on the terminology.
The IP address that shows up on the dashboard with the associated interface/VLAN. Each box can ping that for the VLAN that that box is connected to at the time of the test.
-
@dlponladd said in Devices on different VLANs do not seem to be able to talk across firewall:
, the firewall usually grabs the .1 address.
What do you mean grabs? Pfsense would have static IPs.
From one of your test boxes do a ping to pfsense other IP..
example.. My pc is 192.168.9.100, pfsense is .253, it has another interface on another network 192.168.3.253
Does that work?
$ ping 192.168.3.253 Pinging 192.168.3.253 with 32 bytes of data: Reply from 192.168.3.253: bytes=32 time=1ms TTL=64 Reply from 192.168.3.253: bytes=32 time=16ms TTL=64If you can ping pfsense 192.168.12.x IP from your 192.168.11.x test box, but you can't ping your 192.168.12.testbox IP then that screams firewall on the test box and nothing to do with pfsense.
Windows firewall for example out of the box would not allow a ping from 192.168.11.x/24 if it was on a 192.168.12.x/24 network.
-
RE : Grabs IP
My thought process was something along the lines of trying to make a distinction between what I tell it to assign during configuration and what it reports as being assigned on the Dashboard.
I'm not sure that's a meaningful distinction in this context... but that was what I was thinking when I wrote "...grabs the .1 address."
RE : Test Setup 2, ping from Config Computer to pfSense Interface for VLAN 12.
Config computer IP during test : 192.168.11.11
C:\>ping 192.168.12.1 Pinging 192.168.12.1 with 32 bytes of data: PING : transmit failed. General failure. PING : transmit failed. General failure. PING : transmit failed. General failure. PING : transmit failed. General failure. Ping statistics for 192.168.12.1: Packets: Sent = 4, Received = 0, Lost = 4 (100% lost).The response is quick, almost as if it isn't actually transmitting a ping. But I don't know enough to be sure one way or another. Those results don't seem to give me any information to work with.
I get identical results when trying to ping the switch port at 192.168.12.10 and the test computer at 192.168.12.11.
Just to demonstrate that it is actually connected to the switch, and the switch is connected to pfSense, I pinged the pfSense interface for VLAN 11.
C:\>ping 192.168.11.1 Pinging 192.168.11.1 with 32 bytes of data: Reply from 192.168.11.1: bytes=32 time<1ms TTL=64 Reply from 192.168.11.1: bytes=32 time<1ms TTL=64 Reply from 192.168.11.1: bytes=32 time<1ms TTL=64 Reply from 192.168.11.1: bytes=32 time<1ms TTL=64 Ping statistics for 192.168.11.1: Packets: Sent = 4, Received = 4, Lost = 0 (0% lost). Approximate round trip times in milli-seconds: Minimum = 0ms, Maximum = 0ms, Average = 0msAnd at the end of writing this entire response... I had a thought. What if... I tell pfSense to ping itself. From one interface to another.
PING 192.168.11.1 (192.168.11.1) from 192.168.12.1 : 56 data bytes 64 bytes from 192.168.11.1 : icmp_seq=1 ttl=64 time=0.072ms 64 bytes from 192.168.11.1 : icmp_seq=2 ttl=64 time=0.065ms 64 bytes from 192.168.11.1 : icmp_seq=3 ttl=64 time=0.082ms - - - 192.168.11.1 ping statistics - - - 3 packets transmitted, 3 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 0.065/0.073/0.082/0.007 msI don't pretend to understand what is going on under the hood. What is happening to the packets? Maybe this means that the pfSense configuration is working?
...I don't understand...
[edit : follow-up] Weirdly enough, adding a firewall rule to the VLAN 12 list to prevent this test (pinging interface from interface) doesn't seem to have an effect. Nor does it prevent pinging the switch port (#4), either. Both of the following tests performed from the pfSense GUI.
I put the block rule above the default allow rules previously mentioned.
Just as a reminder :
OPT1 = VLAN11 = 192.168.11.10 to 254
Switch port #4 is assigned VLAN11
OPT2 = VLAN12 = 192.168.12.10 to 254
Switch port #5 is assigned VLAN12I tried doing the following:
Attempt 1 :
Rule : Block IPv4*/Source OPT2 subnets/Port Any/Destination OPT1 subnets/Port Any/Gateway Any/Queue noneTest Ping 192.168.11.1 from 192.168.12.1 --> test pfSense interface
Result : no effect (ping goes through)Test Ping 192.168.11.10 from 192.168.12.1 --> test Switch port
Result : no effect (ping goes through)Attempt 2 :
Rule : Block IPv4*/Source Any/Port Any/Destination OPT1 subnets/Port Any/Gateway Any/Queue noneTest Ping 192.168.11.1 from 192.168.12.1 --> test pfSense interface
Result : no effect (ping goes through)Test Ping 192.168.11.10 from 192.168.12.1 --> test Switch port
Result : no effect (ping goes through)I had hoped to use the two computers in tandem to test and verify network configuration and firewall rules.
RE : Windows Firewall on Config/Test computers
I ran into some text somewhere in the pfSense documentation that suggested that firewalls might block ICMP echo requests by default. I decided to see if turning off the firewalls might do anything, and it did.
Before turning off the firewalls, I couldn't ping the test and config computers from either pfSense or the switch. After turning the firewalls off on both the test and config computers, I was able to successfully ping both computers from both the pfSense computer and the switch.
I poked around in the Advanced Firewall settings to see if there was some behind the scenes "I'm showing 'off' but I'm not really 'off'" configuration that wasn't showing from the basic Firewall menu. I didn't find anything, but I'm not much of an OS admin so I doubt if I would notice anything out of place.
I did find something that suggested that ICMP echo requests would be allowed through. But, with the firewall "off", I don't know what the expected behavior actually is.
-
@dlponladd said in Devices on different VLANs do not seem to be able to talk across firewall:
Pinging 192.168.12.1 with 32 bytes of data:
PING : transmit failed. General failure.That screams no gateway to me.. Your machine doesn't know how to get to 192.168.12.1 and its not on its local network.
Here I removed gateway from my pc, and then tried ping that 192.168.3.253 address
If your device on your 12 network doesn't have a gateway, then no you would never be able to talk to it from some other network.. Its gateway should point to pfsense IP on your 12 network.
-
RE : Gateway
So... I compared my ipconfig to your screenshots.
Both the config and test computers have "Connection-specific DNS Suffix". My Daily Driver does not have this filled out, nor does your screenshot.
I note that my Daily Driver has a Default Gateway, but that your screenshot and my test computers do not have that filled out.
My suspicion is that shows up in the DHCP server config on pfSense.
I have checked and I left the gateway setting in the DHCP server as default (it says firewall interface address).
Which suggests to me that this is a windows thing.
-
@dlponladd So neither of your test boxes have a gateway set? Well no wonder it would never work!
Your saying your clients get an IP from pfsense and their is no gateway?
So here is my client getting dhcp from pfsense. It was like that before because I set it to static and did not set a gateway to show you that general error points to not having a gateway. Here is normal dhcp listing
Notice it lists who the dhcp server is, and has a gateway.. Pointing to the same IP, because pfsense is the dhcp server and gateway (router to get off this network)..
If you do not have a gateway then no its not possible to talk to other networks, or for other networks to talk to you..
So your saying on your dhcp server in pfsense.. You have it like this
And its enabled and shows pfsense IP there, kind of greyed out? See how my lan has 192.168.9.253 there
And here is from the dhcp server on my 192.168.3.0/24 network
Dhcp isn't going to run if your interfaces are set for dhcp, which still confused why you would say "grabbed" if you actually set it on pfsense... This is how your interfaces should be set..
Your 11 and 12 interfaces should be static.. You have to actually set their IP if you want to run dhcp server on them.. And you want your dhcp server to hand out the pfsense IP in dhcp. Are you using ISC for your dhcp, or Kea? Kea is preview maybe there is something going on wrong there?
But no if your clients don't have a gateway that points to pfsense IP on the network they are on - then no your devices will never be able to talk to each other thru pfsense.
-
RE : IP From pfSense
Yep. They get an IP address correctly from pfSense based on the VLAN they are in.
But no Default Gateway.
RE : Normal DHCP listing
The ipconfig report from the test and configuration computers looks very similar to the picture, but the Default Gateway entry is blank.
It seems reasonable to me to expect the DHCP IP and the Default Gateway to be identical. Somehow, my computers are getting one and not the other.
I went diving into the windows Event Viewer, but I didn't see anything relevant.
RE : DHCP Server configuration
Yep. Gateway is left as default, greyed out. All the DHCP servers are configured this way.
RE : Interface configuration
I set up the interfaces with only Static IPv4 for testing purposes. I may include IPv6 later or go another route. I haven't made it that far in my research.
Other than that, the interfaces are set up very similar to the pictures you provided.
OPT1 is set up with static address 192.168.11.1/24.
OPT2 is set up with static address 192.168.12.1/24.RE : ISC/Kea
When I was doing the initial configuration and set up, there was a popup in pfSense saying that ISC was being depreciated and to use Kea. For the testing above, the DHCP servers were using the Kea server.
I set the DHCP server to ISC and rebooted pfSense (to be safe).
It doesn't seem to be setting the Default Gateway after the change. Both computers still get an appropriate IP address.
As an aside, I went through the process of manually resetting the ethernet adapter and flushing netsh. That didn't seem to change anything one way or another.
-
@dlponladd well that explains your issues then.. Cuz never going to work without a gateway.. You could sniff to see if not being offered or asked for..
Or if being asked for and just not offered.. Here is a sniff of my client doing a dhcp discover and getting its IP from pfsense dhcpd.
On the left is the discover, you can see in its request list it asks for the router, then on the right is the offer, and you can see dhcp sent the IP of pfsense.
As a quick test, I would just set your machines your using for test to be static IPs and set the gateway on them to point to pfsense IP on whatever network they are on. This should work then, and you could then dig into why your not getting the gateway from dhcp.
You could also manually put in the gateway in the dhcp server vs letting it do it by default - does that then hand out the gateway to the client?
-
RE : Default Gateway
Manually entering the gateway IP address into the DHCP server does, in fact, cause it to be sent to the attached computers. Woot!
But there's now some weirdness going on.
I did a bunch of testing to see what changed. Failures now are timeouts.
Config Computer Ping pfSense (any VLAN) : success Config Computer Ping Switch (any VLAN) : success Config Computer Ping Test Computer : failure Test Computer Ping pfSense (any VLAN) : success Test Computer Ping Switch (any VLAN) : success Test Computer Ping Config Computer : failure pfSense Ping Switch (any VLAN) : success pfSense Ping Config Computer (any VLAN) : success pfSense Ping Test Computer : mixed Same VLAN : success Different VLAN : failure Switch to pfSense (any VLAN) : success Switch to Config Computer : mixed Same VLAN : success Different VLAN : failure Switch to Test Computer : mixed VLAN 1 : success VLAN 4 : failure VLAN 5 : failure (the VLAN this one is attached to; test computer did successfully get IP address and Default Gateway).I don't know what's going on. The VLANs on the switch used to show up in the DHCP leases menu on pfSense and now they don't.
This doesn't seem right.I am going to bring the whole network down and do a cold start.
If it's still behaving weird after that, I may restore to factory configuration on both pfSense and the switch.
-
I just wanted to follow up after having had some time to test and tinker.
@johnpoz : Thanks for your help and patience! Your insight was invaluable.
RECAP : Issue
My original issue was identified by the supposed failure of pings to traverse through pfSense between two devices on different networks (ex. 192.168.11.xx and 192.168.12.xx).
RECAP : Issue No.1 : Windows Firewall Behavior
Important issue no.1 didn't have anything to do with pfSense or, for that matter, with the network in general. Windows firewall blocks ICMP Echo requests and this behavior seems to continue even with the firewall turned off in the Control Panel.
The weird part with this issue is that both pfSense AND the managed switch could ping both computers. The issue was revealed when the computers could not ping each other (pings timed out).
The simplest way to fix this behavior is to add an Allow Rule to Windows Firewall for ICMP behavior. Just... make sure to turn it off before using those test machines elsewhere.
RECAP : Issue No.2 : pfSense DHCP
Important issue no.2 had to do with weird behavior from the DHCP service on my pfSense machine. I cannot say if this is the result of a bug. I would have to do further testing (which I may follow up on later).
This was described by johnpoz as : "if your device... doesn't have a gateway, then you would never be able to talk to it from some other network."
Or even simpler : No door (gateway), no exit.
This issue was revealed by the ping attempt on one of the computers throwing a "General Failure" error when trying to ping the other computer. Investigation of ipconfig results confirmed the issue (missing network gateway).
The proposed solution that fixed the issue was simply to enter a value in the DHCP configuration screen : Other DHCP Options/Gateway. Adding a value here propagated to the two testing machines.
The value I used was the IP address of the associated firewall interface (... the default value...).
Fin
That's it.
Pings between the two computers works as expected, even when they are in different networks. The ping works in both directions.
Thanks again!
