Can't HTTP, HTTPS, SSH or Map drive across networks

[rmoroni3]

I have pfSense setup with 5 NIC ports used as follows:
WAN1
VLAN10_Private_LAN
VLAN11_VM_Hosts
VLAN12_Smart_Devices_WIFI
WAN2
VLAN10 in where my PC sits, VLAN11 is where my virtual machine host server sits and VLAN12 is where all of my wireless and Smart Home devices are.

My problem is I cannot http, https, ssh or map a drive from VLAN10 to VLAN11. I get this site can't be reached or host not found. If I put my PC on VLAN11, everything works just fine. I can ping and tracert from VLAN 10 to the host server in VLAN 11 just fine.

I have tried booting my PC in safe mode and made sure all Windows firewalls were turned off and all anti-virus programs not running to try to rule out where the problem is. This didn't make any difference so I FEEL it is something in pfSense but I have no idea what.

Right now the firewall rules for the two networks are wide open allowing any protocol, port source and destination so that SHOULDN'T be the problem.

If someone could give me some idea what to check out, I would REALLY appreciate it!

I set it up this way to isolate devices for security reasons.

[bingo600]

@rmoroni3

Sounds a bit like a DNS issue to me.

When you say you can ping the device on VL11 , is that by dns-name or by ip ?

/Bingo

[rmoroni3]

@bingo600 :
Everything is by IP address. I am trying to http or https from my PC on 192.168.10.2 to a VMware Host server on 192.168.11.20. You have to use http://192.168.11.20 to get to the tools like the VMware Client which in turn uses https://192.168.11.20 to actually manage it. None of this works unless my PC is on the same network as the server. I'll try to post a Wire Shark packet capture of what the traffic looks like when it won't connect.

[Griffo]

@rmoroni3 Wireshark will help, you can also do a packet trace on teh firewall itself on the two different interfaces, so you can see what's entering the interface and what's exiting.

You say that your devices has 5 NIC's and then mention vlans. Can you clarify which interfaces are physical and which are vlans?

I'd be double checking all your vlan tagging configs on your physical and virtual switches too.

[rmoroni3]

@griffo
I have two dual-port Intel PRO/1000 NICs plus the RealteK gigabit NIC built-in on the mobo. The Intel NICs are used for the WAN1, VLAN10, VLAN11 and VLAN12 networks. The Realtek NIC is used for WAN2 which is the WiFi Internet failover in case my cable Internet goes down. I can tether that NIC to my cellphone which is set up as a hotspot so I can keep working and all of my Smart devices still work.

When I first embarked on this adventure I started to use VLANs but decided not to. I never changed the network names. SO, there aren't any VLANs, just the 5 separate networks. I really should change that unless VLANs is what will solve the problem!

You mention virtual switches. I don't have any. I do have two unmanaged switches, one plugged into the 192.168.10.X network, and one plugged into the 192.168.11.X network.

I work on the packet capture tomorrow.

[johnpoz]

Well sniff on the destination network interface on pfsense. Then send traffic from the source network.. Do you see pfsense send on the traffic to the destination IP?

If so then its not pfsense.

You say you can ping.. Lets see your rules on the source network interface.

[rmoroni3]

@johnpoz
Here are the requested screenshots/packet captures. When I tried to capture the traffic on VLAN11 from my PC to the server on pfSense, it was blank so I captured the traffic from VLAN10.

I tried to upload the actual packet capture files but it didn't look like they uploaded properly. I got an error on both when I tried. Sorry!

[rmoroni3]

@johnpoz
Here are two traceroutes on pfSense:

The only difference is I turned on ICMP on the one and it completed. I find the interesting and I am sure it is probably telling me something but what??

[johnpoz]

When I tried to capture the traffic on VLAN11 from my PC to the server on pfSense, it was blank

When you ping? Then your not pinging what you think your pinging.

Lets see the sniff on dest vlan interface while you ping from your source network.

Do you have any rules in floating? For example shoving traffic out a specific wan gateway. As I see from your posted rules your doing something or trying to with multiple wans?

How exactly are you doing the captures? Just do packet capture on pfsense.. There is no point in sniffing on the source side to be honest.. Since your not getting any response.. But to rule out pfsense you need to see it sending traffic on your dest interface to whatever your trying to talk to in the destination network.

If your saying its blank, but you can ping - then your not pinging what your think your pinging. Or you would of seen that in the sniff (packet capture)

edit: One thing I could see for sure presenting itself exactly like what your seeing is a floating rule shoving traffic out your wan for tcp.. Or tcp/udp while your normal lan rule with no policy routing allows for the ping.

[rmoroni3]

@johnpoz
Here are some screenshots. Here is the sniff on the dest vlan while pinging from my PC on vlan 10:

Here is the sniff on the same setup but trying to https fo the destination:

As you can see it IS blank.

Here is the firewall Floating rule:

Now, just for chuckles, I disabled the floating rule and now everything works. Here is the sniff while connecting from my PC the VMware host server:

I am not sure I fully understand what is going on. Looking at the rule it sort of makes sense but some explanation would be a great help.

I didn't think to look at the floating rule. Now my Internet failover doesn't work. It was a pain to get it working in the first place and now it is broken. I suppose I need to open a new post for that issue, correct?

Thanks for all of your help on this. This was really frustrating me and I need this to work the way I want it to. Now I can go back and change the rules the way I want them to be.

[johnpoz]

See that rule there in your floating - its forcing traffic that is TCP, ie http or https out your gateway. So no your not going to get to your other vlans.

See the bypassing policy routing section here.

https://docs.netgate.com/pfsense/en/latest/multiwan/policy-route.html#bypassing-policy-routing

You need a rule that allows the traffic to your other vlans - before you shove traffic out the wan.. Which isn't going get your vlans.