Routing between interfaces/VLANs



  • I found several threads with similar issues, however none of the solutions worked for me.

    I have the following setup:

    Interface TRUSTED (renamed from the default LAN) with 192.168.3.0/24 network, no VLAN
    Interface GASTLAN with 192.168.150.0/24 network, VLAN 3
    Interface OpenVPN (auto-created by enabling OpenVPN Servers)

    From all 3 of them access to the internet works. From OpenVPN Clients I can access hosts on TRUSTED (because the TRUSTED-net is defined as local network on the OpenVPN-Servers).

    However, no comm is possible from GASTLAN to a host in TRUSTED.

    For troubleshooting I created any-to-any rules on all 3 of these zones. I tried to find the culprit of my problem using the packet-filter, however, listening on GASTLAN-interface for any traffic to a specified IP (host on TRUSTED) brings an empty result.

    Heres my rules:

    TRUSTED.png

    GASTLAN.png

    OpenVPN.png

    Can anybody point me in the right direction where I have my thinking-issue?



  • Block private networks unchecked on TRUSTET interface?



  • yes



  • logs?



  • which/where to find?


  • LAYER 8 Global Moderator

    @techvic said in Routing between interfaces/VLANs:

    listening on GASTLAN-interface for any traffic to a specified IP (host on TRUSTED) brings an empty result.

    If you see ZERO traffic to an IP in the trusted network hit your gastlan interface - how would pfsense route it if nothing there..

    your client in gastlan not using pfsense as its gateway? Is it running some vpn client on itself?



  • @johnpoz said in Routing between interfaces/VLANs:

    your client in gastlan not using pfsense as its gateway? Is it running some vpn client on itself?

    it is using pfsense as gateway (192.168.150.1). Since access to the internet works thru that, this shouldn't be the reason?


  • LAYER 8 Global Moderator

    @techvic said in Routing between interfaces/VLANs:

    it is using pfsense as gateway (192.168.150.1). Since access to the internet works thru that, this shouldn't be the reason?

    Well then what you say is not possible, or the client is pointing to something else for route to your lan network..

    If you sniff on this gast interface, pfsense packet capture under diag... And you don't see any traffic - how could pfsense ever possible route it??

    Here... Sniffing on my dmz (192.168.3/24), pinging box on my lan.. 192.168.9.100, the firewall doesn't allow the traffic on the dmz... But you can still see the traffic hitting pfsense dmz interface.

    pingcapt.jpg



  • 
    19:10:17.589300 IP 192.168.150.36.56489 > 17.248.146.9.443: tcp 570
    19:10:17.589730 IP 192.168.150.36.56489 > 17.248.146.9.443: tcp 106
    19:10:17.603093 IP 17.248.146.9.443 > 192.168.150.36.56489: tcp 0
    19:10:17.605634 IP 17.248.146.9.443 > 192.168.150.36.56489: tcp 1440
    19:10:17.605653 IP 17.248.146.9.443 > 192.168.150.36.56489: tcp 1183
    19:10:17.608889 IP 192.168.150.36.56489 > 17.248.146.9.443: tcp 0
    19:10:17.645570 IP 192.168.150.36.56489 > 17.248.146.9.443: tcp 620
    19:10:17.645913 IP 192.168.150.36.56489 > 17.248.146.9.443: tcp 106
    19:10:17.659340 IP 17.248.146.9.443 > 192.168.150.36.56489: tcp 0
    19:10:17.661371 IP 17.248.146.9.443 > 192.168.150.36.56489: tcp 1097
    19:10:17.665183 IP 192.168.150.36.56489 > 17.248.146.9.443: tcp 0
    19:10:17.686632 IP 192.168.150.36.56489 > 17.248.146.9.443: tcp 620
    19:10:17.687001 IP 192.168.150.36.56489 > 17.248.146.9.443: tcp 106
    19:10:17.700343 IP 17.248.146.9.443 > 192.168.150.36.56489: tcp 0
    19:10:17.702104 IP 17.248.146.9.443 > 192.168.150.36.56489: tcp 1097
    19:10:17.705444 IP 192.168.150.36.56489 > 17.248.146.9.443: tcp 0
    19:10:17.726415 IP 192.168.150.36.56489 > 17.248.146.9.443: tcp 620
    19:10:17.726785 IP 192.168.150.36.56489 > 17.248.146.9.443: tcp 106
    19:10:17.740321 IP 17.248.146.9.443 > 192.168.150.36.56489: tcp 0
    19:10:17.741835 IP 17.248.146.9.443 > 192.168.150.36.56489: tcp 474
    
    

    192.168.150.36 is a Macbook on GASTLAN while trying to ping 192.168.3.1. But all you can see there is traffic to the internet caused by background processes, no ICMP


  • LAYER 8 Global Moderator

    Set your sniff to icmp to only the the IP your pinging... See my example..

    What is prob your issue can tell you right now is your dest box host firewall..



  • @johnpoz said in Routing between interfaces/VLANs:

    Set your sniff to icmp to only the the IP your pinging... See my example..

    Empty result.

    I'm pinging the pfsense on TRUSTED (192.168.3.1). When the Macbook does that from within TRUSTED the pfsense replies, so how can it be that firewall blocking?


  • LAYER 8 Global Moderator

    Dude if your not seeing anything on the sniff of pfsense for 192.168.3.1 - then its NOT going to pfsense... Maybe you have your masks wrong and your using 192.168/16 ??



  • sorry my fault, when I did the test couple of minutes ago I did not have the any-to-any rules set

    with them I see these on packet sniff:

    
    20:22:59.622362 ARP, Request who-has 192.168.3.1 tell 192.168.150.36, length 42
    20:23:02.814369 ARP, Request who-has 192.168.3.1 tell 192.168.150.36, length 42
    
    

    however, still no reply to the ping


  • LAYER 8 Global Moderator

    that is a arp. and that tells me you have mask wrong... Since a 192.168.150/24 address would never arp for a 192.168.3.x address.

    Firewall rules have ZERO to do with a packet capture.. You could have zero rules, you could have all deny.. You would still see traffic hitting the interface via sniff.

    And would explain why not working... Since if 192.168.150.x thinks 192.168.3.x thinks they are the same network, it would never send traffic to pfsense to get "routed" to the other network.

    Your mask is wrong on the 192.168.150.36 machine. You prob have /16 (255.255.0.0) vs /24 (255.255.255.0)



  • I see. However, where else to configure if not here?

    Bildschirmfoto 2020-01-08 um 04.57.29.png

    Bildschirmfoto 2020-01-08 um 04.59.34.png


  • LAYER 8 Global Moderator

    On your actual client.. 192.168.150.36

    Did you set it static? Is it windows? you can view what the mask is with an ipconfig

    $ ipconfig
    
    Windows IP Configuration
    
    Ethernet adapter Local:
    
       Connection-specific DNS Suffix  . :
       IPv4 Address. . . . . . . . . . . : 192.168.9.100
       Subnet Mask . . . . . . . . . . . : 255.255.255.0
       Default Gateway . . . . . . . . . : 192.168.9.253
    


  • Same on windows and mac:

    Bildschirmfoto 2020-01-08 um 05.05.37.png

    Bildschirmfoto 2020-01-08 um 05.12.36.png


  • LAYER 8 Global Moderator

    Well there is something broken then on it... Because you do not ARP for an IP that is not on your network... You don't

    So 192.168.150.36/24 would NOT arp for a 192.168.3.1 address just not how it works..

    So why are you seeing arps for that address, vs sending traffic to pfsense at 192.168.150.1 to route it for you.

    Your saying the windows machine is doing the same thing? 192.168.150.39

    do the same test you did using the windows machine .39, try and ping 192.168.3.1 (pfsense interface on the lan).. Do still see these arps for that 3.1 address from 150.39 address?

    A device on network A, does not "arp" for addresses on network B... Can not work, does not work - there is no point in ever doing it.. A device will only arp for an IP that is on its network. So if your seeing arps from a device on 192.168.150/24 for a 192.168.3.x address - there is something broken! That device does not have another interface does it, a wireless one on the 192.168.3 network??

    This is how it works..

    device with
    192.168.150.36/24
    gw 192.168.150.1

    Wants to talk to 192.168.3.1
    if do not have the mac address of its gateway 192.168.150.1
    It would arp for 192.168.150.1, once it gets the mac address of 150.1
    Send traffic to that mac, dest for 192.168.3.1

    There is never a scenario where it would directly arp for 192.168.3.1 since that is NOT on its own network.



  • maybe the logical architecture of the network has something to do with it?

    Here's how this setup looks like:

    pfsense > VLAN 3 taggged connected to switch > VLAN 3 tagged to Wifi Access Points > Access Points broadcasts the VLAN 3 thru it's SSID as untagged.

    For the Windows-Machine: thats a VM on ESXi, where ESXi passes the VLAN 3 as an untagged network to the VM

    Everything is Level 2 Switching, so no routing involved by the switches

    All clients are DHCP, nothing static. The DHCP is running on pfsense. So if anything is misconfigured, the source must be in pfsense


  • LAYER 8 Global Moderator

    @techvic said in Routing between interfaces/VLANs:

    maybe the logical architecture of the network has something to do with it?

    No doesn't matter... Comes down to how tcp works at a basic level... The client would never arp for an IP outside its network... Because that is not how it works..

    A client would never on its own arp for that IP, since with the settings on its interface that IP is not on its network.. So there would be no way it could ever see the arp... Arps are at Layer 2.. Could you manually send out such an arp - sure.. But the OS should never do such a thing on its own.

    If your seeing arps for that 3.1 address from 150.36/24 then it thinks for whatever reason that 3.1 is part of its local network.

    If devices arped for IPs that are not on its network - every single IP you talk to on the internet - the client would arp for...

    You have something wrong for sure with that client to why you can not send traffic to its gateway 150.1.. Could it be something odd going on with esxi - ok maybe.. Not sure on the details of your setup... But in all the years working with esxi, no I can not think of reason such a thing would happen. Nor have I seen it... it makes no sense for anything to arp for an IP outside its network..

    But what would explain it completely is that client thinks 192.168.3.1 is on the same network as 192.168.150 - so mask of /16 would do that..

    Reset the networking on that client doing the arps...

    And again - are you seeing the same sort of problem from your 150.39 device?

    The DHCP is running on pfsense. So if anything is misconfigured, the source must be in pfsense

    Well that makes no sense, since your pfsense shows mask of /24... Client "shows" a mask of 24... So to validate - do a simple sniff of yoru dhcp traffic... What is being handed to the client?

    edit:

    Do you have something on the network that might have the same duplicate IP sending those arps? Maybe it has wrong mask on it... What is your esxi vmkern IP? etc. etc..

    edit2: Look at the actual details of the arp your seeing... Is the MAC of the arp from your your client?

    arp.png

    Sniff on your windows machine.. While your pinging, that 192.168.3.1 address -- do you see it arp for the gateway IP? Do you see it actually send the pings?

    wireshark is FREE, download it and install it on your client your having problems with... And again is your other machine having the same problem? You have given 2 machine 150.39 and .36 - are both of them doing the same exact thing??



  • @johnpoz said in Routing between interfaces/VLANs:

    And again - are you seeing the same sort of problem from your 150.39 device?

    no, indeed on the Windows VM (192.168.150.39) everything works as it should. So there I can ping 192.168.3.1 and can access other hosts on the 192.168.3.0/24 net.

    So for testing I connected a physical windows machine to the Wifi on 192.168.150.0/24, where I see the same issue as on the Mac yesterday (nothing to see on the packet capture at all). The machine gets a DHCP lease (192.168.150.21) and has access to the internet, but can't reach any host on 192.168.3.0/24.

    Then I connected this machine by cable to the switch to a port with VLAN 3 untagged and PVID 3. There everything works.

    So the culprit must be somewhere around the wireless system (TP-Link EAP managed Wifi). How is that possible, that accessing Internet works normal but anything to another net is not passing the Wifi system? here's a packet capture of opening google.com on the client thru Wifi:

    17:31:54.819122 IP 143.204.101.25.443 > 192.168.150.21.52711: tcp 411
    17:31:54.824123 IP 188.172.219.132.5938 > 192.168.150.21.52647: tcp 24
    17:31:54.853304 IP 192.168.150.21.52712 > 143.204.101.114.443: tcp 0
    17:31:54.854656 IP 192.168.150.21.56643 > 192.168.150.1.53: UDP, length 46
    17:31:54.854834 IP 192.168.150.1.53 > 192.168.150.21.56643: UDP, length 110
    17:31:54.861782 IP 192.168.150.21.57307 > 192.168.150.1.53: UDP, length 46
    17:31:54.876373 IP 143.204.101.114.443 > 192.168.150.21.52712: tcp 0
    17:31:54.879854 IP 192.168.150.21.52712 > 143.204.101.114.443: tcp 0
    17:31:54.879907 IP 192.168.150.1.53 > 192.168.150.21.57307: UDP, length 127
    

    Sure I'm aware this is nothing related to pfSense anymore, but any ideas would be appreciated.


  • LAYER 8 Global Moderator

    @techvic said in Routing between interfaces/VLANs:

    that accessing Internet works normal but anything to another net is not passing the Wifi system? here's a packet capture of opening google.com on the client thru Wifi:

    Because!! 143.204.25.443 is NOT on a 192.168/16 network - so it would actually send this traffic to the gateway..

    When you say wireless.. You sure client is actually getting lease from pfsense? And not your wireless doing nat with 192.168.150 on both sides?

    So sniff on pfsense and ping pfsense IP from this client 192.168.150.1 Do you see this traffic in your packet capture? In the sniff check the mac address, validate its your actual clients mac address, and not the mac address of your wireless device.



  • I found the culprit: because this GASTLAN is a net for guests, there on the Wifi-settings were "client isolation" enabled. Due to that, it blocked any traffic not bound to the internet. I even wasn't able to ping the gateway 192.168.150.1

    Thank you very much for your patience and not giving up to find the solution till the end. Thats very rare nowadays


  • LAYER 8 Global Moderator

    But WTF was arping 3.1??? That makes no sense at all.

    Not sure how your doing your wifi - but why would it be set to a guest, when its a specific vlan already.. Which you control at pfsense..



  • @johnpoz said in Routing between interfaces/VLANs:

    Not sure how your doing your wifi - but why would it be set to a guest, when its a specific vlan already.. Which you control at pfsense..

    thats why I disabled that feature now > not needed.

    @johnpoz said in Routing between interfaces/VLANs:

    But WTF was arping 3.1??? That makes no sense at all.

    It's TP-Link, China quality, wouldn't surprise me if it's just a bug (wouldn't be the first one). Or is it a hidden spy feature? 👀


  • LAYER 8 Global Moderator

    What IP is on the tplink device - maybe its mask is wrong?


  • LAYER 8 Moderator

    @johnpoz said in Routing between interfaces/VLANs:

    What IP is on the tplink device - maybe its mask is wrong?

    Or the VLAN isolation isn't throrough and leaking through. I thought to remember sth like that from the smallest TP-Link Switches a few years ago...



  • @JeGr said in Routing between interfaces/VLANs:

    Or the VLAN isolation isn't throrough and leaking through. I thought to remember sth like that from the smallest TP-Link Switches a few years ago...

    And also TP-Link access points. I have one here and can't configure a 2nd SSID because of leaks between VLAN and native.


Log in to reply