Routing with L3 switch



  • Hi,
    I am using pfsense 2.3.4, and my L3 switch Brocade ICX 7250. I configured VLAN in L3 switch. I can ping the 8.8.8.8 & google.com from L3 switch but not pinging from PC, Ping details are below,

    Firewall IP details
    –-------------------
    WAN Ip: 117.83.134.238/30
    gateway ip: 117.83.134.237

    Lan: 192.168.2.1/24

    L3 Switch

    Ip route: 0.0.0.0 0.0.0.0 192.168.2.1
    dns: 192.168.2.1 8.8.8.8

    eth 1/1/1  ip: 192.168.2.10/24

    vlan 100
    tagged eth 1/1/2
    router-interface ve2 ip address 192.168.100.1/24

    vlan 101
    tagged eth 1/1/2
    router-interface ve3 ip address 192.168.101.1/24

    L2 Switch

    vlan 100
    tagged eth 1/1/1
    untagged eth 1/1/2

    vlan 100
    tagged eth 1/1/1
    untagged eth 1/1/3

    PC

    192.168.100.3/24
    gateway: 192.168.100.1
    P dns: 192.168.2.1
    S DNs: 8.8.8.8

    Ping from firewall to 8.8.8.8

    PING 8.8.8.8 (8.8.8.8): 56 data bytes
    64 bytes from 8.8.8.8: icmp_seq=0 ttl=59 time=65.019 ms
    64 bytes from 8.8.8.8: icmp_seq=1 ttl=59 time=58.046 ms
    64 bytes from 8.8.8.8: icmp_seq=2 ttl=59 time=59.222 ms

    Ping from firewall to L3

    PING 192.168.2.10 (192.168.2.10): 56 data bytes
    64 bytes from 192.168.2.10: icmp_seq=0 ttl=64 time=0.507 ms
    64 bytes from 192.168.2.10: icmp_seq=1 ttl=64 time=0.456 ms
    64 bytes from 192.168.2.10: icmp_seq=2 ttl=64 time=0.479 ms

    --- 192.168.2.10 ping statistics ---
    3 packets transmitted, 3 packets received, 0.0% packet loss
    round-trip min/avg/max/stddev = 0.456/0.481/0.507/0.021 ms

    Ping from firewall to VLAN100

    PING 192.168.100.1 (192.168.100.1): from 192.168.2.1: 56 data bytes

    --- 192.168.100.1 ping statistics ---
    3 packets transmitted, 0 packets received, 100.0% packet loss

    Ping from firewall to VLAN100

    PING 192.168.100.3 (192.168.100.3) from 192.168.2.1: 56 data bytes

    --- 192.168.100.3 ping statistics ---
    3 packets transmitted, 0 packets received, 100.0% packet loss

    Ping from L3 to firewall

    ICX7250-24 Router#ping 192.168.2.1
    Sending 1, 16-byte ICMP Echo to 192.168.2.1, timeout 5000 msec, TTL 64
    Type Control-c to abort
    Reply from 192.168.2.1    : bytes=16 time<1ms TTL=64
    Success rate is 100 percent (1/1), round-trip min/avg/max=0/0/0 ms.

    Ping from L3 to 4.2.2.2

    ICX7250-24 Router#ping 4.2.2.2
    Sending 1, 16-byte ICMP Echo to 4.2.2.2, timeout 5000 msec, TTL 64
    Type Control-c to abort
    Reply from 4.2.2.2        : bytes=16 time=159ms TTL=55
    Success rate is 100 percent (1/1), round-trip min/avg/max=159/159/159 ms.

    Ping from L3 to vlan100 pc

    ICX7250-24 Router#ping 192.168.100.3
    Sending 1, 16-byte ICMP Echo to 192.168.100.3, timeout 5000 msec, TTL 64
    Type Control-c to abort
    Reply from 192.168.100.3  : bytes=16 time=1ms TTL=64
    Success rate is 100 percent (1/1), round-trip min/avg/max=1/1/1 ms.

    Ping from L3 vlan100 to firewall & 4.2.2.2

    ICX7250-24 Router#ping 192.168.2.1 source 192.168.100.1
    Sending 1, 16-byte ICMP Echo to 192.168.2.1, timeout 5000 msec, TTL 64
    Type Control-c to abort
    Request timed out.
    No reply from remote host.

    ICX7250-24 Router#ping 4.2.2.2 source 192.168.100.1
    Sending 1, 16-byte ICMP Echo to 4.2.2.2, timeout 5000 msec, TTL 64
    Type Control-c to abort
    Request timed out.
    No reply from remote host.

    Ping from pc

    C:\Users\lenovo>ping 192.168.100.1

    Pinging 192.168.100.1 with 32 bytes of data:
    Reply from 192.168.100.1: bytes=32 time<1ms TTL=64
    Reply from 192.168.100.1: bytes=32 time<1ms TTL=64
    Reply from 192.168.100.1: bytes=32 time<1ms TTL=64
    Reply from 192.168.100.1: bytes=32 time<1ms TTL=64

    Ping statistics for 192.168.100.1:
        Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
    Approximate round trip times in milli-seconds:
        Minimum = 0ms, Maximum = 0ms, Average = 0ms

    C:\Users\lenovo>ping 192.168.2.10

    Pinging 192.168.2.10 with 32 bytes of data:
    Reply from 192.168.2.10: bytes=32 time=25ms TTL=64
    Reply from 192.168.2.10: bytes=32 time=2ms TTL=64
    Reply from 192.168.2.10: bytes=32 time=44ms TTL=64
    Reply from 192.168.2.10: bytes=32 time=2ms TTL=64

    Ping statistics for 192.168.2.10:
        Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
    Approximate round trip times in milli-seconds:
        Minimum = 2ms, Maximum = 44ms, Average = 18ms

    C:\Users\lenovo>ping 192.168.2.1

    Pinging 192.168.2.1 with 32 bytes of data:
    Request timed out.
    Request timed out.
    Request timed out.
    Request timed out.

    Ping statistics for 192.168.2.1:
        Packets: Sent = 4, Received = 0, Lost = 4 (100% loss),

    C:\Users\lenovo>ping 8.8.8.8

    Pinging 8.8.8.8 with 32 bytes of data:
    Request timed out.
    Request timed out.
    Request timed out.
    Request timed out.

    Ping statistics for 8.8.8.8:
        Packets: Sent = 4, Received = 0, Lost = 4 (100% loss),

    I think it's reverse routing issue in pfsense firewall, i tried to add the same in    System -> RoutingStatic -> Routes

    but it's not working. can some one help me on this issue.



  • Yes, it is most likely a routing issue.

    Make sure you add your L3 switch (192.168.2.10) as a gateway on the LAN interface
    Next add a static routes to 192.168.100.0/24 and select the LAN gateway you just added from the drop-down list.
    You might also want to repeat that for 192.168.101.0/24 so that both VLAN 100 and 101 are reachable.
    Lastly, make sure you have firewall rules to allow these subnets out.


  • Netgate

    I don't see a router-interface on the transit network to pfSense. Pretty sure you need that. What you have looks like it's for management only.

    vlan X
    untagged eth 1/1/1
    router-interface ve1 ip address 192.168.2.10/24

    1/1/1 connected to pfSense . Note that I would tag this on principle and assign transit to vlan X on pfSense. That way you can tag other VLANs to your switch if necessary without mixing tagged and untagged traffic (see brocade dual-mode ports for that).

    I would not put the Layer 3 switch on LAN with a bunch of hosts. There should be two things on 192.168.2.0/24 - pfSense and the Switch. No other hosts or you will have asymmetric routing issues which are unnecessary and bad.




  • Derelict,
    Nice diagram, very clear!



  • There are only 4 important things which you need to get right using a layer 3 switch. 
    1. Make sure the layer 3 switch uses the LAN interface on pfsense for it's default gateway.
    2.  With static routes pfsense needs routing statements for all networks on the layer 3 switch pointing to the gateway IP address on the layer 3 switch
    3.  You need firewall statements on pfsense to allow all the networks on the layer 3 switch out through the firewall on pfsense
    4. The PCs on the layer 3 switch need to have the layer 3 switch's network as their default gateway

    I think these are the important steps which make a layer 3 switch work with pfsense or any router.


  • Netgate

    5. Don't put any other hosts on LAN with the switch or you will have asymmetric routing issues.



  • @Derelict:

    5. Don't put any other hosts on LAN with the switch or you will have asymmetric routing issues.

    I have heard that before but it works if you let the layer 3 switch handle the local routing.  And if you don't believe me try it.


  • Netgate

    It is still asymmetric and it is still bad design. Even if it works for you now it will likely bite you in the ass later.



  • It may be bad design but it happens even in production environments.


  • Netgate

    Bad design is bad design, regardless of scale.



  • I think both agree on bad design.