Routing with L3 switch

[johnsonmathewv]

Hi,
I am using pfsense 2.3.4, and my L3 switch Brocade ICX 7250. I configured VLAN in L3 switch. I can ping the 8.8.8.8 & google.com from L3 switch but not pinging from PC, Ping details are below,

Firewall IP details
–-------------------
WAN Ip: 117.83.134.238/30
gateway ip: 117.83.134.237

Lan: 192.168.2.1/24

L3 Switch

Ip route: 0.0.0.0 0.0.0.0 192.168.2.1
dns: 192.168.2.1 8.8.8.8

eth 1/1/1  ip: 192.168.2.10/24

vlan 100
tagged eth 1/1/2
router-interface ve2 ip address 192.168.100.1/24

vlan 101
tagged eth 1/1/2
router-interface ve3 ip address 192.168.101.1/24

L2 Switch

vlan 100
tagged eth 1/1/1
untagged eth 1/1/2

vlan 100
tagged eth 1/1/1
untagged eth 1/1/3

PC

192.168.100.3/24
gateway: 192.168.100.1
P dns: 192.168.2.1
S DNs: 8.8.8.8

Ping from firewall to 8.8.8.8

PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: icmp_seq=0 ttl=59 time=65.019 ms
64 bytes from 8.8.8.8: icmp_seq=1 ttl=59 time=58.046 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=59 time=59.222 ms

Ping from firewall to L3

PING 192.168.2.10 (192.168.2.10): 56 data bytes
64 bytes from 192.168.2.10: icmp_seq=0 ttl=64 time=0.507 ms
64 bytes from 192.168.2.10: icmp_seq=1 ttl=64 time=0.456 ms
64 bytes from 192.168.2.10: icmp_seq=2 ttl=64 time=0.479 ms

--- 192.168.2.10 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.456/0.481/0.507/0.021 ms

Ping from firewall to VLAN100

PING 192.168.100.1 (192.168.100.1): from 192.168.2.1: 56 data bytes

--- 192.168.100.1 ping statistics ---
3 packets transmitted, 0 packets received, 100.0% packet loss

Ping from firewall to VLAN100

PING 192.168.100.3 (192.168.100.3) from 192.168.2.1: 56 data bytes

--- 192.168.100.3 ping statistics ---
3 packets transmitted, 0 packets received, 100.0% packet loss

Ping from L3 to firewall

ICX7250-24 Router#ping 192.168.2.1
Sending 1, 16-byte ICMP Echo to 192.168.2.1, timeout 5000 msec, TTL 64
Type Control-c to abort
Reply from 192.168.2.1    : bytes=16 time<1ms TTL=64
Success rate is 100 percent (1/1), round-trip min/avg/max=0/0/0 ms.

Ping from L3 to 4.2.2.2

ICX7250-24 Router#ping 4.2.2.2
Sending 1, 16-byte ICMP Echo to 4.2.2.2, timeout 5000 msec, TTL 64
Type Control-c to abort
Reply from 4.2.2.2        : bytes=16 time=159ms TTL=55
Success rate is 100 percent (1/1), round-trip min/avg/max=159/159/159 ms.

Ping from L3 to vlan100 pc

ICX7250-24 Router#ping 192.168.100.3
Sending 1, 16-byte ICMP Echo to 192.168.100.3, timeout 5000 msec, TTL 64
Type Control-c to abort
Reply from 192.168.100.3  : bytes=16 time=1ms TTL=64
Success rate is 100 percent (1/1), round-trip min/avg/max=1/1/1 ms.

Ping from L3 vlan100 to firewall & 4.2.2.2

ICX7250-24 Router#ping 192.168.2.1 source 192.168.100.1
Sending 1, 16-byte ICMP Echo to 192.168.2.1, timeout 5000 msec, TTL 64
Type Control-c to abort
Request timed out.
No reply from remote host.

ICX7250-24 Router#ping 4.2.2.2 source 192.168.100.1
Sending 1, 16-byte ICMP Echo to 4.2.2.2, timeout 5000 msec, TTL 64
Type Control-c to abort
Request timed out.
No reply from remote host.

Ping from pc

C:\Users\lenovo>ping 192.168.100.1

Pinging 192.168.100.1 with 32 bytes of data:
Reply from 192.168.100.1: bytes=32 time<1ms TTL=64
Reply from 192.168.100.1: bytes=32 time<1ms TTL=64
Reply from 192.168.100.1: bytes=32 time<1ms TTL=64
Reply from 192.168.100.1: bytes=32 time<1ms TTL=64

Ping statistics for 192.168.100.1:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 0ms, Maximum = 0ms, Average = 0ms

C:\Users\lenovo>ping 192.168.2.10

Pinging 192.168.2.10 with 32 bytes of data:
Reply from 192.168.2.10: bytes=32 time=25ms TTL=64
Reply from 192.168.2.10: bytes=32 time=2ms TTL=64
Reply from 192.168.2.10: bytes=32 time=44ms TTL=64
Reply from 192.168.2.10: bytes=32 time=2ms TTL=64

Ping statistics for 192.168.2.10:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 2ms, Maximum = 44ms, Average = 18ms

C:\Users\lenovo>ping 192.168.2.1

Pinging 192.168.2.1 with 32 bytes of data:
Request timed out.
Request timed out.
Request timed out.
Request timed out.

Ping statistics for 192.168.2.1:
    Packets: Sent = 4, Received = 0, Lost = 4 (100% loss),

C:\Users\lenovo>ping 8.8.8.8

Pinging 8.8.8.8 with 32 bytes of data:
Request timed out.
Request timed out.
Request timed out.
Request timed out.

Ping statistics for 8.8.8.8:
    Packets: Sent = 4, Received = 0, Lost = 4 (100% loss),

I think it's reverse routing issue in pfsense firewall, i tried to add the same in    System -> RoutingStatic -> Routes

but it's not working. can some one help me on this issue.

[awebster]

Yes, it is most likely a routing issue.

Make sure you add your L3 switch (192.168.2.10) as a gateway on the LAN interface
Next add a static routes to 192.168.100.0/24 and select the LAN gateway you just added from the drop-down list.
You might also want to repeat that for 192.168.101.0/24 so that both VLAN 100 and 101 are reachable.
Lastly, make sure you have firewall rules to allow these subnets out.

[Derelict]

I don't see a router-interface on the transit network to pfSense. Pretty sure you need that. What you have looks like it's for management only.

vlan X
untagged eth 1/1/1
router-interface ve1 ip address 192.168.2.10/24

1/1/1 connected to pfSense . Note that I would tag this on principle and assign transit to vlan X on pfSense. That way you can tag other VLANs to your switch if necessary without mixing tagged and untagged traffic (see brocade dual-mode ports for that).

I would not put the Layer 3 switch on LAN with a bunch of hosts. There should be two things on 192.168.2.0/24 - pfSense and the Switch. No other hosts or you will have asymmetric routing issues which are unnecessary and bad.

[awebster]

Derelict,
Nice diagram, very clear!

[coxhaus]

There are only 4 important things which you need to get right using a layer 3 switch. 
1. Make sure the layer 3 switch uses the LAN interface on pfsense for it's default gateway.
2.  With static routes pfsense needs routing statements for all networks on the layer 3 switch pointing to the gateway IP address on the layer 3 switch
3.  You need firewall statements on pfsense to allow all the networks on the layer 3 switch out through the firewall on pfsense
4. The PCs on the layer 3 switch need to have the layer 3 switch's network as their default gateway

I think these are the important steps which make a layer 3 switch work with pfsense or any router.

[Derelict]

5. Don't put any other hosts on LAN with the switch or you will have asymmetric routing issues.

[coxhaus]

@Derelict:

5. Don't put any other hosts on LAN with the switch or you will have asymmetric routing issues.

I have heard that before but it works if you let the layer 3 switch handle the local routing.  And if you don't believe me try it.

[Derelict]

It is still asymmetric and it is still bad design. Even if it works for you now it will likely bite you in the ass later.

[coxhaus]

It may be bad design but it happens even in production environments.

[Derelict]

Bad design is bad design, regardless of scale.

[coxhaus]

I think both agree on bad design.