Routing subnet through another router



  • Hi all,

    I'm relatively new to pfSense, so I'm probably missing something obvious here, or doing something stupid. Hoping someone here can point me in the right direction  :)

    Ok, done a quick sketch to help explain, so here's the setup:

    ZyXEL routers at each site, with a VPN between them. Site 1 uses subnet 192.168.0.0/24 and site 2 uses subnet 192.168.10.0/24. Site 1 can route to any computer at site 2 and vice versa (ping, etc, all working perfectly and has been for a couple of years or more).

    Now, the plan is to upgrade Site 2 to use a pfSense device, but to migrate over to it gradually, and in stages. The pfSense device has been installed at Site 2 with it's own WAN IP and it has been connected (with DHCP disabled) to the existing 192.168.10.0/24 network with a static IP of 192.168.10.254. So far, so good; the computers at site 2 can connect to the LAN and the internet using either the USG20W or the pfSense unit as their gateway.

    However, neither the pfSense device nor any computers using it as their gateway, are able to route to or ping any computers at site 2 (ie 192.168.0.0/24) and pfSense cannot ping the USG20W, even though it can ping everything else on the network and everything else can ping the USG20W. I've tried adding the USG20W as a gateway to pfSense, but this didn't work.

    I'd prefer not to connect to the VPN directly using pfSense since the VPN connection is to be replaced with a BT Short Haul Data Services site-to-site link soon, which will connect directly to the pfSense device (on a different LAN/subnet).

    Any suggestions would be greatly appreciated!  :)

    EDIT: Using pfSense 2.2-RELEASE (amd64)



  • @Zaphod:

    I've tried adding the USG20W as a gateway to pfSense, but this didn't work.

    Did you add a static route to 192.168.0.0/24 via 192.168.10.1?
    I would also check the Static route filtering box under advanced, firewall.

    edit-moved reply out of quote


  • LAYER 8 Netgate

    Do you have enough interfaces (or you can use a VLAN) to put a direct connection between pfSense and the USG at site 2?



  • Thanks very much for the replies guys  :)

    @dotdash:

    @Zaphod:

    I've tried adding the USG20W as a gateway to pfSense, but this didn't work.

    Did you add a static route to 192.168.0.0/24 via 192.168.10.1?
    I would also check the Static route filtering box under advanced, firewall.

    I did set a static route but I didn't have the Static Route Filtering box checked. Tried that, but still no difference unfortunately.

    When I add 192.168.10.1 as a gateway, it always shows as offline (unless I disable monitoring to force it online) and pfSense gets no ping response from it either, even though the USG20W responds to pings from any computer on the network.

    @Derelict:

    Do you have enough interfaces (or you can use a VLAN) to put a direct connection between pfSense and the USG at site 2?

    I could probably spare an interface, though I'm setting this up remotely, so I would have to do any physical reconfiguring next time I'm on-site, but I can remotely create VLANs of course. Why do you think a direct connection or VLAN would make a difference, and how should I configure/allocate the interface?

    Thanks again guys :)

    Here's some screenshots, in case there's something blindingly obvious I'm doing wrong here …


  • Banned

    Why are you having so many GWs? The BTnet and WAN1_DHCP look like duplicate ones. Plus, when the GW is DHCP, shouldn't it be dynamic instead of hardcoded?



  • @doktornotor:

    Why are you having so many GWs? The BTnet and WAN1_DHCP look like duplicate ones. Plus, when the GW is DHCP, shouldn't it be dynamic instead of hardcoded?

    You can ignore those; they're a work in progress. The issue remains whether they're enabled or not.

    I was testing different multi-wan failover and load balancing configurations. Some of those gateways will be removed/reconfigured later.

    Thanks  :)



  • The ping problem is a show-stopper. That has to "just work" on a normal LAN.
    What is the switch connecting USG20W and pfSense LANs? Anything fancy about it?
    Does USG20W have some sort of firewall blocking the ping?
    Can you ping in the other direction - USG20W to pfSense?

    When you solve that, then you will have asymmetric routing - packets coming back from 192.168.0.0/24 will be delivered directly by USG20W to clients in 192.168.10.0/24. pfSense will not see them and so there will be trouble with states.

    One way to get around that is to NAT out on LAN for traffic leaving pfSense LAN destined to 192.168.0.0/24 - that way when the reply comes, USG20W will deliver it back to pfSense LAN interface. pfSense will unNAT it and deliver to the client.



  • @phil.davis:

    The ping problem is a show-stopper. That has to "just work" on a normal LAN.

    That's what I was thinking.

    And what seems to make no sense is that I can ping any computer from pfSense and any computer can ping either pfSense or the USG20W, yet pfSense can't ping the USG20W.

    @phil.davis:

    What is the switch connecting USG20W and pfSense LANs? Anything fancy about it?

    Fancy-ish. They connect through a HP 1920-48G, which is a managed switch, but presently unconfigured (factory defaults). The computers pass through the same switch though and can ping each other and both the USG20W and pfSense unit, without any issues.

    @phil.davis:

    Does USG20W have some sort of firewall blocking the ping?

    No. It responds to pings from everything else. I've even tried turning its firewall off.

    @phil.davis:

    Can you ping in the other direction - USG20W to pfSense?

    As far as I can tell, there's no way to send a ping from the USG20W.

    @phil.davis:

    When you solve that, then you will have asymmetric routing - packets coming back from 192.168.0.0/24 will be delivered directly by USG20W to clients in 192.168.10.0/24. pfSense will not see them and so there will be trouble with states.

    One way to get around that is to NAT out on LAN for traffic leaving pfSense LAN destined to 192.168.0.0/24 - that way when the reply comes, USG20W will deliver it back to pfSense LAN interface. pfSense will unNAT it and deliver to the client.

    Ah, I hadn't thought about that. I can see I probably need to rethink this.

    Would NAT-ing allow me to pass all 192.168.0.0/24 bound traffic through a single 192.168.10.0/24 IP, or would I need to dedicate a full subnet to it?

    Thanks very much for your help :)



  • Would NAT-ing allow me to pass all 192.168.0.0/24 bound traffic through a single 192.168.10.0/24 IP, or would I need to dedicate a full subnet to it?

    Just let it NAT to the pfSense LAN IP. Then devices in 192.168.0.0/24 will think that all the traffic is coming from pfSense LAN IP.
    If there are more specific filtering rules that you want to apply related to particular 192.168.10.0/24 addresses being allowed across the inter-site VPN then the NATting to pfSense LAN IP defeats that, and you would implement such filtering at pfSense LAN.

    Pain about the ping! I can't think what to say about that.


  • LAYER 8 Netgate

    However, neither the pfSense device nor any computers using it (ETA: USG20W) as their gateway, are able to route to or ping any computers at site 2 (ie 192.168.0.0/24) and pfSense cannot ping the USG20W, even though it can ping everything else on the network and everything else can ping the USG20W. I've tried adding the USG20W as a gateway to pfSense, but this didn't work.

    USG20W <– 172.25.243.2/30 ----- .1 --> pfSense <== WAN & LAN interfaces

    Firewall rules on the 172.25.243.0/30 interfaces passing all traffic, since you're replacing an unfiltered layer 2 subnet in the first place.

    Add a gateway on pfSense for 172.25.243.2.  Add a route to pfSense for 192.168.0.0/24 to that gateway

    Add a route on the USG20W for 192.168.10.0/24 to 172.25.243.1, however you do that.

    You probably need to disable the interface to 192.168.10.0/24 on the USG20W.  It will have two routes to 192.168.10.0/24 and will probably prefer the connected interface over the static route by default.

    Now all your computers put all their traffic through one gateway, pfSense 192.168.10.254, regardless of destination.  This is what you want.  You eliminate all the same interface "routing" nonsense and the USG20W can be replaced any time.  If the BT Short Haul Data Services site-to-site link is a layer 3 device, you might consider making the link subnet larger than a /30 so you can just put that device on it, add a gateway for it to pfSense, and route what you want to it without using the same IP as the USG20W.

    No idea what to tell you about being able to ping the USG.  Sounds like something else is wrong.



  • Regarding the communication between the two, have you done a packet capture to see if anything it actually sent or received on the proper interface? You should at least be able to see outgoing ICMP packets, and if they are not blocked by the ZyXEL, ICMP responses.



  • Thanks Derelict, phil.davis! I'll try those suggestions.

    @johnkeates:

    Regarding the communication between the two, have you done a packet capture to see if anything it actually sent or received on the proper interface? You should at least be able to see outgoing ICMP packets, and if they are not blocked by the ZyXEL, ICMP responses.

    I hadn't. I'm pretty new to pfSense, to be honest, and my networking knowledge certainly has room for improvement. I'm not sure if I'm doing this right, or what it tells me, but this is the result (I selected LAN1 and a host address of 192.168.10.1 then ran the capture while pinging the USG20W from pfSense):

    10:02:31.726354 ARP, Request who-has 192.168.10.222 tell 192.168.10.1, length 46
    10:02:32.725723 ARP, Request who-has 192.168.10.222 tell 192.168.10.1, length 46
    10:02:33.562867 IP 192.168.10.254 > 192.168.10.1: ICMP echo request, id 15725, seq 0, length 64
    10:02:33.725296 ARP, Request who-has 192.168.10.222 tell 192.168.10.1, length 46
    10:02:34.587525 IP 192.168.10.254 > 192.168.10.1: ICMP echo request, id 15725, seq 1, length 64
    10:02:35.184593 ARP, Request who-has 192.168.10.222 tell 192.168.10.1, length 46
    10:02:35.554343 ARP, Request who-has 192.168.10.1 (ff:ff:ff:ff:ff:ff) tell 192.168.10.83, length 46
    10:02:35.613409 IP 192.168.10.254 > 192.168.10.1: ICMP echo request, id 15725, seq 2, length 64
    10:02:35.861489 ARP, Request who-has 192.168.10.1 (ff:ff:ff:ff:ff:ff) tell 192.168.10.81, length 46
    10:02:36.184124 ARP, Request who-has 192.168.10.222 tell 192.168.10.1, length 46
    10:02:37.183662 ARP, Request who-has 192.168.10.222 tell 192.168.10.1, length 46
    10:02:42.609838 ARP, Request who-has 192.168.10.1 tell 192.168.10.152, length 46
    10:02:43.676265 ARP, Request who-has 192.168.10.1 tell 192.168.10.153, length 46
    10:02:44.750631 ARP, Request who-has 192.168.10.52 (ff:ff:ff:ff:ff:ff) tell 192.168.10.1, length 46
    10:02:44.900685 ARP, Request who-has 192.168.10.52 (ff:ff:ff:ff:ff:ff) tell 192.168.10.1, length 46
    10:02:45.070560 ARP, Request who-has 192.168.10.52 (ff:ff:ff:ff:ff:ff) tell 192.168.10.1, length 46
    10:02:45.229967 ARP, Request who-has 192.168.10.221 tell 192.168.10.1, length 46
    10:02:45.230532 ARP, Request who-has 192.168.10.52 (ff:ff:ff:ff:ff:ff) tell 192.168.10.1, length 46
    10:02:45.400446 ARP, Request who-has 192.168.10.52 (ff:ff:ff:ff:ff:ff) tell 192.168.10.1, length 46
    10:02:45.489836 ARP, Request who-has 192.168.10.222 tell 192.168.10.1, length 46
    10:02:45.540933 ARP, Request who-has 192.168.10.52 (ff:ff:ff:ff:ff:ff) tell 192.168.10.1, length 46
    10:02:45.713759 ARP, Request who-has 192.168.10.52 (ff:ff:ff:ff:ff:ff) tell 192.168.10.1, length 46
    10:02:45.870172 ARP, Request who-has 192.168.10.52 (ff:ff:ff:ff:ff:ff) tell 192.168.10.1, length 46
    10:02:46.489379 ARP, Request who-has 192.168.10.222 tell 192.168.10.1, length 46
    10:02:47.488877 ARP, Request who-has 192.168.10.222 tell 192.168.10.1, length 46
    

  • LAYER 8 Netgate

    pfSense is pinging out and not getting a response.

    10:02:33.562867 IP 192.168.10.254 > 192.168.10.1: ICMP echo request, id 15725, seq 0, length 64



  • 10:02:33.562867 IP 192.168.10.254 > 192.168.10.1: ICMP echo request, id 15725, seq 0, length 64
    

    Those ones are pfSense sending out ping (echo request). There should be responses back with addresses the other way around. They do not appear, which I guess we already know!
    If the USG20W has some packet capture then you should do that looking for packets from 192.168.10.254 - to establish if the ping packet ever arrives at USG20W.



  • Like the others posted: the ping is going out, but ZyXEL isn't responding at all. Since this is a capture, it has nothing to do with the firewall or routes, as it's in the L2 interface level, nothing has been done to the packets yet.

    I think you might have a problem on the ZyXEL side, not on the pfSense side.

    Your trace is correct at least, so now you know how to capture packets, which is a great tool with network diagnostics! Beware though, packet captures are not always useful when for example you have a firewall or routing problem since firewalling and routing happen after the packets leave the interface and to in to processing software like pf.


Log in to reply