Issues with routing…



  • Hey Guys,
    This is my first post to this forum so please bare with me.  For the last few days, I've been trying to get the routing working between my Layer3 HP 5500 switch to the pfSense with no luck.  It seems as if the traffic only works one direction in that we can communicate with the multiple VLAN addresses or PC's on my HP L3 switch from anywhere but from the VLAN IP's on the HP switch or from a PC, the traffic never gets out.  Seems weird to me since when the flow works one way, it should normally work the other.  The setup is basic as follows…

    Internet
                                          |
                                    em1 - WAN
                                      pfSense
            192.168.10.1 - em0        em3 - 192.168.253.17/28 - route 10.96.16.0 255.255.248.0 192.168.253.18
                                    |            |
                  OLD L2 Switch            HP 5500 L3 - 192.168.253.18/28 - route 0.0.0.0 0.0.0.0 192.168.253.17
    VLAN1:192.168.10.0/24            VLAN1:192.168.253.18 IP 192.168.253.18, VLAN16:10.96.16.0/23 IP:10.96.16.1
                  |                                                                                                    |
          PC1:192.168.10.100                                                                      PC2:10.96.16.2

    Ping from PC1 to PC2 works perfectly
    Ping from PC1 to 8.8.8.8 work perfectly
    Ping from PC1 to VLAN1 on HP5500 L3 works perfectly
    Ping from PC1 to VLAN16 on HP5500 L3 work perfectly
    Ping from PC2 to PC1, fails
    Ping from PC2 to 8.8.8.8, fails
    Ping from VLAN16 to PC1, fails
    Ping from VLAN16 to 8.8.8.8 fails

    If I'm plugged into the OLD L2 Switch, I can ping anything within it's own subnet and anything within the other HP 5500 switch.  But if I connect to the PC1 on the 10.96.16.0/23 subnet, I can't reach other on the WEB or in any of the other subnets.  I'm thinking it has something to do with NATTING because until I switch the NAT rules from Automatic to Manual and added the following OUTBOUND NAT "EM3, any, *, *, *, EM3 address, *, YES", nothing worked!  Long story short, I'm trying to do all my routing on the HP L3 switch instead of on the pfSense.

    Any ideas of what I'm doing wrong?

    (See attached image for some of the pfSense configurations)




  • if it works from one direction to the other, then generally, routing is working.

    so, big chance its a firewall issue: either on the client or the firewall itself



  • Thanks for the reply.

    The FW rules on each of the interfaces are PASS everything so I don't know where it's failing.  The thing that bugs me is that NAT was how we got it working in one direction so I'm sure it has something to do with that.  Here are some attachments of the configuration and a TCP dump from the interface em3.

    You'll see that in the TCP DUMP the 10.96.16.2 (PC2) is trying to reach its configured DNS server of 8.8.8.8 because it's trying to do a Windows update.  It never reaches it because somehow there is no return path.





  • LAYER 8 Netgate

    As was said, look at the local "software" firewall on the host you're trying to ping.

    If you are using manual outbound NAT you need to add an entry for 10.96.16.0/23 on your WAN.

    I believe automatic NAT is now smart enough to add the entry for the routed subnet.



  • As well as the firewall of the host you are trying to ping, if you are using the routing features of the HP, you likely need to add some static routes on the HP for it to know where to find the PC1 network on the other PfSense interface.

    This would explain why the NAT is causing it to work, as it is now shown as coming from the PfSense interface facing the HP, not the network of PC1.



  • @Derelict:

    As was said, look at the local "software" firewall on the host you're trying to ping.

    If you are using manual outbound NAT you need to add an entry for 10.96.16.0/23 on your WAN.

    I believe automatic NAT is now smart enough to add the entry for the routed subnet.

    There is no software/hardware firewall on the HP 5500 L3 switch and it is unable to ping anything outside of the pfSense interface.  I also tested on the PC1 and 2 to make sure there is no software FW.

    Automatic NAT breaks the communication altogether and as you'll see, there is already an outbound NAT for 10.96.16.0/23 on the WAN.


  • LAYER 8 Netgate

    Did you set a default gateway in the switch?

    This stuff really does work without much hassle.  Don't overthink it.

    ETA - I see you say you have entered the default gateway into the swtich.  Anything in the firewall logs?

    EATA - both PCs on your OP are labelled PC1.



  • @Derelict:

    Did you set a default gateway in the switch?

    This stuff really does work without much hassle.  Don't overthink it.

    Yes the default gateway is set. I know it should normally work without issues but we're really stumped on this one.


  • LAYER 8 Netgate

    Both PCs on your OP are labelled PC1


  • LAYER 8 Netgate

    Get rid of that WH NAT rule.  It's nonsensical.



  • @Derelict:

    Get rid of that WH NAT rule.  It's nonsensical.

    I agree but if I remove that WH NAT rule, it breaks everything and I can't even ping the HP 5500 L3 interface.  It's a weird one.


  • LAYER 8 Netgate

    It's wrong.  Get rid of it.  It NATs the source address of all connections going OUT the WH interface to the WH interface address.  If you did not have a route in place before, that might have appeared to fix some routing, but it was really just making things appear to be coming from the WH subnet.



  • @Derelict:

    It's wrong.  Get rid of it.  It NATs the source address of all connections going OUT the WH interface to the WH interface address.  If you did not have a route in place before, that might have appeared to fix some routing, but it was really just making things appear to be coming from the WH subnet.

    It's gone now and I'm on automatic so here is what appears.  (see attached)

    ![new nat.JPG](/public/imported_attachments/1/new nat.JPG)
    ![new nat.JPG_thumb](/public/imported_attachments/1/new nat.JPG_thumb)


  • LAYER 8 Netgate

    OK.  It did not pick up the NAT for the routed subnet.  You should have a gateway defined in pfSense for 192.168.253.18 and a route defined for 10.96.16.0 255.255.248.0 with that gateway as the destination. If NAT still doesn't have an entry for the 10.96.16.0/21 you'll need to add one using hybrid or manual mode.



  • @Derelict:

    OK.  It did not pick up the NAT for the routed subnet.  You should have a gateway defined in pfSense for 192.168.253.18 and a route defined for 10.96.16.0 255.255.248.0 with that gateway as the destination. If NAT still doesn't have an entry for the 10.96.16.0/21 you'll need to add one using hybrid or manual mode.

    The route was already there and when the NAT is on Automatic, the 10.96.16.0/21 shows up but I have no communication until I add the manual NAT  "EM3, any, *, *, *, EM3 address, *, YES".


  • LAYER 8 Netgate

    Dude.  Look at the automatic NAT screen you posted.  The NAT entry for 10.96.16.0/21 is not there.

    I am telling you you are doing it wrong. You can either listen or not. If you are going to just dismiss what I say just let me know so I can stop wasting my time.

    There is a very good reason adding that NAT entry makes some connectivity happen but doesn't fix everything as I explained above.

    Get rid of the NAT entry for the WH interface and add a hybrid Outbound NAT rule for the 10.96.16/21 subnet on WAN.

    Then post how you configured the routes and gateway in System > Routing.



  • @Derelict:

    Dude.  Look at the automatic NAT screen you posted.  The NAT entry for 10.96.16.0/21 is not there.

    I am telling you you are doing it wrong. You can either listen or not. If you are going to just dismiss what I say just let me know so I can stop wasting my time.

    There is a very good reason adding that NAT entry makes some connectivity happen but doesn't fix everything as I explained above.

    Get rid of the NAT entry for the WH interface and add a hybrid Outbound NAT rule for the 10.96.16/21 subnet on WAN.

    Then post how you configured the routes and gateway in System > Routing.

    Sorry about that, the screenshot was from when I was testing.  Here is the latest screenshot but I'm not sure I understand if you still want me to add anything manual since it shows up on the WAN?

    ![new nat hybrid.JPG](/public/imported_attachments/1/new nat hybrid.JPG)
    ![new nat hybrid.JPG_thumb](/public/imported_attachments/1/new nat hybrid.JPG_thumb)



  • I'm going to do another test because I'm beginning to this it might have something to do with the pfSense and the fact that it's been running for a long time and that there may be some bad configuration we don't see.  This is standard routing so it should be simple.  I'm going to take another device we have, install a fresh copy and start the config from scratch.  In the meantime, I'm still opened to suggestions cause it would be great to fix it rather then start over.  Thanks.


  • LAYER 8 Netgate

    Yeah.  Post how you configured the gateway and the route like I asked for in the previous message.



  • Looks like this one for the books because I was able to get everything working from a fresh install.  I'm guessing there was an inherited setting from all the past upgrades that we weren't seeing in the WebConfigurator.  All the settings are now identical to the configuration I posted earlier with Automatic NAT and it worked right away.  Same configuration, same rules, same subnets, same connections on nearly identical hardware.

    Thanks for  your help.


Log in to reply