Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Is this a routing problem or something else? pfSense&OpenStack (SOLVED)

    Scheduled Pinned Locked Moved General pfSense Questions
    23 Posts 4 Posters 5.1k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • T
      tsmalmbe
      last edited by

      Hello, I have just installed pfSense in an OpenStack environment where my WAN address is private. This installation is different to my other (working) installations in that it is a double NAT. And now I'm having I do not know how to solve.

      I have attached a simple network topology describing my setup.

      These statements describe the issue, based on the connectivity troubleshooting in the documentation:

      • pfSense is connected to two networks: 192.168.100 and 172.14.100
      • pfsense ip's are 192.168.100.5 (LAN) and 172.17.100.5 (WAN)
      • OpenStack Router is the WAN GW we are using: 172.17.100.50
      • pfsense from the shell can ping 8.8.8.8
      • pfsense can resolve names and download packages (internet connectivity is OK)
      • 192.168.100.6 can ping 192.168.100.5
      • 192.168.100.6 IS NOT able to ping 8.8.8.8 nor 172.17.100.5 nor 172.17.100.50
      • I can see the 192.168.100.6 as a source in my pfsense logs

      I'm using the default NAT rules.

      I have not used tcpdump to debug this, I'm not really familiar with that tool.

      Any tips on what to look at next greatly appreciated.

      NOTE. the pic says .14 which is wrong, it is .17
      openstack-topologia.png
      openstack-topologia.png_thumb

      Security Consultant at Mint Security Ltd - www.mintsecurity.fi

      1 Reply Last reply Reply Quote 0
      • M
        muswellhillbilly
        last edited by

        For starters, 192.168.100.6 should have the PFS LAN IP set as it's default gateway (192.168.100.5). Is this correct?
        Secondly, are you using the default installed firewall rules, or have you changed them in any way?

        1 Reply Last reply Reply Quote 0
        • T
          tsmalmbe
          last edited by

          The default gateway for .6 is set as .5. This is actually set up in the OpenStack dhcp for that network, so the linuxbox should set itself up very correctly. Anyway, I have checked the default gw and the ip several times and they are correct.

          The firewall rules are indeed customized. I have allowed all egress traffic (ping, tcp, udp) from the subnet where the .6 machine is. There is some ingress traffic also allowed, but that should not be relevant (at this stage), correct?

          Security Consultant at Mint Security Ltd - www.mintsecurity.fi

          1 Reply Last reply Reply Quote 0
          • M
            muswellhillbilly
            last edited by

            @tsmalmbe:

            The default gateway for .6 is set as .5. This is actually set up in the OpenStack dhcp for that network, so the linuxbox should set itself up very correctly. Anyway, I have checked the default gw and the ip several times and they are correct.

            Hold on - Are you saying Openstack is handling DHCP for your LAN (192.168.100.x) clients? When you say the gateway for .6 is .5, you do mean that the gateway for 192.168.100.6 is 192.168.100.5, right? If you actually mean that 192.168.100.6 has it's gateway set to 172.14.100.5, then that won't work.

            @tsmalmbe:

            The firewall rules are indeed customized.

            How about cutting to the chase and posting your firewall rules up instead of making people guess these things? You never mentioned at the start that you had customised the default firewall rules. I suggest you start with the default ruleset which will absolutely work provided your routing is correct. You can then make the changes to the default you need and test each time to see that you haven't broken anything.

            1 Reply Last reply Reply Quote 0
            • T
              tsmalmbe
              last edited by

              What I am saying is

              • The gateway for 192.168.100.6 is set to 192.168.100.5 and that 192.168.100.6 is able to ping the gateway ip 192.168.100.5
              • The only added active rule is the egress rule that allows everything out. Of course I can revert to the default rules here if that would make the difference.

              Security Consultant at Mint Security Ltd - www.mintsecurity.fi

              1 Reply Last reply Reply Quote 0
              • T
                tsmalmbe
                last edited by

                Cutting to the chase as prescribed, these are the firewall rules that mainly allow everything out and nothing else.

                FW rules WAN:
                IPv4 TCP * * This Firewall 443 (HTTPS) * none   Allow HTTPS to WWW config

                FW rules INTERNAL:

                    • INTERNAL Address 443 * * Anti-Lockout Rule
                      IPv4 * INTERNAL net * * * * none   Default allow LAN to any rule

                FW rules DMZ:
                IPv4 ICMP DMZ net * * * * none   Allow all ICMP
                Pv4 TCP/UDP DMZ net * * * * none   Allow all TCP and UDP

                The DMZ interface is just created, there is really nothing there yet as I wanted even one server in the INTERNAL segment to get connectivity first.

                Security Consultant at Mint Security Ltd - www.mintsecurity.fi

                1 Reply Last reply Reply Quote 0
                • johnpozJ
                  johnpoz LAYER 8 Global Moderator
                  last edited by

                  "172.14.100.5"

                  Where is double nat? 172.14 is a public IP..

                  NetRange:      172.0.0.0 - 172.15.255.255
                  CIDR:          172.0.0.0/12
                  Organization:  AT&T Internet Services (SIS-80)

                  "- 192.168.100.6 IS NOT able to ping 8.8.8.8 nor 172.14.100.5 nor 172.14.100.50"

                  your saying pfsense can not even ping pfsense wan IP 172.14.100.5??  Well something is wrong then..  And if you can not even ping pfsense wan I wouldn't expect you to ping anything past it either.

                  An intelligent man is sometimes forced to be drunk to spend time with his fools
                  If you get confused: Listen to the Music Play
                  Please don't Chat/PM me for help, unless mod related
                  SG-4860 24.11 | Lab VMs 2.8, 24.11

                  1 Reply Last reply Reply Quote 0
                  • T
                    tsmalmbe
                    last edited by

                    Yes, there is a typo in the picture. It is not .14 it is .17 (which indeed is private).

                    Security Consultant at Mint Security Ltd - www.mintsecurity.fi

                    1 Reply Last reply Reply Quote 0
                    • T
                      tsmalmbe
                      last edited by

                      @johnpoz:

                      "172.14.100.5"

                      Where is double nat? 172.14 is a public IP..

                      NetRange:      172.0.0.0 - 172.15.255.255
                      CIDR:          172.0.0.0/12
                      Organization:  AT&T Internet Services (SIS-80)

                      "- 192.168.100.6 IS NOT able to ping 8.8.8.8 nor 172.14.100.5 nor 172.14.100.50"

                      your saying pfsense can not even ping pfsense wan IP 172.14.100.5??  Well something is wrong then..  And if you can not even ping pfsense wan I wouldn't expect you to ping anything past it either.

                      192.168.100.6 is the server in the INTERNAL network. pfsense is 192.168.100.5 - the gateway.
                      192.168.100.6 can ping 192.168.100.5 but nothing beyond it.

                      Security Consultant at Mint Security Ltd - www.mintsecurity.fi

                      1 Reply Last reply Reply Quote 0
                      • johnpozJ
                        johnpoz LAYER 8 Global Moderator
                        last edited by

                        192.168.1.5???  Where is that - do you mean 192.168.100.5??

                        Can you ping pfsense WAN IP???  172.17.x.x ???  If not then yeah you have something wrong on your box or on pfsense rules..

                        here this is me from a machine on my lan 192.168.9/24 pinging my pfsense wan interface IP, and then pinging through to my isp gateway.

                        user@ubuntu:~$ ping 24.13.public
                        PING 24.13.public (24.13.public) 56(84) bytes of data.
                        64 bytes from 24.13.public: icmp_seq=1 ttl=64 time=0.393 ms
                        64 bytes from 24.13.public: icmp_seq=2 ttl=64 time=0.407 ms
                        64 bytes from 24.13.public: icmp_seq=3 ttl=64 time=0.290 ms

                        –- 24.13.public ping statistics ---
                        3 packets transmitted, 3 received, 0% packet loss, time 2000ms
                        rtt min/avg/max/mdev = 0.290/0.363/0.407/0.054 ms
                        user@ubuntu:~$ ping 24.13.publicgateway
                        PING 24.13.publicgateway (24.13.publicgateway) 56(84) bytes of data.
                        64 bytes from 24.13.publicgateway: icmp_seq=1 ttl=254 time=12.1 ms
                        64 bytes from 24.13.publicgateway: icmp_seq=2 ttl=254 time=12.3 ms
                        64 bytes from 24.13.publicgateway: icmp_seq=3 ttl=254 time=10.1 ms

                        --- 24.13.publicgateway ping statistics ---
                        3 packets transmitted, 3 received, 0% packet loss, time 2003ms
                        rtt min/avg/max/mdev = 10.187/11.574/12.344/0.986 ms
                        user@ubuntu:~$

                        If you can not do this when your firewall rules allows it then yeah you have a problem if you can ping pfsense wan IP, but not pfsense gateway you sure its gateway answers ping??  You sure your natting on pfsense, if not your a downstream router and your upstream router has to know how to get to your network behind pfsense and to go past has to allow and nat that network.  If your double natting you should be fine.  But first your going to have to be able to get to pfsense wan from its lan.

                        An intelligent man is sometimes forced to be drunk to spend time with his fools
                        If you get confused: Listen to the Music Play
                        Please don't Chat/PM me for help, unless mod related
                        SG-4860 24.11 | Lab VMs 2.8, 24.11

                        1 Reply Last reply Reply Quote 0
                        • T
                          tsmalmbe
                          last edited by

                          @johnpoz:

                          But first your going to have to be able to get to pfsense wan from its lan.

                          Which pretty much is the issue at hand here.

                          Security Consultant at Mint Security Ltd - www.mintsecurity.fi

                          1 Reply Last reply Reply Quote 0
                          • johnpozJ
                            johnpoz LAYER 8 Global Moderator
                            last edited by

                            well then you dicked with the default rules?  Or you have the wrong interfaces setup?  Or you configured somethong not right with masks on your clients? Out of the box pfsense would allow you ping your wan IP that is for sure!  You say you can ping its lan IP.. You sure your pinging pfsense lan?

                            So your making pfsense a downstream router, so you are natting?  Which is out of the box the default.  If not you would have issues if the upstream router doesn't know where this downstream network is..  But that has nothing to do with simple ping to pfsense own wan IP from lan client.

                            An intelligent man is sometimes forced to be drunk to spend time with his fools
                            If you get confused: Listen to the Music Play
                            Please don't Chat/PM me for help, unless mod related
                            SG-4860 24.11 | Lab VMs 2.8, 24.11

                            1 Reply Last reply Reply Quote 0
                            • T
                              tsmalmbe
                              last edited by

                              The dicking part - I should be able to see that in the logs, correct?

                              I have triplediplechecked the interfaces for typos. I also run two other pfSenses and doublecheck (sanitycheck) the setups against those.

                              perhaps I need to triplediplechek the actual server once more.

                              It really is the simplest setup, what's what baffles me. The only new thing is the doublenat - but as pfsense is itself able to communicate outbounds… it baffles me even more.

                              Security Consultant at Mint Security Ltd - www.mintsecurity.fi

                              1 Reply Last reply Reply Quote 0
                              • johnpozJ
                                johnpoz LAYER 8 Global Moderator
                                last edited by

                                well depends..  Here is a question is your lan the default any any rule or did you modify or add some rules?  Did you mess with the outbound nat?  192.168.100 is not the default lan network.  So if you had changed your outbound nat to manual or something and didn't put it in right then you would have issues getting anywhere, etc.

                                An intelligent man is sometimes forced to be drunk to spend time with his fools
                                If you get confused: Listen to the Music Play
                                Please don't Chat/PM me for help, unless mod related
                                SG-4860 24.11 | Lab VMs 2.8, 24.11

                                1 Reply Last reply Reply Quote 0
                                • T
                                  tsmalmbe
                                  last edited by

                                  the lan any-any is the default rule:
                                  IPv4 * INTERNAL net * * * * none   Default allow LAN to any rule

                                  there is also the dafult antilockout rule.

                                  NAT - I have not changed. It is automatic.

                                  Security Consultant at Mint Security Ltd - www.mintsecurity.fi

                                  1 Reply Last reply Reply Quote 0
                                  • johnpozJ
                                    johnpoz LAYER 8 Global Moderator
                                    last edited by

                                    so then if that is the case you should be able to ping the pfsense wan IP no matter what IP it is..  So look at your nat rules and make sure they show your 192.168.100 network..

                                    And your floating rules tab is empty?

                                    outboundnat.jpg
                                    outboundnat.jpg_thumb

                                    An intelligent man is sometimes forced to be drunk to spend time with his fools
                                    If you get confused: Listen to the Music Play
                                    Please don't Chat/PM me for help, unless mod related
                                    SG-4860 24.11 | Lab VMs 2.8, 24.11

                                    1 Reply Last reply Reply Quote 0
                                    • T
                                      tsmalmbe
                                      last edited by

                                      Affirmative on both - the network is in the NAT-list and there are no floating rules.

                                      Security Consultant at Mint Security Ltd - www.mintsecurity.fi

                                      1 Reply Last reply Reply Quote 0
                                      • johnpozJ
                                        johnpoz LAYER 8 Global Moderator
                                        last edited by

                                        so your client at 192.168.100.6 can ping 192.168.100.5, what is the gateway on 192.168.100.6 box?  And you validated that 192.168.100.5 that your pinging and that your using as your gateway is actually the pfsense box via your arp table on the 192.168.100.5 box..

                                        An intelligent man is sometimes forced to be drunk to spend time with his fools
                                        If you get confused: Listen to the Music Play
                                        Please don't Chat/PM me for help, unless mod related
                                        SG-4860 24.11 | Lab VMs 2.8, 24.11

                                        1 Reply Last reply Reply Quote 0
                                        • T
                                          tsmalmbe
                                          last edited by

                                          The settings regarding this I will have to triple-re-check.

                                          Security Consultant at Mint Security Ltd - www.mintsecurity.fi

                                          1 Reply Last reply Reply Quote 0
                                          • T
                                            tsmalmbe
                                            last edited by

                                            For people in or from the future. This proved to be the solution:
                                            https://ask.openstack.org/en/question/26980/problem-using-pfsense-vm-inside-a-tenant/
                                            http://www.honnix.com/technology/software/cloud/network/2015/11/24/pfsense-as-router-in-openstack

                                            The issue is how openstack works, not how pfSense works.

                                            Openstack really is not click-drag-drop-works, it's a lot of overall fiddling and tuning (unless you just want one plain server directly on the internet).

                                            Security Consultant at Mint Security Ltd - www.mintsecurity.fi

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.