Is this a routing problem or something else? pfSense&OpenStack (SOLVED)

tsmalmbe · Sep 12, 2016, 6:57 PM

Cutting to the chase as prescribed, these are the firewall rules that mainly allow everything out and nothing else.

FW rules WAN:
IPv4 TCP * * This Firewall 443 (HTTPS) * none Allow HTTPS to WWW config

FW rules INTERNAL:

- - INTERNAL Address 443 * * Anti-Lockout Rule
    IPv4 * INTERNAL net * * * * none Default allow LAN to any rule

FW rules DMZ:
IPv4 ICMP DMZ net * * * * none Allow all ICMP
Pv4 TCP/UDP DMZ net * * * * none Allow all TCP and UDP

The DMZ interface is just created, there is really nothing there yet as I wanted even one server in the INTERNAL segment to get connectivity first.

johnpoz · Sep 12, 2016, 7:28 PM

"172.14.100.5"

Where is double nat? 172.14 is a public IP..

NetRange: 172.0.0.0 - 172.15.255.255
CIDR: 172.0.0.0/12
Organization: AT&T Internet Services (SIS-80)

"- 192.168.100.6 IS NOT able to ping 8.8.8.8 nor 172.14.100.5 nor 172.14.100.50"

your saying pfsense can not even ping pfsense wan IP 172.14.100.5?? Well something is wrong then.. And if you can not even ping pfsense wan I wouldn't expect you to ping anything past it either.

tsmalmbe · Sep 12, 2016, 8:04 PM

Yes, there is a typo in the picture. It is not .14 it is .17 (which indeed is private).

tsmalmbe · Sep 12, 2016, 9:11 PM

@johnpoz:

"172.14.100.5"

Where is double nat? 172.14 is a public IP..

NetRange: 172.0.0.0 - 172.15.255.255
CIDR: 172.0.0.0/12
Organization: AT&T Internet Services (SIS-80)

"- 192.168.100.6 IS NOT able to ping 8.8.8.8 nor 172.14.100.5 nor 172.14.100.50"

your saying pfsense can not even ping pfsense wan IP 172.14.100.5?? Well something is wrong then.. And if you can not even ping pfsense wan I wouldn't expect you to ping anything past it either.

192.168.100.6 is the server in the INTERNAL network. pfsense is 192.168.100.5 - the gateway.
192.168.100.6 can ping 192.168.100.5 but nothing beyond it.

johnpoz · Sep 12, 2016, 8:53 PM

192.168.1.5??? Where is that - do you mean 192.168.100.5??

Can you ping pfsense WAN IP??? 172.17.x.x ??? If not then yeah you have something wrong on your box or on pfsense rules..

here this is me from a machine on my lan 192.168.9/24 pinging my pfsense wan interface IP, and then pinging through to my isp gateway.

user@ubuntu:~$ ping 24.13.public
PING 24.13.public (24.13.public) 56(84) bytes of data.
64 bytes from 24.13.public: icmp_seq=1 ttl=64 time=0.393 ms
64 bytes from 24.13.public: icmp_seq=2 ttl=64 time=0.407 ms
64 bytes from 24.13.public: icmp_seq=3 ttl=64 time=0.290 ms

–- 24.13.public ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2000ms
rtt min/avg/max/mdev = 0.290/0.363/0.407/0.054 ms
user@ubuntu:~$ ping 24.13.publicgateway
PING 24.13.publicgateway (24.13.publicgateway) 56(84) bytes of data.
64 bytes from 24.13.publicgateway: icmp_seq=1 ttl=254 time=12.1 ms
64 bytes from 24.13.publicgateway: icmp_seq=2 ttl=254 time=12.3 ms
64 bytes from 24.13.publicgateway: icmp_seq=3 ttl=254 time=10.1 ms

--- 24.13.publicgateway ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 10.187/11.574/12.344/0.986 ms
user@ubuntu:~$

If you can not do this when your firewall rules allows it then yeah you have a problem if you can ping pfsense wan IP, but not pfsense gateway you sure its gateway answers ping?? You sure your natting on pfsense, if not your a downstream router and your upstream router has to know how to get to your network behind pfsense and to go past has to allow and nat that network. If your double natting you should be fine. But first your going to have to be able to get to pfsense wan from its lan.

tsmalmbe · Sep 12, 2016, 9:13 PM

@johnpoz:

But first your going to have to be able to get to pfsense wan from its lan.

Which pretty much is the issue at hand here.

johnpoz · Sep 13, 2016, 12:51 PM

well then you dicked with the default rules? Or you have the wrong interfaces setup? Or you configured somethong not right with masks on your clients? Out of the box pfsense would allow you ping your wan IP that is for sure! You say you can ping its lan IP.. You sure your pinging pfsense lan?

So your making pfsense a downstream router, so you are natting? Which is out of the box the default. If not you would have issues if the upstream router doesn't know where this downstream network is.. But that has nothing to do with simple ping to pfsense own wan IP from lan client.

tsmalmbe · Sep 13, 2016, 5:34 PM

The dicking part - I should be able to see that in the logs, correct?

I have triplediplechecked the interfaces for typos. I also run two other pfSenses and doublecheck (sanitycheck) the setups against those.

perhaps I need to triplediplechek the actual server once more.

It really is the simplest setup, what's what baffles me. The only new thing is the doublenat - but as pfsense is itself able to communicate outbounds… it baffles me even more.

johnpoz · Sep 13, 2016, 6:17 PM

well depends.. Here is a question is your lan the default any any rule or did you modify or add some rules? Did you mess with the outbound nat? 192.168.100 is not the default lan network. So if you had changed your outbound nat to manual or something and didn't put it in right then you would have issues getting anywhere, etc.

tsmalmbe · Sep 13, 2016, 6:24 PM

the lan any-any is the default rule:
IPv4 * INTERNAL net * * * * none Default allow LAN to any rule

there is also the dafult antilockout rule.

NAT - I have not changed. It is automatic.

johnpoz · Sep 13, 2016, 6:57 PM

so then if that is the case you should be able to ping the pfsense wan IP no matter what IP it is.. So look at your nat rules and make sure they show your 192.168.100 network..

And your floating rules tab is empty?

tsmalmbe · Sep 13, 2016, 7:00 PM

Affirmative on both - the network is in the NAT-list and there are no floating rules.

johnpoz · Sep 13, 2016, 7:05 PM

so your client at 192.168.100.6 can ping 192.168.100.5, what is the gateway on 192.168.100.6 box? And you validated that 192.168.100.5 that your pinging and that your using as your gateway is actually the pfsense box via your arp table on the 192.168.100.5 box..

tsmalmbe · Sep 14, 2016, 4:05 AM

The settings regarding this I will have to triple-re-check.

tsmalmbe · Sep 27, 2016, 9:04 PM

For people in or from the future. This proved to be the solution:
https://ask.openstack.org/en/question/26980/problem-using-pfsense-vm-inside-a-tenant/
http://www.honnix.com/technology/software/cloud/network/2015/11/24/pfsense-as-router-in-openstack

The issue is how openstack works, not how pfSense works.

Openstack really is not click-drag-drop-works, it's a lot of overall fiddling and tuning (unless you just want one plain server directly on the internet).

SoulChild · Sep 28, 2016, 2:27 PM

I can't help but find running pfsense in openstack a bit… redundant. Maybe I'm missing something, but what exactly is wrong with using the many firewall layers of openstack that Neutron has built in?

I'm not saying there's no merit in this, but aren't you trying to solve a problem that openstack already has many tools to help you out? And surely, performance will suffer from the many overlay networks used.

tsmalmbe · Sep 28, 2016, 4:17 PM

Good question.

Openstack provides simple "port open or then not" -types of solutions whereas pfSense is a platform for building security.

Perhaps I have misunderstood OpenStack in this sense, but isn't it just iptables with a very very simplified interface on top?

SoulChild · Sep 29, 2016, 2:25 PM

You're right, but it just feels a bit weird implementing a virtual firewall on your openstack to access your virtual IP's

But god knows, openstack is the wild west so far as best practices are concerned, so don't let me tell you otherwise :D