All but 1 Network can reach the internet
This is my problem and I'm drawing a blank:
1. Running esxi 6.5 on Xeon D, 8 Core
2. pfSense 2.3.4 on 1 core with 4GB RAM
3. pfs has 4 networks [ management, secure_computers, wifi, iot (aka Lights) ]
4. All but management have the simular firewalls for test purposes [ png's below ]
Descripion of thing I know/tried:
- Secure_computers and wifi connect to the internet with no issues - both IPs and URLs
- I've also gone through the esx.conf file and have verified all nets are setup the same. All are in "Listen" mode
- IoT with minimal firewall does not connect to the internet.
- Iot Net can ping all internal nets
- Iot Net can not ping any external URLs
- Iot Net can not ping any external IPs
I've confirmed using Packet Capture and States that IOT does not get to the internet on port 53, 80, & 443.
Can someone point me in the right direction?
your first 2 rules on iot are pointless
Since they are allow and then you have an any any rule that allows everything anyway - so unless you wanted those rules to log that traffic there is no point to them.
Rules are evaluated top down, first rule to trigger wins, no other rules are evaluated.
What does your outbound nat look like? Did you mess with it other than leaving it on automatic?
Your saying you can get to your other networks, and pfsense IP on your iot network.. But can not go outbound, but all your other nets can. This points to outbound nat not doing nat for your iot network?
Thanks johnpos.. I did realize that the first rule of IoT were pointless. However, what about the Wifi rule. Do you see any mistakes or issues with it…
I'm headed to the NAT rules now. I did have a VPN setup and my have left it in the incorrect configurations..
I'll be back a soon as I look. THANKS!!!
Without knowing what is in your aliases not sure if they accomplish what your goal is.
Normally I would be more specific in the rules - are you sure you want dns to anyway? Other local segments, anywhere on the internet?
Normally you would be specific on the source net vs just any.. Unless you had downstream routers? There really should never be any non wifi net hitting your wifi net interface for example.
Are you wanting your wifi net to go to your local networks as well in your MyPorts rules? Same goes for your nest IPs rule? They can hit any of your other local networks on whatever you have in the alias.
What uses this wifi network? You seem to have normal clients and your nest devices. I have a thermo and protect from nest, I have them isolated on their own vlan - keep my other wifi devices and guests isolated from the IoT segment(s), etc.
Thanks for the advise, its what I've been looking for.
This is my 3rd vm version of pfs and 2 other with a dedicated 8 core atom 2750 CPU. Everyone of them has been a learning experience ( I didn't know what a vlan was a year ago ). Do you have a preference between VM vs. non-VM systems. I like the possibilities of the additional VMs for home automation, BUT I think the dedicated hardware is "probably" more reliable, easier to maintain and more secure. Any thoughts on this subject?
Mixing home automation and firewalls in one complicated esxi device seems a bit to much. AND, if I get hit by a truck my wife will have no idea how to maintain/use the system and just through it away!!!
I am taking what you suggested to heart and will start from scratch tomorrow and rebuild the VM pfs from the ground up.
This stuff is really fun.
No reason to build from the ground up ;) Just add some vlans and adjust your rules.
I am a fan of the VM, I run mine on an old hp n40l that is really starting to show its age.. Should of been refreshed for sure a year+ ago.. If the thing would just blow up - be easier to get pass the budget committee (wife)..
The need of a new shed blew my esxi host budget ;) hehehe Now how much a nice freaking shed costs.. Could of put together a cluster of esxi hosts ;)
Currently running on beta code and when need to update its so nice to just take a 2 second snapshot if anything goes wrong.. My host has 4 nics, if it had 2 more I wouldn't be complaining. While vlans are fantastic.. When your routing at the edge the more vlans you put on the same physical interface at your router/firewall does hinder intervlan performance since your hairpinning on the same physical interface when your vlans..
My esxi host boots up and everything is running, it auto starts the vms I need for the home network. Pfsense, Plex, etc. So if away and worse case wife can always reboot the esxi box if I can not get in remote to take a look. That being said got a sg-2440 for remote work location (more to come) and sweet little box.. So I can see the draw back to hardware..
But VM sure allows for play in a home lab that is for sure.. While I might break my pfsense back out to physical just for pure performance and such.. I would never get rid of my esxi host..
Yea, I started with a SG-2220 and it was the reason I spent so much time learning vlans. It was/and is a nice and inexpensive little box that is still my backup when I have issues. But, like you I really enjoy the home lab setup. I retired and have all ways been a electronic control freak. People think I'm strange when they ask what I do for fun and I say learning networking, home automation, and hard walking and thinking.
Back to a networking question… I have all of my switches and pfs control setup on one management vlan because I like to be able to have access from my two Macs in the house ( one of the aliases you asked about. You really didn't see my actual setup just the lab version for testing. ). I've spent to a great deal of time trying to secure the controls from all internal nets and the internet. Is that a bad practice? And, is it safe to tag it and put on my secured Mac network. The SG-2220 was all vlan with one trunk, but you know that. I had heard and understand that a vlan setup incorrectly is extremely insecure.
Also, I've seperated my IoT ( lights and Nest ) from both of the the above mentioned nets. I access it using wifi which NATs to the secure nets and I also deny the IoT back to the Secure. Is this a bad practice. I do a lot of test to make sure it works, BUT I know me and I can make stupid mistakes thinking I understand something [ especially women ].
" I access it using wifi which NATs to the secure nets"
What would be the point of the nats? Vlans if done correctly are very secure, atleast if the switching doesn't have security issues - say for example the tl-sg108e that doesn't allow you to remove vlan 1 ;) So the managment interface is going to be available on any port on that switch. But for home/lab this really shouldn't be a real concern. Now in an enterprise or say dod setup that would be a no go ;)
A management vlan is very common practice for sure. You just want to secure what ports and or devices actually have access to the management vlan is all.
As to be strange learning and playing with network home automation - people that don't think is fun are the strange ones ;) I am lucky in the sense that what I do for a living is also my hobby and passion.. I love nothing more than playing with different tech and hardware.. In another sense I spend way to much time with this sort of stuff… hehe.. I do it at work, and then come home and do the same sort of stuff.. Many of my co workers that setup routing and switching all day at work - run 1 layer 2 at home with all their devices on the same segment. I personally think they are freaking nuts for doing this. Do they not understand the complete lack of security these iot devices ship with? Damn straight I am going to isolate them and log what they are doing.. I log all outbound traffic from my iot devices - take a look at it now and then to make sure nothing funny is going on - this is fun to me! ;)
The reason I said NAT is that I started this endeavor with a Cisco SG300MMP-8 and an Ubiquiti Edgerouter X as an inexpensive way to learn ACLs and router firewall rules. I know the SG300 is level 3 but I didn't really understand the difference between L2 & L3 at the time. I do now. I also wanted POE for Wifi access points.
My Fundamental Networking Question:
When I make a connection between IoT and Wifi I view it as a stateful connection using the pfs router which again i assume requires NAT at the router level of pfs between the two nets ( established and related ). I put a rule on the Iot input denying anything back to the wifi net. I allow IoT to the internet and nothing else, and I monitor the logs constantly. Is this remotely correct???
Everyone I talk to in the real world does not understand the danger of Crap IoT devices. To very honest they are the reason I started the networking phase of my life to protect my prvacy and money.
Yes sir, you had better really enjoy what you do all day. I was fortunate early in my career. My boss in 1984 asked me to replace all the relay logic in the mfg facility with PLCs which included several multi-ton hydraulic presses, many conveyors, the design of several pick and place robots. He new I enjoyed it, but he didn't know I had only worked with 1 PLC up to that point. It was absolutely GREAT!! There was on week I only went home to shower and eat, them back to work for 7 days…. You had better really love what you do!! It is absolutely necessary that you have an outstanding Maintenance Dept to get all of that accomplished. The most under appreciated people on earth next to IT. People only talk to you when thesystem is down and the world is coming to an end :).
While the sg300 can do L3, I have one - I only use it as L2. In a small network its fairly uncommon to need a downstream L3.. Part of the reason I went with the sg300 was the ability to do L3 if need be - never know what you might want to play with or setup in your home/lab - at the price point great little switch for sure.
I think your confusing NAT with routing? There should be zero reason you would ever need to nat inside a local network.
So while pfsense will for sure route between any network it has connected to it, or it has a route and gateway for. There would be no reason to nat between rfc1918 network connected to pfsense. The only time you would really need to nat would be when you need to change your connection from rfc1918 to public space and your limited number of public IPs.. This is where the beauty of ipv6 comes in - no more nat will be required! You can use public routeable addresses behind your router/firewall since there is pretty much unlimited space. I have a /48 from HE for example - I can not see ever needing 64K different networks in my home/lan ;)
Maybe you think your natting when your not - out of the box pfsense will only nat networks to another interface IP when that interface has a gateway set on it - and pfsense now considers that a WAN connection. Creating a gateway and then routes that use that gateway will not kick off the automatic outbound nat does. So you could have many many networks behind pfsense. I currently have like 8 vlans/networks (not all of them are vlans) behind pfsense in my local home network. The only time nat is done is whenever any of these networks go out to the internet via ipv4 where I only have the 1 ipv4 address. So it needs to do NAPT (network address port translation).. If I have ipv6 enabled on that segment and using ipv6 then no nat happens even going out to the internet.
If you want or need then sure you can nat between network segments on pfsense on the outbound tab.. But out of the box in automatic mode it will not be natting between local segments/vlans you connect to pfsense. Every now and then you might want/need to source nat something - this can come in handy if your remote in via say a vpn and there are devices that do not support connections from other network. Many a home soho AP/wifi router native firmware has no way to set a gateway on its lan interface for example. Many iot devices design to be on same L2 as everything else might not have ability to set a gateway - in that case you would need to source nat to get to it from another network/vlan.
And I am with you - the old saying if you love what you do, you will never work a day in your life is very very true..
Sorry, you must forgive my lack of IT terminology. When I said NAT I was stuck in the Edgerouter (ER) terminology. I used it to isolate networks using just a router with 4 nets, not the SG300. SG-2220 <-> ER <-> SG300 <-> separate networks
I wasn’t thinking of NAT and NAT firewalls on pfs. The reason for my confusion is that on the (ER) you could see in the ARP tables that there was translation of IPs between nets. So I thought when I connected to the IoT net from the Wifi net on pfs it created a stateful connection to the IoT. But because of my Block All rule to all networks from the IoT it could only respond to communications from the requesting net. Kind of a traditional router thing I thought. And when I set it up that way it seemed to work exactly as expected. I tried to connect ( using no firewall rules ) from IoT to Wifi using a laptop I temporarily inserted into the IoT. It appeared to be truly isolated. The lights responded correctly and the IoT net could not Ping or address any net except the WAN.
Did all of that make sense?? I had an expectation of an outcome and it appeared to work that way, so I jumped to the conclusion I was correct. Hopefully that last sentence made sense.
Are you saying I can use ipv6 for my internal net?
You are giving me one hell of a lot to think about :). Thanks, that is what I’m looking for….
No problem - happy to help describe any terms your unfamiliar with. I guess I can see how you could think of it how a nat router works without port forward, etc.
Yes you need the rules to allow the traffic, but its not actually doing any sort of nat or napt to allow the connection. It just creates a state table entry if allowed.
Sure you can use ipv6 on your local segments. I don't have it on all of them - just the ones I want to play with it in on. I think off the top like 4 or 5 of my segments. I don't have it enabled on my iot segment for example. But I do have it on my lan, and my wlan and my guest wlan and my dmz, etc.
If you need any help with ipv6 just ask - also check out the hurricane electric certification for ipv6. Get sage and you get a free tshirt ;) Will help you learn about ipv6.. Which should be fun for you to play with ;) If you have not yet started to play with it - its way different than ipv4, not just longer addresses.
I'm running ESXi 6.5 free version. I seriously looked at the vSphere 3 server package but really could not justify $660 price. I’m pretty good at changing vSwitch, port group names, etc. by editing the esx.conf file. If you use the free version how do you back it up? 3rd party software or another way.
I run esxi 6.5 free yes - why would you be editing esx.conf?? Why would you not just use the web gui interface that is built into esxi
Or for that matter - the fat client still works.. Even though it is suppose to be discontinued.. Still works just not latest and greatest features can be edited, etc.
Backup what? Yes I have some vm images saved off, I have ova that I can deploy a new linux vm in like 1 minute with, etc. Takes all of a few minutes to rebuilt esxi if it crashed, then load up my vms via ova, etc. What are you wanting to backup?
You can't change the names of the switches, port groups, etc. from the gui. It's easy it do that in /etc/vmware/esx.conf with vi.
I also have philosophy "If it ain't broke then I'll brake it by screwing with it". Sort of a part of the learning process for me. Snapshot work great but sometimes i get carried away and make changes to esx.conf file and screw it up. A backup would help that situation.
There is a few 3rd part programs that are available. I wondered if you had used any. Vertical Backup is about $10 but sometimes cheap isn't better. https://verticalbackup.com/. It also seams to controled by cron so scheduling looks easy.
And I do backup all VMs manually as well as my esx.conf file. My life revolves around a good internet connection…. phone, cable is cut so all internet, Facetime [ not Facebook ! ], several pearl scrips that gather data for me all day…. disgusting isn't
One other thing, The web gui is why I dont have the ability to change somethings. After looking at esx and vsphere over the past few months I "think" I discovered that the push for vmware was to move everything to web interfaces. Thats fine if your running vcenter but not so much for free esxi. vCenter allows backups and rightly so, its expensive if you dont really need the high availablity, etc. If thats incorrect it would make me happy, let me know.
One final rant, I spent the better part of my life programming with/using Windows. I hate Windows. I used it when my job required it, but now there is no job and i can use what I want. I switched to mac, freebsd, unix and linux about 20 years ago. Sorry, I get carried away.
"You can't change the names of the switches, port groups, etc. from the gui. "
You create them with the names you want - see my vmonlyswitch. You can for sure change the names of the portgroups with the fat client.
Yeah the push is to go web, they try to push vcenter.
"You can for sure change the names of the portgroups with the fat client."
What is the FAT Client?
The 2nd pic..
Download URLs for VMware vSphere Client (2089791)
The vsphere client.. version 6 update 3 was the last to come out..
I really appreciate the advice you gave me last week. So, I tried over the past few days to incorporate what I think you were telling me.
I have (4) networks: [ I think that I have configured then as state below ]
Equip: All switches, AP, Routers attached. Allowed to reach all other nets. 192.168.1.0
Secure: Wide open and allowed to reach all other nets [ will be modified as needed ] 192.168.40.0
Wifi: Wireless devices such as iPad, iPhone can reach the internet and iPad is allowed to reach the Wifi Address. 192.168.20.0
Lights and Nest are controlled by this net.
No other nets are allowed to reach it in any way and it cannot reach the firewall except for the iPad
IoT: All lights are controlled through this. 192.168.30.0
It cannot reach any other net or firewall and does not respond to pings.
It does somehow reach the iPad and iPhone Hue.app that controls the lights??? This is what we talked about last week. I’m still confused about how.
After testing it seems to perform as expected.
What do you think. Please be critical if I’ve made mistakes ::) or was to sloppy :)….
Also: I already had the FAT ESXi Window interface - found it when I looked bake at my Win 7 VM…