Newbie questions
-
@johnpoz said in Newbie questions:
@ldl said in Newbie questions:
I'm basically trying to get my head around how proxmox works
Would your question be better suited on their forums?
I believe both would be suited, seeing as pfsense handles the DHCP side and Proxmox handling the network itself, however it doesn't hurt asking in both areas, as there may be something that I may have missed out on pfsense.
@stephenw10 said in Newbie questions:
It sounds like you need to add some VLANs to your config and extend those through the switch to the two PXE hosts.
I forgot to mention that part mate, I've got vlan tags setup both on pfsense and my Cisco router and with the IPs assigned.
Next step I'll be looking at I guess will be bridging.Edit: Also tracert comes back with dead ends when trying from 172.16.1.x trace to 172.16.0.1
Cheers all.
-
@ldl said in Newbie questions:
Also tracert comes back with dead ends when trying from 172.16.1.x trace to 172.16.0.1
That should be routed through pfSense? It will route that traffic as long as firewall rules exist to pass it. So if that's not happening there's either some issue with the VLAN config somewhere. Or potentially something is still using an old subnet mask if you just changed it to /25.
I would run some pings and check the pfSense state table in Diag > States to make sure that traffic is arriving and being passed in and out.
-
@ldl
Where do you have pfsense located in your topology??My network topology in simple terms goes as; Modem -> Router (which can be changed to AP) -> Unmanaged netgear switch (unmanaged) -> Cisco switch -> servers
And how is the router connected (from modem to WAN port?)
You say the router "can be changed to AP" which I interpret as it is currently not? But no DHCP active?If it's connected on the WAN port, you still rely on it's firewall, which adds some complexity. Do you have enough ports on the unmanaged and Cisco switches to completely remove the router from the equation?
And what do you mean with "my router pointing to pfsense for DHCP"? Pfsense getting DHCP requests, and handing out IP's means it would typically set itself as the gateway unless you have changed that. And if it does, and it is not connected on the WAN side, you don't get far...
WRT Proxmox, neither it nor your VM's need to recognize VLAN.
Whatever subnet the Cisco switch port belongs to, will be what Proxmox and the servers pick up.You CAN however selectively set the VLAN for each VM. So if you have a trunk port connected to Proxmox, you can place individual VM's on different VLAN's by means of VLAN tagging.
-
@stephenw10 said in Newbie questions:
@ldl said in Newbie questions:
Also tracert comes back with dead ends when trying from 172.16.1.x trace to 172.16.0.1
That should be routed through pfSense? It will route that traffic as long as firewall rules exist to pass it. So if that's not happening there's either some issue with the VLAN config somewhere. Or potentially something is still using an old subnet mask if you just changed it to /25.
I would run some pings and check the pfSense state table in Diag > States to make sure that traffic is arriving and being passed in and out.
At the moment, I'm testing it via my PC, which is routed through pfSense, sorry I wasn't clear earlier.
It handles traffic fine on 172.16.0.x range, just not 172.16.1.x range.
I tried to run a trace up root (172.16.0.x to 172.16.1.1), even pinging returns no results.
I do recall vaguely from the course I was on that I need to set up vlans, which I have done (which I only had a year access to, to cram all the knowledge of what I didn't know into my brain)@Gblenn said in Newbie questions:
@ldl
Where do you have pfsense located in your topology??My network topology in simple terms goes as; Modem -> Router (which can be changed to AP) -> Unmanaged netgear switch (unmanaged) -> Cisco switch -> servers
And how is the router connected (from modem to WAN port?)
You say the router "can be changed to AP" which I interpret as it is currently not? But no DHCP active?If it's connected on the WAN port, you still rely on it's firewall, which adds some complexity. Do you have enough ports on the unmanaged and Cisco switches to completely remove the router from the equation?
And what do you mean with "my router pointing to pfsense for DHCP"? Pfsense getting DHCP requests, and handing out IP's means it would typically set itself as the gateway unless you have changed that. And if it does, and it is not connected on the WAN side, you don't get far...
WRT Proxmox, neither it nor your VM's need to recognize VLAN.
Whatever subnet the Cisco switch port belongs to, will be what Proxmox and the servers pick up.You CAN however selectively set the VLAN for each VM. So if you have a trunk port connected to Proxmox, you can place individual VM's on different VLAN's by means of VLAN tagging.
And how is the router connected (from modem to WAN port?)**
You say the router "can be changed to AP" which I interpret as it is currently not? But no DHCP active?**Its directly connected over ethernet, I mention the AP as I was reading somewhere online about people using their router as an AP to then hook it up to their server with their setup, but to state the obvious, that's obviously just acting as an access.
If it's connected on the WAN port, you still rely on it's firewall, which adds some complexity. Do you have enough ports on the unmanaged and Cisco switches to completely remove the router from the equation?
I was looking through my firewalls as well, I'm familiar when it comes to opening ports, I have a little bit of experience when it comes to dealing with NICs or even VM NICs
I've currently got both my servers trunking, my Cisco switch is a managed L3 switch whilst my Netgear is a L2 switch if I recall, I originally got that just for 1 server I had at the time in case that was in question.
That said, I have enough ports to take the router out of the equation, yeah, it'll be connected directly to my modem however.And what do you mean with "my router pointing to pfsense for DHCP"? Pfsense getting DHCP requests, and handing out IP's means it would typically set itself as the gateway unless you have changed that. And if it does, and it is not connected on the WAN side, you don't get far...
On my router settings, I have the option to change the default gateway, though I think I'll change it back as I'll see about using vlan tags as you suggest. (screenshot below)
As mentioned above though to Stephenw10, I forgot to say that connection to the internet is possible via 172.16.0.x range, just not 172.16.1.x range, which yeah, it could (most likely) be the firewall, I've had to ponder about it, as mentioned above I just have to familiarize myself with firewall for the interface settings, my router as well has a firewall, last night when I posted this thread my mind was focused mainly on the vlans, so I'll see about the firewalls next.
-
Ah so you have a router here that isn't pfSense?
A diagram might help here.
-
@stephenw10 As far as I am aware, nah mate it's most likely not compatible with it (though I thought that was optional), I have an Asus router RT-AC3200.
Diagram is as;
Modem -> Asus router -> Unmanaged Netgear switch -> Managed Cisco Switch -> Servers
-
Hmm, so where are you running pfSense there? As a VM?
-
@stephenw10
Yeah mate via Proxmox, I've got other VMs as well running through two physical servers. -
Ah OK. Then I guess this isn't really a pfSense issue?
-
@ldl said in Newbie questions:
@stephenw10 As far as I am aware, nah mate it's most likely not compatible with it (though I thought that was optional), I have an Asus router RT-AC3200.
Diagram is as;
Modem -> Asus router -> Unmanaged Netgear switch -> Managed Cisco Switch -> Servers
Ok but it is still not clear where you have pfsense connected, and which ports (WAN, LAN, LAN2) on pfsense are connected where??
Also, in your first post you said the modem had IP 192.168.0.X/24 range.
To me that means it is not just a "modem"... it looks more like it is the ISP router which is meant to hand out IP's on your LAN. And as a consequence your Asus router has a WAN address of 192.168.0.something?? So already here you are double NATed, and adding pfsense makes it triple NAT...If this is true, I would think of ways to eliminate one or the other. I suppose you bought the Asus for a reason, so removing the ISP device would be my choice, and eventually replacing the router with pfsense.
If your ISP has "locked" your external IP to the MAC of their modem, you can spoof that on the Asus router, as well as on pfsense if you want.What is clear however is that traffic is going through the router. So anything on LAN 172..16.0.X will have internet.
Similarly you have to set up pfsense so anything "controlled" by it goes through it... meaning if you want to play around with it as a homelab thing, you connect the pfsense WAN to a LAN port on your Asus router (or the unmanaged switch in this case).
Then you move the Cisco switch to the LAN port of pfsense. And your PC for managing pfsense plus all the servers have to be connected to the Cisco switch. Now you can play around with VLANs etc on pfsense and the Cisco switch, and Proxmox if you like. Don't mess with VLAN on the pfsense VM though...
The Asus router should in this case have DHCP turned on again, and just leave it as your standard router.
Your topology will now look like:
Modem -> Asus router -> Unmanaged Netgear switch -> (WAN) pfsense (LAN) -> Managed Cisco Switch -> Servers and your PC
-
Thanks mate, yeah that makes sense.
To answer your question though, pfSense is connected at the server end as it's on a VM, which is through LAN.
And yeah, I've assigned my Asus router a static IP on the 192.168.0.x range.I was also speaking to my friend at work a few months back regarding double NATed, and they also explained that to me, and that it can cause complications down the line, but for the time I've had it set up as such and had no issues, and that's been since I got on the internet a good 20 years ago, but I guess I should break that habit, switching over to AP would help, I guess.
I cannot eliminate the ISP router as its fiber, unless that's where SFP comes in?On the ISP router, all the ports are in use, though if I get this network set up correctly, then I'll be redirecting that through this new setup that I'm trying to achieve.
Thanks for the tips mate, I do appreciate it, for me this is a whole new field for me to explore, and it does seem interesting to go through.
-
Yup double NAT can cause problems but it will work fine for almost everything. It certainly won't cause a complete connectivity failure as long as there are no conflicting subnets.
-
@ldl said in Newbie questions:
Thanks mate, yeah that makes sense.
To answer your question though, pfSense is connected at the server end as it's on a VM, which is through LAN.Ok, but if you want to start using pfsense to route traffic, even if it's just for learning purposes, it needs both a WAN port and a LAN port connected. If your Proxmox machine only has one physical port, you need to start working with VLAN's to solve this (Proxmox VLAN as I showed above, in conjunction with your Cisco switch which needs to be set up appropriately).
Think about how the other routers are connected, WAN <> internal firewall <> LAN. The same applies to pfsense and you want to connect both sides for it to work. Things happening on the LAN side do not involve the firewall, it's mainly handled by the switches.
As I understand how you have it set up, it can hand out IP's and the devices think that pfsense is the gateway. But the traffic has no where to go, since WAN is not connected anywhere...?And yeah, I've assigned my Asus router a static IP on the 192.168.0.x range.
I was also speaking to my friend at work a few months back regarding double NATed, and they also explained that to me, and that it can cause complications down the line, but for the time I've had it set up as such and had no issues, and that's been since I got on the internet a good 20 years ago, but I guess I should break that habit, switching over to AP would help, I guess.
Double NAT is not really a problem for most normal internet use. If you were doing gaming for example, you may end up having trouble playing with friends, since you may not get Open or at least Moderate NAT in the game. But if you plan to go further with your servers and perhaps want to access things from the outside, you need to fix the double NAT situation somehow. The same applies if you are looking at many smart home solutions as well.
I cannot eliminate the ISP router as its fiber, unless that's where SFP comes in?
Yes the fiber comes in with the SFP. And in many cases you have a split setup with a media converter that takes the SFP and converts it into Ethernet (RJ45). From the media converter the ethernet cable goes into the WAN port on your router.
This router can then be any router, not just the one your ISP supplied, meaning you can replace it with your own.
If you want to try this, it's likely a good idea to clone the WAN MAC from the ISP router and enter it in your Asus router like this (type in the MAC that you find in the UI of the ISP router (and/or printed on the back):
There are newer models where the router has the SFP integrated, in which case you can't eliminate it... but you may be able to set it to Bridge Mode instead.
On the ISP router, all the ports are in use, though if I get this network set up correctly, then I'll be redirecting that through this new setup that I'm trying to achieve.
So are you saying that your main home network is actually on the 192.168.0.1/24 subnet? Do you have switches connected there as well? Is your Asus router what you use for wifi?
If you need the ports on both the ISP router and your Asus, you can change one or the other into a "switch". For example if you turn off DHCP on the ISP router, and move the WAN cable over to your Asus router (after cloining the MAC). Then you can still make use of the LAN ports on the ISP unit.Topology in this scenario:
Fiber to Ethernet (media converter) > Asus router > ISP router (using only LAN ports) > Netgear switch > WAN pfsense LAN > Cisco switch > serversIt will be a bit tricky to get the last part working unless you have more than one physical port on the Proxmox machine where you run pfsense. Preferably you should have at least three ports, of which two are dedicated to pfsense (WAN and LAN).
Perhaps you should draw yourself a diagram for the setup so you fully understand what you are doing. Especially if you have to use VLANs to make it work.
I don't know Cisco switches but how I'm thinking you could do this is the following :
Set port 1 to VLAN ID 10 (not entirely sure how this will work towards the netgear switch?)
Set port 2 to VLAN ID 10 and 1
Leave all other ports at ID 1 (default).The idea is to only allow traffic with VLAN tag 10 to pass between ports 1 and 2.
The cable coming from your Netgear switch will go into port 1 and your Proxmox server with the pfsense VM will connect to port 2.
You need to go into the Proxmox UI and make sure you have two ports for the pfsense VM, both using the same bridge port (vmbr0). One of these will be the WAN port and for this one you have to set the VLAN tag to 10. The other you leave at default.
This way your WAN port on pfsense will be communicating up towards the router via port 1 on the Cisco switch. And the LAN port will use default VLAN covering ports 2-N.
-
Apologises in the delay.
That's most likely where I'm going wrong, as I've been leaving WAN blank, but yeah, to answer your question, I do have 4 physical ports, on both servers, all connected up to the switch.
@ double nat, ah okay well I learn something new every day, but yeah, I will be eliminating the double nat.
The ISP runs on 192.168.0.x range, no other switches are connected to there, just on the 172.16.0.x range that I have two on (unmanaged Netgear + Managed Cisco), I also have another switch, that isn't in use.
But yeah, I've been trying to find some softwares I can use to draw up a diagram, sure I could just use Paint or something, but I'd want some sort of software that I can keep my IPs in order, though that's another subject.
Again, thanks for the helpful information.
-
@ldl If you have as many as 4 ports on each server, it will of be much simpler and no need to fiddle with VLAN's.
Still, consider removing the ISP router and connecting the Asus directly, as a first step. Then when you feel confident using pfsense, you replace the Asus and move that over to the LAN side of pfsense (only using LAN ports and disabling DHCP).
In your current setup, the Proxmox machine with pfsense VM should have one port connected to the Netgear switch, which will be your WAN for pfsense. All other ports on that Proxmox as well as the other machine, should be conncted to the Cisco switch which will palce all VM's entirely in the pfsense "domain".
So the topology you are looking at for starters is:
Fiber to Ethernet (media converter) > Asus router > ISP router (using only LAN ports) > Netgear switch > WAN pfsense LAN > Cisco switch > all other server ports: -
@Gblenn Again, apologises in the delay.
Okay thanks for the information, I've also been looking at alternative ISP purely on the cost and higher up/download speeds, one in particular says they would use a direct RJ45 connection, but I personally want to keep the fiber lead.
Out of curiosity, would it be beneficial in my requirements to use the upstream gateway?
Cheers.
-
@ldl said in Newbie questions:
@Gblenn Again, apologises in the delay.
Okay thanks for the information, I've also been looking at alternative ISP purely on the cost and higher up/download speeds, one in particular says they would use a direct RJ45 connection, but I personally want to keep the fiber lead.
Out of curiosity, would it be beneficial in my requirements to use the upstream gateway?
Cheers.
When you say, "use the upstream gateway", do you mean the ISP provided router?
I have never found any benefit in using the ISP's equipment. Although my current ISP have actually provided a quite powerful Zyxel device capable of 10Gig on the LAN side, and wifi 6. But it still ended up in it's box in storage...
Instead I'm using TPLink Omada gear, for both switching and wifi and it's so much simpler having just one interface to work with. And then I have pfsense as my gateway/firewall.It's mainly the functionality that will be lacking when using the ISP equipment, or even the Asus router you have. Which is why you would want to move towards having pfsense as your "entrypoint" and bring your fiber directly into it (perhaps via a media converter). A 1Gbit model will start at around 20USD and a 2.5GBit perhaps 2 - 3 times that.
When I said "for starters", I meant that you run with the topology you have, until you feel you want to use pfsense the way it's intended. Your Asus router can then be used as your wifi AP, perhaps together with your ISP router in some other location in the home to add wifi coverage.Since you already have fiber to your home, perhaps the ISP you talked to mean that they will provide a media converter which is what my current ISP did when I had 1Gbit. I got one of these super devices: https://www.amazon.com/s?k=media+converter+1gb&crid=3I07NTVFKVYZU&sprefix=media+converter+1g%2Caps%2C157&ref=nb_sb_ss_ts-doa-p_1_18
And there is no harm in using that of course. But perhaps you want to keep building and experimenting with your pfsense machine and then you can always put an SFP/SFP+ card in it. Which then gives you the possibility to plug the fiber module directly into the WAN port for pfsense.
-
Apologises again in the delay.
I was referring to the pfSense's upstream gateway, as I'm currently experimenting quite a lot with this, trying to get to what I need to achieve.
I will at some point be changing out my Asus router for something more suitable to my needs, as its outdated as well.
According to the response I got on their forums (yeah, they have a forum, I've never known one to have one xD), it'll be connected via ONT, and then terminated in an RJ45.
I've not come across this before, so if I do go for this ISP, then maybe it'll be better, though they claim to have 2x faster speed than my current ISP (An ISP that brags to be the best in the UK)
Thanks for the feedback again as well.
-
@ldl Ok, so it's like I mentioned, this other ISP will terminate with what I referred to as a media converter, the ONT. It is then entirely up to you what you decide to use as a router/firewall.
And you have already built two Proxmox servers, with multi NIC's, and you have pfsense up and running as a VM. Given this, I'd say you are ready to change out your Asus router already, and replace it with pfsense.
To simplify things I'd make sure to clone the MAC address of the ISP router to the WAN interface of pfsese before connecting to the ONT. If you change ISP, you just change the MAC to what the new router you get from them has. It's written on the back of the device, and you can likely find it in the UI. Or you can connect it's WAN port your pfsense LAN and find it in the list of DHCP Leases in pfsense, where you can easily copy paste it.
For Proxmox, you should look into IOMMU (pass thru), to have the necessary NICs completely handed over to pfsense. Availability of this functionality depends on the generation HW you have (CPU/motherboard). But it will give the best performance and control from a pfsense perspective.
With pfsense and your cisco switch you have all the possibilities to continue playing around with VLAN's and all sorts of fun stuff. If your Asus router supports VLAN, you can start creating multiple isolated wifi networks, for guests, IoT stuff etc. But if not, it will still be able to serve as a wifi AP, as long as you remember to use LAN ports only, set a different IP compared to the pfsense UI, and turn off DHCP.
-
@Gblenn said in Newbie questions:
@ldl Ok, so it's like I mentioned, this other ISP will terminate with what I referred to as a media converter, the ONT. It is then entirely up to you what you decide to use as a router/firewall.
And you have already built two Proxmox servers, with multi NIC's, and you have pfsense up and running as a VM. Given this, I'd say you are ready to change out your Asus router already, and replace it with pfsense.
To simplify things I'd make sure to clone the MAC address of the ISP router to the WAN interface of pfsese before connecting to the ONT. If you change ISP, you just change the MAC to what the new router you get from them has. It's written on the back of the device, and you can likely find it in the UI. Or you can connect it's WAN port your pfsense LAN and find it in the list of DHCP Leases in pfsense, where you can easily copy paste it.
For Proxmox, you should look into IOMMU (pass thru), to have the necessary NICs completely handed over to pfsense. Availability of this functionality depends on the generation HW you have (CPU/motherboard). But it will give the best performance and control from a pfsense perspective.
With pfsense and your cisco switch you have all the possibilities to continue playing around with VLAN's and all sorts of fun stuff. If your Asus router supports VLAN, you can start creating multiple isolated wifi networks, for guests, IoT stuff etc. But if not, it will still be able to serve as a wifi AP, as long as you remember to use LAN ports only, set a different IP compared to the pfsense UI, and turn off DHCP.
I forgot to mention, that this ISP I will be switching to near the end of next month (as I have to give 30 days notice to my current ISP), is that they give me the option at a cost per month to use one of their routers, or I can use my own, so there will be no MAC issues, which is good.
I will be upgrading my router sometime next month as well, because currently my WAN port on my router only has a max output of 1Gb, which tbf, at the time, for me is enough as I was only able to get up to 1Gb download and 100Mb Upload from my current ISP, but this new ISP offers 2.5Gb for both up/download, as well as offering IPv6, which is something else I want to get familiar with.
Right now I'm just looking at a 2.5Gb WAN for a router and 1Gb for the LANs, I've found two that I will decide on the next month.
It's also obviously good to update my current router, as it no longer has firmware updates available.On to the subject though of pfSense, I ran into this weird issue two days ago, where for some reason, when I reset my servers, pfSense was no longer able to communicate to the network, it was able to ping out to the internet (8.8.8.8), just not on the intranet/network, I resolved that by setting the interfaces, though I'm having to use 23 mask on my router and wanting to use 25 mask on pfSense, if I recall though, they all need to be on the same mask, as the course I'm currently on leading towards Cyber Security, covered CompTIA, and the networking side of things, yeah it is indeed something fun to get into, I'm learning quite a lot, but on this course, I could have swear they said I need to set the mask to the same one across the board, unless they just meant on those servers trying to communicate with each other, but how I see it, if this setup is correct, obviously for the router itself, when I set it to 23 mask, it's able to talk to the IP range that my servers are on, whereas, I'm guessing if it's on the 25 mask, that my servers are on, then it'll be limited to the range it's currently set on, do correct me if I'm wrong on this.
Sorry for all this hassle, I also set up routes as well from my Asus router to allow communications between the devices, as well, the devices on the 172.16.1.x range don't appear on the routers connected devices which is on the 172.16.0.x range, I'm guessing I need to mess around with SNMP or something for this?
