Newbie questions
-
Hi all,
So I'm trying to set something up on my network.
Currently I have two servers active that are connected to my switch, which is a Cisco Switch.They are both using Proxmox, and currently I'm trying to create two segments of networks, I know how to go about segmenting the network into split sections.
Currently I have pfsense set up as a /25 mask so one of my servers are meant to be on one segment and the other on the other half.
I'm basically trying to get my head around how proxmox works with pfsense in terms of DHCP when it comes to working on a different range as well as no connection possible local and over the internet on the range below for my server.
My network topology in simple terms goes as; Modem -> Router (which can be changed to AP) -> Unmanaged netgear switch (unmanaged) -> Cisco switch -> servers
IP range setup:
192.168.0.x /24 - modem
172.16.0.x /24 - router (DHCP disabled) - PC is connected to the router
172.16.1.x /25 split for my network
I've got my router pointing to pfsense for the DHCP, and pfsense picks up incoming requests from my servers, but currently, I'm getting no internet and also on the network, and on the server range and I cannot connect to the IP range unless I change my IP setting manually to the range.Let me know if you require further details, thanks.
And any help is appreciated, again, thanks!Edit: Also, I've routed the traffic on my router to allow traffic from 172.16.1.1 range to 172.16.0.1.
-
@ldl said in Newbie questions:
I'm basically trying to get my head around how proxmox works
Would your question be better suited on their forums?
-
It sounds like you need to add some VLANs to your config and extend those through the switch to the two PXE hosts.
-
@johnpoz said in Newbie questions:
@ldl said in Newbie questions:
I'm basically trying to get my head around how proxmox works
Would your question be better suited on their forums?
I believe both would be suited, seeing as pfsense handles the DHCP side and Proxmox handling the network itself, however it doesn't hurt asking in both areas, as there may be something that I may have missed out on pfsense.
@stephenw10 said in Newbie questions:
It sounds like you need to add some VLANs to your config and extend those through the switch to the two PXE hosts.
I forgot to mention that part mate, I've got vlan tags setup both on pfsense and my Cisco router and with the IPs assigned.
Next step I'll be looking at I guess will be bridging.Edit: Also tracert comes back with dead ends when trying from 172.16.1.x trace to 172.16.0.1
Cheers all.
-
@ldl said in Newbie questions:
Also tracert comes back with dead ends when trying from 172.16.1.x trace to 172.16.0.1
That should be routed through pfSense? It will route that traffic as long as firewall rules exist to pass it. So if that's not happening there's either some issue with the VLAN config somewhere. Or potentially something is still using an old subnet mask if you just changed it to /25.
I would run some pings and check the pfSense state table in Diag > States to make sure that traffic is arriving and being passed in and out.
-
@ldl
Where do you have pfsense located in your topology??My network topology in simple terms goes as; Modem -> Router (which can be changed to AP) -> Unmanaged netgear switch (unmanaged) -> Cisco switch -> servers
And how is the router connected (from modem to WAN port?)
You say the router "can be changed to AP" which I interpret as it is currently not? But no DHCP active?If it's connected on the WAN port, you still rely on it's firewall, which adds some complexity. Do you have enough ports on the unmanaged and Cisco switches to completely remove the router from the equation?
And what do you mean with "my router pointing to pfsense for DHCP"? Pfsense getting DHCP requests, and handing out IP's means it would typically set itself as the gateway unless you have changed that. And if it does, and it is not connected on the WAN side, you don't get far...
WRT Proxmox, neither it nor your VM's need to recognize VLAN.
Whatever subnet the Cisco switch port belongs to, will be what Proxmox and the servers pick up.You CAN however selectively set the VLAN for each VM. So if you have a trunk port connected to Proxmox, you can place individual VM's on different VLAN's by means of VLAN tagging.
-
@stephenw10 said in Newbie questions:
@ldl said in Newbie questions:
Also tracert comes back with dead ends when trying from 172.16.1.x trace to 172.16.0.1
That should be routed through pfSense? It will route that traffic as long as firewall rules exist to pass it. So if that's not happening there's either some issue with the VLAN config somewhere. Or potentially something is still using an old subnet mask if you just changed it to /25.
I would run some pings and check the pfSense state table in Diag > States to make sure that traffic is arriving and being passed in and out.
At the moment, I'm testing it via my PC, which is routed through pfSense, sorry I wasn't clear earlier.
It handles traffic fine on 172.16.0.x range, just not 172.16.1.x range.
I tried to run a trace up root (172.16.0.x to 172.16.1.1), even pinging returns no results.
I do recall vaguely from the course I was on that I need to set up vlans, which I have done (which I only had a year access to, to cram all the knowledge of what I didn't know into my brain)@Gblenn said in Newbie questions:
@ldl
Where do you have pfsense located in your topology??My network topology in simple terms goes as; Modem -> Router (which can be changed to AP) -> Unmanaged netgear switch (unmanaged) -> Cisco switch -> servers
And how is the router connected (from modem to WAN port?)
You say the router "can be changed to AP" which I interpret as it is currently not? But no DHCP active?If it's connected on the WAN port, you still rely on it's firewall, which adds some complexity. Do you have enough ports on the unmanaged and Cisco switches to completely remove the router from the equation?
And what do you mean with "my router pointing to pfsense for DHCP"? Pfsense getting DHCP requests, and handing out IP's means it would typically set itself as the gateway unless you have changed that. And if it does, and it is not connected on the WAN side, you don't get far...
WRT Proxmox, neither it nor your VM's need to recognize VLAN.
Whatever subnet the Cisco switch port belongs to, will be what Proxmox and the servers pick up.You CAN however selectively set the VLAN for each VM. So if you have a trunk port connected to Proxmox, you can place individual VM's on different VLAN's by means of VLAN tagging.
And how is the router connected (from modem to WAN port?)**
You say the router "can be changed to AP" which I interpret as it is currently not? But no DHCP active?**Its directly connected over ethernet, I mention the AP as I was reading somewhere online about people using their router as an AP to then hook it up to their server with their setup, but to state the obvious, that's obviously just acting as an access.
If it's connected on the WAN port, you still rely on it's firewall, which adds some complexity. Do you have enough ports on the unmanaged and Cisco switches to completely remove the router from the equation?
I was looking through my firewalls as well, I'm familiar when it comes to opening ports, I have a little bit of experience when it comes to dealing with NICs or even VM NICs
I've currently got both my servers trunking, my Cisco switch is a managed L3 switch whilst my Netgear is a L2 switch if I recall, I originally got that just for 1 server I had at the time in case that was in question.
That said, I have enough ports to take the router out of the equation, yeah, it'll be connected directly to my modem however.And what do you mean with "my router pointing to pfsense for DHCP"? Pfsense getting DHCP requests, and handing out IP's means it would typically set itself as the gateway unless you have changed that. And if it does, and it is not connected on the WAN side, you don't get far...
On my router settings, I have the option to change the default gateway, though I think I'll change it back as I'll see about using vlan tags as you suggest. (screenshot below)
As mentioned above though to Stephenw10, I forgot to say that connection to the internet is possible via 172.16.0.x range, just not 172.16.1.x range, which yeah, it could (most likely) be the firewall, I've had to ponder about it, as mentioned above I just have to familiarize myself with firewall for the interface settings, my router as well has a firewall, last night when I posted this thread my mind was focused mainly on the vlans, so I'll see about the firewalls next.
-
Ah so you have a router here that isn't pfSense?
A diagram might help here.
-
@stephenw10 As far as I am aware, nah mate it's most likely not compatible with it (though I thought that was optional), I have an Asus router RT-AC3200.
Diagram is as;
Modem -> Asus router -> Unmanaged Netgear switch -> Managed Cisco Switch -> Servers
-
Hmm, so where are you running pfSense there? As a VM?
-
@stephenw10
Yeah mate via Proxmox, I've got other VMs as well running through two physical servers. -
Ah OK. Then I guess this isn't really a pfSense issue?
-
@ldl said in Newbie questions:
@stephenw10 As far as I am aware, nah mate it's most likely not compatible with it (though I thought that was optional), I have an Asus router RT-AC3200.
Diagram is as;
Modem -> Asus router -> Unmanaged Netgear switch -> Managed Cisco Switch -> Servers
Ok but it is still not clear where you have pfsense connected, and which ports (WAN, LAN, LAN2) on pfsense are connected where??
Also, in your first post you said the modem had IP 192.168.0.X/24 range.
To me that means it is not just a "modem"... it looks more like it is the ISP router which is meant to hand out IP's on your LAN. And as a consequence your Asus router has a WAN address of 192.168.0.something?? So already here you are double NATed, and adding pfsense makes it triple NAT...If this is true, I would think of ways to eliminate one or the other. I suppose you bought the Asus for a reason, so removing the ISP device would be my choice, and eventually replacing the router with pfsense.
If your ISP has "locked" your external IP to the MAC of their modem, you can spoof that on the Asus router, as well as on pfsense if you want.What is clear however is that traffic is going through the router. So anything on LAN 172..16.0.X will have internet.
Similarly you have to set up pfsense so anything "controlled" by it goes through it... meaning if you want to play around with it as a homelab thing, you connect the pfsense WAN to a LAN port on your Asus router (or the unmanaged switch in this case).
Then you move the Cisco switch to the LAN port of pfsense. And your PC for managing pfsense plus all the servers have to be connected to the Cisco switch. Now you can play around with VLANs etc on pfsense and the Cisco switch, and Proxmox if you like. Don't mess with VLAN on the pfsense VM though...
The Asus router should in this case have DHCP turned on again, and just leave it as your standard router.
Your topology will now look like:
Modem -> Asus router -> Unmanaged Netgear switch -> (WAN) pfsense (LAN) -> Managed Cisco Switch -> Servers and your PC
-
Thanks mate, yeah that makes sense.
To answer your question though, pfSense is connected at the server end as it's on a VM, which is through LAN.
And yeah, I've assigned my Asus router a static IP on the 192.168.0.x range.I was also speaking to my friend at work a few months back regarding double NATed, and they also explained that to me, and that it can cause complications down the line, but for the time I've had it set up as such and had no issues, and that's been since I got on the internet a good 20 years ago, but I guess I should break that habit, switching over to AP would help, I guess.
I cannot eliminate the ISP router as its fiber, unless that's where SFP comes in?On the ISP router, all the ports are in use, though if I get this network set up correctly, then I'll be redirecting that through this new setup that I'm trying to achieve.
Thanks for the tips mate, I do appreciate it, for me this is a whole new field for me to explore, and it does seem interesting to go through.
-
Yup double NAT can cause problems but it will work fine for almost everything. It certainly won't cause a complete connectivity failure as long as there are no conflicting subnets.
-
@ldl said in Newbie questions:
Thanks mate, yeah that makes sense.
To answer your question though, pfSense is connected at the server end as it's on a VM, which is through LAN.Ok, but if you want to start using pfsense to route traffic, even if it's just for learning purposes, it needs both a WAN port and a LAN port connected. If your Proxmox machine only has one physical port, you need to start working with VLAN's to solve this (Proxmox VLAN as I showed above, in conjunction with your Cisco switch which needs to be set up appropriately).
Think about how the other routers are connected, WAN <> internal firewall <> LAN. The same applies to pfsense and you want to connect both sides for it to work. Things happening on the LAN side do not involve the firewall, it's mainly handled by the switches.
As I understand how you have it set up, it can hand out IP's and the devices think that pfsense is the gateway. But the traffic has no where to go, since WAN is not connected anywhere...?And yeah, I've assigned my Asus router a static IP on the 192.168.0.x range.
I was also speaking to my friend at work a few months back regarding double NATed, and they also explained that to me, and that it can cause complications down the line, but for the time I've had it set up as such and had no issues, and that's been since I got on the internet a good 20 years ago, but I guess I should break that habit, switching over to AP would help, I guess.
Double NAT is not really a problem for most normal internet use. If you were doing gaming for example, you may end up having trouble playing with friends, since you may not get Open or at least Moderate NAT in the game. But if you plan to go further with your servers and perhaps want to access things from the outside, you need to fix the double NAT situation somehow. The same applies if you are looking at many smart home solutions as well.
I cannot eliminate the ISP router as its fiber, unless that's where SFP comes in?
Yes the fiber comes in with the SFP. And in many cases you have a split setup with a media converter that takes the SFP and converts it into Ethernet (RJ45). From the media converter the ethernet cable goes into the WAN port on your router.
This router can then be any router, not just the one your ISP supplied, meaning you can replace it with your own.
If you want to try this, it's likely a good idea to clone the WAN MAC from the ISP router and enter it in your Asus router like this (type in the MAC that you find in the UI of the ISP router (and/or printed on the back):
There are newer models where the router has the SFP integrated, in which case you can't eliminate it... but you may be able to set it to Bridge Mode instead.
On the ISP router, all the ports are in use, though if I get this network set up correctly, then I'll be redirecting that through this new setup that I'm trying to achieve.
So are you saying that your main home network is actually on the 192.168.0.1/24 subnet? Do you have switches connected there as well? Is your Asus router what you use for wifi?
If you need the ports on both the ISP router and your Asus, you can change one or the other into a "switch". For example if you turn off DHCP on the ISP router, and move the WAN cable over to your Asus router (after cloining the MAC). Then you can still make use of the LAN ports on the ISP unit.Topology in this scenario:
Fiber to Ethernet (media converter) > Asus router > ISP router (using only LAN ports) > Netgear switch > WAN pfsense LAN > Cisco switch > serversIt will be a bit tricky to get the last part working unless you have more than one physical port on the Proxmox machine where you run pfsense. Preferably you should have at least three ports, of which two are dedicated to pfsense (WAN and LAN).
Perhaps you should draw yourself a diagram for the setup so you fully understand what you are doing. Especially if you have to use VLANs to make it work.
I don't know Cisco switches but how I'm thinking you could do this is the following :
Set port 1 to VLAN ID 10 (not entirely sure how this will work towards the netgear switch?)
Set port 2 to VLAN ID 10 and 1
Leave all other ports at ID 1 (default).The idea is to only allow traffic with VLAN tag 10 to pass between ports 1 and 2.
The cable coming from your Netgear switch will go into port 1 and your Proxmox server with the pfsense VM will connect to port 2.
You need to go into the Proxmox UI and make sure you have two ports for the pfsense VM, both using the same bridge port (vmbr0). One of these will be the WAN port and for this one you have to set the VLAN tag to 10. The other you leave at default.
This way your WAN port on pfsense will be communicating up towards the router via port 1 on the Cisco switch. And the LAN port will use default VLAN covering ports 2-N.
-
Apologises in the delay.
That's most likely where I'm going wrong, as I've been leaving WAN blank, but yeah, to answer your question, I do have 4 physical ports, on both servers, all connected up to the switch.
@ double nat, ah okay well I learn something new every day, but yeah, I will be eliminating the double nat.
The ISP runs on 192.168.0.x range, no other switches are connected to there, just on the 172.16.0.x range that I have two on (unmanaged Netgear + Managed Cisco), I also have another switch, that isn't in use.
But yeah, I've been trying to find some softwares I can use to draw up a diagram, sure I could just use Paint or something, but I'd want some sort of software that I can keep my IPs in order, though that's another subject.
Again, thanks for the helpful information.
-
@ldl If you have as many as 4 ports on each server, it will of be much simpler and no need to fiddle with VLAN's.
Still, consider removing the ISP router and connecting the Asus directly, as a first step. Then when you feel confident using pfsense, you replace the Asus and move that over to the LAN side of pfsense (only using LAN ports and disabling DHCP).
In your current setup, the Proxmox machine with pfsense VM should have one port connected to the Netgear switch, which will be your WAN for pfsense. All other ports on that Proxmox as well as the other machine, should be conncted to the Cisco switch which will palce all VM's entirely in the pfsense "domain".
So the topology you are looking at for starters is:
Fiber to Ethernet (media converter) > Asus router > ISP router (using only LAN ports) > Netgear switch > WAN pfsense LAN > Cisco switch > all other server ports: -
@Gblenn Again, apologises in the delay.
Okay thanks for the information, I've also been looking at alternative ISP purely on the cost and higher up/download speeds, one in particular says they would use a direct RJ45 connection, but I personally want to keep the fiber lead.
Out of curiosity, would it be beneficial in my requirements to use the upstream gateway?
Cheers.
-
@ldl said in Newbie questions:
@Gblenn Again, apologises in the delay.
Okay thanks for the information, I've also been looking at alternative ISP purely on the cost and higher up/download speeds, one in particular says they would use a direct RJ45 connection, but I personally want to keep the fiber lead.
Out of curiosity, would it be beneficial in my requirements to use the upstream gateway?
Cheers.
When you say, "use the upstream gateway", do you mean the ISP provided router?
I have never found any benefit in using the ISP's equipment. Although my current ISP have actually provided a quite powerful Zyxel device capable of 10Gig on the LAN side, and wifi 6. But it still ended up in it's box in storage...
Instead I'm using TPLink Omada gear, for both switching and wifi and it's so much simpler having just one interface to work with. And then I have pfsense as my gateway/firewall.It's mainly the functionality that will be lacking when using the ISP equipment, or even the Asus router you have. Which is why you would want to move towards having pfsense as your "entrypoint" and bring your fiber directly into it (perhaps via a media converter). A 1Gbit model will start at around 20USD and a 2.5GBit perhaps 2 - 3 times that.
When I said "for starters", I meant that you run with the topology you have, until you feel you want to use pfsense the way it's intended. Your Asus router can then be used as your wifi AP, perhaps together with your ISP router in some other location in the home to add wifi coverage.Since you already have fiber to your home, perhaps the ISP you talked to mean that they will provide a media converter which is what my current ISP did when I had 1Gbit. I got one of these super devices: https://www.amazon.com/s?k=media+converter+1gb&crid=3I07NTVFKVYZU&sprefix=media+converter+1g%2Caps%2C157&ref=nb_sb_ss_ts-doa-p_1_18
And there is no harm in using that of course. But perhaps you want to keep building and experimenting with your pfsense machine and then you can always put an SFP/SFP+ card in it. Which then gives you the possibility to plug the fiber module directly into the WAN port for pfsense.
