Newbie questions
-
Ah so you have a router here that isn't pfSense?
A diagram might help here.
-
@stephenw10 As far as I am aware, nah mate it's most likely not compatible with it (though I thought that was optional), I have an Asus router RT-AC3200.
Diagram is as;
Modem -> Asus router -> Unmanaged Netgear switch -> Managed Cisco Switch -> Servers
-
Hmm, so where are you running pfSense there? As a VM?
-
@stephenw10
Yeah mate via Proxmox, I've got other VMs as well running through two physical servers. -
Ah OK. Then I guess this isn't really a pfSense issue?
-
@ldl said in Newbie questions:
@stephenw10 As far as I am aware, nah mate it's most likely not compatible with it (though I thought that was optional), I have an Asus router RT-AC3200.
Diagram is as;
Modem -> Asus router -> Unmanaged Netgear switch -> Managed Cisco Switch -> Servers
Ok but it is still not clear where you have pfsense connected, and which ports (WAN, LAN, LAN2) on pfsense are connected where??
Also, in your first post you said the modem had IP 192.168.0.X/24 range.
To me that means it is not just a "modem"... it looks more like it is the ISP router which is meant to hand out IP's on your LAN. And as a consequence your Asus router has a WAN address of 192.168.0.something?? So already here you are double NATed, and adding pfsense makes it triple NAT...If this is true, I would think of ways to eliminate one or the other. I suppose you bought the Asus for a reason, so removing the ISP device would be my choice, and eventually replacing the router with pfsense.
If your ISP has "locked" your external IP to the MAC of their modem, you can spoof that on the Asus router, as well as on pfsense if you want.What is clear however is that traffic is going through the router. So anything on LAN 172..16.0.X will have internet.
Similarly you have to set up pfsense so anything "controlled" by it goes through it... meaning if you want to play around with it as a homelab thing, you connect the pfsense WAN to a LAN port on your Asus router (or the unmanaged switch in this case).
Then you move the Cisco switch to the LAN port of pfsense. And your PC for managing pfsense plus all the servers have to be connected to the Cisco switch. Now you can play around with VLANs etc on pfsense and the Cisco switch, and Proxmox if you like. Don't mess with VLAN on the pfsense VM though...
The Asus router should in this case have DHCP turned on again, and just leave it as your standard router.
Your topology will now look like:
Modem -> Asus router -> Unmanaged Netgear switch -> (WAN) pfsense (LAN) -> Managed Cisco Switch -> Servers and your PC
-
Thanks mate, yeah that makes sense.
To answer your question though, pfSense is connected at the server end as it's on a VM, which is through LAN.
And yeah, I've assigned my Asus router a static IP on the 192.168.0.x range.I was also speaking to my friend at work a few months back regarding double NATed, and they also explained that to me, and that it can cause complications down the line, but for the time I've had it set up as such and had no issues, and that's been since I got on the internet a good 20 years ago, but I guess I should break that habit, switching over to AP would help, I guess.
I cannot eliminate the ISP router as its fiber, unless that's where SFP comes in?On the ISP router, all the ports are in use, though if I get this network set up correctly, then I'll be redirecting that through this new setup that I'm trying to achieve.
Thanks for the tips mate, I do appreciate it, for me this is a whole new field for me to explore, and it does seem interesting to go through.
-
Yup double NAT can cause problems but it will work fine for almost everything. It certainly won't cause a complete connectivity failure as long as there are no conflicting subnets.
-
@ldl said in Newbie questions:
Thanks mate, yeah that makes sense.
To answer your question though, pfSense is connected at the server end as it's on a VM, which is through LAN.Ok, but if you want to start using pfsense to route traffic, even if it's just for learning purposes, it needs both a WAN port and a LAN port connected. If your Proxmox machine only has one physical port, you need to start working with VLAN's to solve this (Proxmox VLAN as I showed above, in conjunction with your Cisco switch which needs to be set up appropriately).
Think about how the other routers are connected, WAN <> internal firewall <> LAN. The same applies to pfsense and you want to connect both sides for it to work. Things happening on the LAN side do not involve the firewall, it's mainly handled by the switches.
As I understand how you have it set up, it can hand out IP's and the devices think that pfsense is the gateway. But the traffic has no where to go, since WAN is not connected anywhere...?And yeah, I've assigned my Asus router a static IP on the 192.168.0.x range.
I was also speaking to my friend at work a few months back regarding double NATed, and they also explained that to me, and that it can cause complications down the line, but for the time I've had it set up as such and had no issues, and that's been since I got on the internet a good 20 years ago, but I guess I should break that habit, switching over to AP would help, I guess.
Double NAT is not really a problem for most normal internet use. If you were doing gaming for example, you may end up having trouble playing with friends, since you may not get Open or at least Moderate NAT in the game. But if you plan to go further with your servers and perhaps want to access things from the outside, you need to fix the double NAT situation somehow. The same applies if you are looking at many smart home solutions as well.
I cannot eliminate the ISP router as its fiber, unless that's where SFP comes in?
Yes the fiber comes in with the SFP. And in many cases you have a split setup with a media converter that takes the SFP and converts it into Ethernet (RJ45). From the media converter the ethernet cable goes into the WAN port on your router.
This router can then be any router, not just the one your ISP supplied, meaning you can replace it with your own.
If you want to try this, it's likely a good idea to clone the WAN MAC from the ISP router and enter it in your Asus router like this (type in the MAC that you find in the UI of the ISP router (and/or printed on the back):
There are newer models where the router has the SFP integrated, in which case you can't eliminate it... but you may be able to set it to Bridge Mode instead.
On the ISP router, all the ports are in use, though if I get this network set up correctly, then I'll be redirecting that through this new setup that I'm trying to achieve.
So are you saying that your main home network is actually on the 192.168.0.1/24 subnet? Do you have switches connected there as well? Is your Asus router what you use for wifi?
If you need the ports on both the ISP router and your Asus, you can change one or the other into a "switch". For example if you turn off DHCP on the ISP router, and move the WAN cable over to your Asus router (after cloining the MAC). Then you can still make use of the LAN ports on the ISP unit.Topology in this scenario:
Fiber to Ethernet (media converter) > Asus router > ISP router (using only LAN ports) > Netgear switch > WAN pfsense LAN > Cisco switch > serversIt will be a bit tricky to get the last part working unless you have more than one physical port on the Proxmox machine where you run pfsense. Preferably you should have at least three ports, of which two are dedicated to pfsense (WAN and LAN).
Perhaps you should draw yourself a diagram for the setup so you fully understand what you are doing. Especially if you have to use VLANs to make it work.
I don't know Cisco switches but how I'm thinking you could do this is the following :
Set port 1 to VLAN ID 10 (not entirely sure how this will work towards the netgear switch?)
Set port 2 to VLAN ID 10 and 1
Leave all other ports at ID 1 (default).The idea is to only allow traffic with VLAN tag 10 to pass between ports 1 and 2.
The cable coming from your Netgear switch will go into port 1 and your Proxmox server with the pfsense VM will connect to port 2.
You need to go into the Proxmox UI and make sure you have two ports for the pfsense VM, both using the same bridge port (vmbr0). One of these will be the WAN port and for this one you have to set the VLAN tag to 10. The other you leave at default.
This way your WAN port on pfsense will be communicating up towards the router via port 1 on the Cisco switch. And the LAN port will use default VLAN covering ports 2-N.
-
Apologises in the delay.
That's most likely where I'm going wrong, as I've been leaving WAN blank, but yeah, to answer your question, I do have 4 physical ports, on both servers, all connected up to the switch.
@ double nat, ah okay well I learn something new every day, but yeah, I will be eliminating the double nat.
The ISP runs on 192.168.0.x range, no other switches are connected to there, just on the 172.16.0.x range that I have two on (unmanaged Netgear + Managed Cisco), I also have another switch, that isn't in use.
But yeah, I've been trying to find some softwares I can use to draw up a diagram, sure I could just use Paint or something, but I'd want some sort of software that I can keep my IPs in order, though that's another subject.
Again, thanks for the helpful information.
-
@ldl If you have as many as 4 ports on each server, it will of be much simpler and no need to fiddle with VLAN's.
Still, consider removing the ISP router and connecting the Asus directly, as a first step. Then when you feel confident using pfsense, you replace the Asus and move that over to the LAN side of pfsense (only using LAN ports and disabling DHCP).
In your current setup, the Proxmox machine with pfsense VM should have one port connected to the Netgear switch, which will be your WAN for pfsense. All other ports on that Proxmox as well as the other machine, should be conncted to the Cisco switch which will palce all VM's entirely in the pfsense "domain".
So the topology you are looking at for starters is:
Fiber to Ethernet (media converter) > Asus router > ISP router (using only LAN ports) > Netgear switch > WAN pfsense LAN > Cisco switch > all other server ports: -
@Gblenn Again, apologises in the delay.
Okay thanks for the information, I've also been looking at alternative ISP purely on the cost and higher up/download speeds, one in particular says they would use a direct RJ45 connection, but I personally want to keep the fiber lead.
Out of curiosity, would it be beneficial in my requirements to use the upstream gateway?
Cheers.
-
@ldl said in Newbie questions:
@Gblenn Again, apologises in the delay.
Okay thanks for the information, I've also been looking at alternative ISP purely on the cost and higher up/download speeds, one in particular says they would use a direct RJ45 connection, but I personally want to keep the fiber lead.
Out of curiosity, would it be beneficial in my requirements to use the upstream gateway?
Cheers.
When you say, "use the upstream gateway", do you mean the ISP provided router?
I have never found any benefit in using the ISP's equipment. Although my current ISP have actually provided a quite powerful Zyxel device capable of 10Gig on the LAN side, and wifi 6. But it still ended up in it's box in storage...
Instead I'm using TPLink Omada gear, for both switching and wifi and it's so much simpler having just one interface to work with. And then I have pfsense as my gateway/firewall.It's mainly the functionality that will be lacking when using the ISP equipment, or even the Asus router you have. Which is why you would want to move towards having pfsense as your "entrypoint" and bring your fiber directly into it (perhaps via a media converter). A 1Gbit model will start at around 20USD and a 2.5GBit perhaps 2 - 3 times that.
When I said "for starters", I meant that you run with the topology you have, until you feel you want to use pfsense the way it's intended. Your Asus router can then be used as your wifi AP, perhaps together with your ISP router in some other location in the home to add wifi coverage.Since you already have fiber to your home, perhaps the ISP you talked to mean that they will provide a media converter which is what my current ISP did when I had 1Gbit. I got one of these super devices: https://www.amazon.com/s?k=media+converter+1gb&crid=3I07NTVFKVYZU&sprefix=media+converter+1g%2Caps%2C157&ref=nb_sb_ss_ts-doa-p_1_18
And there is no harm in using that of course. But perhaps you want to keep building and experimenting with your pfsense machine and then you can always put an SFP/SFP+ card in it. Which then gives you the possibility to plug the fiber module directly into the WAN port for pfsense.
-
Apologises again in the delay.
I was referring to the pfSense's upstream gateway, as I'm currently experimenting quite a lot with this, trying to get to what I need to achieve.
I will at some point be changing out my Asus router for something more suitable to my needs, as its outdated as well.
According to the response I got on their forums (yeah, they have a forum, I've never known one to have one xD), it'll be connected via ONT, and then terminated in an RJ45.
I've not come across this before, so if I do go for this ISP, then maybe it'll be better, though they claim to have 2x faster speed than my current ISP (An ISP that brags to be the best in the UK)
Thanks for the feedback again as well.
-
@ldl Ok, so it's like I mentioned, this other ISP will terminate with what I referred to as a media converter, the ONT. It is then entirely up to you what you decide to use as a router/firewall.
And you have already built two Proxmox servers, with multi NIC's, and you have pfsense up and running as a VM. Given this, I'd say you are ready to change out your Asus router already, and replace it with pfsense.
To simplify things I'd make sure to clone the MAC address of the ISP router to the WAN interface of pfsese before connecting to the ONT. If you change ISP, you just change the MAC to what the new router you get from them has. It's written on the back of the device, and you can likely find it in the UI. Or you can connect it's WAN port your pfsense LAN and find it in the list of DHCP Leases in pfsense, where you can easily copy paste it.
For Proxmox, you should look into IOMMU (pass thru), to have the necessary NICs completely handed over to pfsense. Availability of this functionality depends on the generation HW you have (CPU/motherboard). But it will give the best performance and control from a pfsense perspective.
With pfsense and your cisco switch you have all the possibilities to continue playing around with VLAN's and all sorts of fun stuff. If your Asus router supports VLAN, you can start creating multiple isolated wifi networks, for guests, IoT stuff etc. But if not, it will still be able to serve as a wifi AP, as long as you remember to use LAN ports only, set a different IP compared to the pfsense UI, and turn off DHCP.
-
@Gblenn said in Newbie questions:
@ldl Ok, so it's like I mentioned, this other ISP will terminate with what I referred to as a media converter, the ONT. It is then entirely up to you what you decide to use as a router/firewall.
And you have already built two Proxmox servers, with multi NIC's, and you have pfsense up and running as a VM. Given this, I'd say you are ready to change out your Asus router already, and replace it with pfsense.
To simplify things I'd make sure to clone the MAC address of the ISP router to the WAN interface of pfsese before connecting to the ONT. If you change ISP, you just change the MAC to what the new router you get from them has. It's written on the back of the device, and you can likely find it in the UI. Or you can connect it's WAN port your pfsense LAN and find it in the list of DHCP Leases in pfsense, where you can easily copy paste it.
For Proxmox, you should look into IOMMU (pass thru), to have the necessary NICs completely handed over to pfsense. Availability of this functionality depends on the generation HW you have (CPU/motherboard). But it will give the best performance and control from a pfsense perspective.
With pfsense and your cisco switch you have all the possibilities to continue playing around with VLAN's and all sorts of fun stuff. If your Asus router supports VLAN, you can start creating multiple isolated wifi networks, for guests, IoT stuff etc. But if not, it will still be able to serve as a wifi AP, as long as you remember to use LAN ports only, set a different IP compared to the pfsense UI, and turn off DHCP.
I forgot to mention, that this ISP I will be switching to near the end of next month (as I have to give 30 days notice to my current ISP), is that they give me the option at a cost per month to use one of their routers, or I can use my own, so there will be no MAC issues, which is good.
I will be upgrading my router sometime next month as well, because currently my WAN port on my router only has a max output of 1Gb, which tbf, at the time, for me is enough as I was only able to get up to 1Gb download and 100Mb Upload from my current ISP, but this new ISP offers 2.5Gb for both up/download, as well as offering IPv6, which is something else I want to get familiar with.
Right now I'm just looking at a 2.5Gb WAN for a router and 1Gb for the LANs, I've found two that I will decide on the next month.
It's also obviously good to update my current router, as it no longer has firmware updates available.On to the subject though of pfSense, I ran into this weird issue two days ago, where for some reason, when I reset my servers, pfSense was no longer able to communicate to the network, it was able to ping out to the internet (8.8.8.8), just not on the intranet/network, I resolved that by setting the interfaces, though I'm having to use 23 mask on my router and wanting to use 25 mask on pfSense, if I recall though, they all need to be on the same mask, as the course I'm currently on leading towards Cyber Security, covered CompTIA, and the networking side of things, yeah it is indeed something fun to get into, I'm learning quite a lot, but on this course, I could have swear they said I need to set the mask to the same one across the board, unless they just meant on those servers trying to communicate with each other, but how I see it, if this setup is correct, obviously for the router itself, when I set it to 23 mask, it's able to talk to the IP range that my servers are on, whereas, I'm guessing if it's on the 25 mask, that my servers are on, then it'll be limited to the range it's currently set on, do correct me if I'm wrong on this.
Sorry for all this hassle, I also set up routes as well from my Asus router to allow communications between the devices, as well, the devices on the 172.16.1.x range don't appear on the routers connected devices which is on the 172.16.0.x range, I'm guessing I need to mess around with SNMP or something for this?
-
@ldl said in Newbie questions:
I could have swear they said I need to set the mask to the same one across the board
If I were you, I would take their advice. Do not "split" subnets. That's not how it is supposed to be done. Set your subnet mask to /24 and have multiple subnets with that mask. Use routing to move traffic from one subnet to another. subnets.
-
@kjk54 Yeah mate, this is what I thought originally, I had just a year to cram all this knowledge, being Network, Security, Modern Desktop, Hardware/Software (I know a bit regarding hardware and software anyway) etc, but yeah, I do recall hearing this, just found it weird though did kind of make sense with the mask being lower with access to the IP ranges given by the router after setting it.
That said, I set up routes both ways, and it was having issues connecting to them with the same mask across the board, thinking it was a firewall issue, I temporarily disabled it to rule out that issue, next would have been checking bridged connections, but that led to some complications, though probably a rookie error on me that is. -
@ldl said in Newbie questions:
I will be upgrading my router sometime next month as well, because currently my WAN port on my router only has a max output of 1Gb, which tbf, at the time, for me is enough as I was only able to get up to 1Gb download and 100Mb Upload from my current ISP, but this new ISP offers 2.5Gb for both up/download, as well as offering IPv6, which is something else I want to get familiar with.
Right now I'm just looking at a 2.5Gb WAN for a router and 1Gb for the LANs, I've found two that I will decide on the next month.
It's also obviously good to update my current router, as it no longer has firmware updates available.Do you mean you are shopping for NIC's to use in pfsense? Or are you thinking of replacing the Asus router with something newer?
I don't really see a point of having multiple routers, in fact it just complicates things. Make pfsense your main router and upgrade it with a multi NIC card, with either two or four 2.5G ports. Later on you can upgrade your PC and your switches to accomodate 2.5G. Many or most new PC's, motherboards and Laptops come with 2.5G ethernet today. Don't limit yourself on your LAN side, especially when you are paying for 2.5G internet... -
@Gblenn Replacing my asus router with something newer, as the Asus one is outdated (the main reason), sure still works but yeah.
Another reason as to why I want to replace it, is that if I'm going to use my own router, then other people in my house will obviously be on the same line, so I want to accommodate them as well, because currently, they're not on my router as that's in another room, they're on the ISP router, when I switch over to the newer ISP then that will change and all devices (being wired) will be connected to the router, so with my current router I have it only has the capability for 1Gb on WAN, the routers I've been looking at support 2.5Gb WAN which then has 1Gb on the LANs, I'm not looking to utilise all that connection speed on to one device alone, I simply want to guarantee 1Gb speed to my servers, as with this setup I have, I'm hosting a good few gaming servers for closed sessions of friends.
Sure that most likely wouldn't use up 1Gb, but it'd be nice to have that speed on it.
I'm also currently only getting low speeds to what I'm currently paying for as well (screenshot below) on Cat7
That also said, my PC, servers and switches only go up to 1Gb.
If this ISP offered lower speed if that was possible, say 1.5Gb, then I'd go for that instead, that said, it's only a £10 difference from the next package down, I also have the option to downgrade in the future.
I will be considering upgrading the NICs and switches in the future however if I feel the need for more than 1Gb
