Confusion on subnets and local port forwarding
-
I tried asking this over on the r/pfsense subreddit, but was told that my setup looks fine and should be working and to see if anyone more knowledgeable over here might be able to help.
Post hereSo this is a bit of an unusual setup here. I have an isolated network (no internet/WAN connection) managed by a pfSense router that contains about a dozen devices on two different subnets, some of which are embedded devices whose configured static IPs cannot be changed.
This is my current setup, ignoring some of the devices for now and changing the IP addresses a bit:
devices:
- pfSense with LAN interface on address 10.10.60.1 (gateway for 10.10.60.0 devices)
- virtual ip of 10.10.60.10 (forwarded)
- virtual ip of 10.10.240.1 (gateway for 10.10.240.0 devices)
- virtual ip of 10.10.240.10 (forwarded)
.
-
VM on ip 10.10.60.12
-
laptop on ip 10.10.60.230
-
server on ip 10.10.240.93
-
raspberry pi on ip 10.10.240.50
configuration:
-
firewall rule with source 10.10.60.0 and destination any
-
firewall rule with source 10.10.240.0 and destination any
-
port forward on LAN interface forwarding 10.10.60.10 to 10.10.240.93
-
port forward on LAN interface forwarding 10.10.240.10 to 10.10.240.93
Currently, any of these devices can ssh into any other device through the target device's ip directly, even across subnets, as expected. So the laptop on 10.10.60.230 can ssh into the server at 10.10.240.93 no problem. The devices on the 10.10.60.0 subnet can also ssh into the server through 10.10.60.10 via the port forwarding rule. They can also access it through 10.10.240.10 via the other port forward. However, the devices on the 10.10.240.0 (the raspberry pi) subnet cannot ssh into the server via either 10.10.60.10 or 10.10.240.10. The port forwarding seems to work only from the 10.10.60.0 subnet, not the 10.10.240.0 one.
I suspect this has something to do with the fact that the pfsense LAN is actually sitting on the 10.10.60.0 subnet, but I'm not familiar enough with how subnets and firewalls work to advance further.
I'm by no means a network engineer, I was just tasked with getting this set up because I'm the only one on my team with any pfsense experience at all. I'm happy to provide any other information about the setup if it'll help diagnose the issue.
Any ideas as to why the 240 subnet can't use the port forward rules set up? Any recommendations or things that I could at least look at would be awesome.
PS. the reason I'm trying to do all this is because there's an NTP server running on the server at 10.10.240.93 that all other devices use to get the time, so they all need access to the server. Unfortunately the server occasionally moves addresses (not something that can be helped), so I'd prefer to change the address the the port forwards point to in pfsense and point all the devices to 10.10.60.10, rather than going into all the dozen devices and updating the NTP server's address each time it changes. The 10.10.240.10 port forward exists because I wasn't sure if the devices on 10.10.240.0 could use the 10.10.60.10 port forward or not, so I set it up as a test.
Some extra information from the comments:
subnet mask anywhere that pfsense allows entering one (like LAN rules) I've used /24 and on devices that are configurable (like the ubuntu laptop) I've set netmask to 255.255.255.0. As far as I understand it, those are basically the same thing, so it should match up (?)
"Can any device in 10.10.240.0 ssh to a 10.20.60.0 device, period?"
Yes, any of the four devices can ssh into any of the other four, including the 10.10.240.0 devices into the 10.10.60.0 devices.
"Can you confirm they can ping?"
Yes, all devices can ping each other no problem.
"If yes, can they ping 10.10.60.10?"
Yes, all devices, including those on 10.10.240.0, can ping 10.10.60.10.
"Lastly, if that's all successful, your port forward rule or security rule is bad and I'd have to see them. Specifically, make sure the right interface is selected for each, that usually is the issue for users port forwarding."
nat and rules -
@jblackburn said in Confusion on subnets and local port forwarding:
some of which are embedded devices whose configured static IPs cannot be changed.
Also embedded devices should give you an option to configure it's IP.
pfSense with LAN interface on address 10.10.60.1 (gateway for 10.10.60.0 devices) virtual ip of 10.10.60.10 (forwarded) virtual ip of 10.10.240.1 (gateway for 10.10.240.0 devices) virtual ip of 10.10.240.10 (forwarded)This is not a reliable network set up at all. And further it doesn't give you any more security than putting all devices in a single L2 network.
You should assign different subnets to different network segment for proper routing.
However, the devices on the 10.10.240.0 (the raspberry pi) subnet cannot ssh into the server via either 10.10.60.10 or 10.10.240.10. The port forwarding seems to work only from the 10.10.60.0 subnet, not the 10.10.240.0 one.
I suspect this has something to do with the fact that the pfsense LAN is actually sitting on the 10.10.60.0 subnet, but I'm not familiar enough with how subnets and firewalls work to advance further.
No, you will have an asymmetric routing issue.
The devices in 10.10.240.0/24 might send the request packets to one of the virtual IPs on pfSense though, they get forwarded then to 10.10.240.93. Now the server sends responses back to the source IP, but this doesn't pass pfSense, since the destination is within the same subnet.
A IP communication has to use the same route in both directions, otherwise it doesn't work.PS. the reason I'm trying to do all this is because there's an NTP server running on the server at 10.10.240.93 that all other devices use to get the time, so they all need access to the server.
pfSense can also provide an NTP server. Services > NTP
It would be better to use this one, since this is the gateway on all devices and all can communicate with it.Unfortunately the server occasionally moves addresses (not something that can be helped)
What? If it's your network it should do, what you tell it to do.
subnet mask anywhere that pfsense allows entering one (like LAN rules) I've used /24 and on devices that are configurable (like the ubuntu laptop) I've set netmask to 255.255.255.0. As far as I understand it, those are basically the same thing, so it should match up (?)
Yes.
Apart from my security concerns, you can workaround the asymmetric routing though by masquerading the source IP in packets going to the server with an outbound NAT rule.
But doing this, the server will only see the pfSense virtual IP in all requests. -
@viragomann said in Confusion on subnets and local port forwarding:
@jblackburn said in Confusion on subnets and local port forwarding:
some of which are embedded devices whose configured static IPs cannot be changed.
Also embedded devices should give you an option to configure it's IP.
This isn't my home network, it's a network located in a test vehicle at work. The devices that can't be configured are devices that we've been loaned to test out without being given any access to. Basically a 'here, this device is on this ip address and spits out this data, have fun' type of situation.
pfSense with LAN interface on address 10.10.60.1 (gateway for 10.10.60.0 devices) virtual ip of 10.10.60.10 (forwarded) virtual ip of 10.10.240.1 (gateway for 10.10.240.0 devices) virtual ip of 10.10.240.10 (forwarded)This is not a reliable network set up at all. And further it doesn't give you any more security than putting all devices in a single L2 network.
I'm not doing this for security, it's a local only network anyway, so there's no outside access to begin with.
You should assign different subnets to different network segment for proper routing.
What do you mean by different network segments? I'm not very familiar with networking terminology yet.
However, the devices on the 10.10.240.0 (the raspberry pi) subnet cannot ssh into the server via either 10.10.60.10 or 10.10.240.10. The port forwarding seems to work only from the 10.10.60.0 subnet, not the 10.10.240.0 one.
I suspect this has something to do with the fact that the pfsense LAN is actually sitting on the 10.10.60.0 subnet, but I'm not familiar enough with how subnets and firewalls work to advance further.
No, you will have an asymmetric routing issue.
The devices in 10.10.240.0/24 might send the request packets to one of the virtual IPs on pfSense though, they get forwarded then to 10.10.240.93. Now the server sends responses back to the source IP, but this doesn't pass pfSense, since the destination is within the same subnet.
A IP communication has to use the same route in both directions, otherwise it doesn't work.So in addition to forwarding 10.10.60.10 => 10.10.240.93, I also need to forward 10.10.240.93 => 10.10.60.10? I'll give that a try.
PS. the reason I'm trying to do all this is because there's an NTP server running on the server at 10.10.240.93 that all other devices use to get the time, so they all need access to the server.
pfSense can also provide an NTP server. Services > NTP
It would be better to use this one, since this is the gateway on all devices and all can communicate with it.It would, but in this case the NTP server needs to run on the server. I'm not 100% sure the details, but I know it doesn't serve the actual time of day, it serves an arbitrarily settable time, which is changed depending on what's being done with the whole system at the moment.
Unfortunately the server occasionally moves addresses (not something that can be helped)
What? If it's your network it should do, what you tell it to do.
I agree if this was my home network. It's not so much that the server randomly jumps addresses, but we change out server hardware sometimes (again sometimes we have access and sometimes we don't) so the address changes occasionally.
subnet mask anywhere that pfsense allows entering one (like LAN rules) I've used /24 and on devices that are configurable (like the ubuntu laptop) I've set netmask to 255.255.255.0. As far as I understand it, those are basically the same thing, so it should match up (?)
Yes.
Apart from my security concerns, you can workaround the asymmetric routing though by masquerading the source IP in packets going to the server with an outbound NAT rule.
But doing this, the server will only see the pfSense virtual IP in all requests.
I will try that as well, only seeing the VIPs shouldn't be an issue in this case.
Thank you for the help! -
@jblackburn said in Confusion on subnets and local port forwarding:
What do you mean by different network segments? I'm not very familiar with networking terminology yet.
Different network segments are connected to different interfaces on the router.
So any traffic to a destination inside of a network segment goes directly to the destination device and back again, any outside passes the router in both directions.But i you need to redirect a traffic even within the same network segment for whatever reason, this won't work.
So in addition to forwarding 10.10.60.10 => 10.10.240.93, I also need to forward 10.10.240.93 => 10.10.60.10? I'll give that a try.
No, you need an outbound NAT rule.
Set the outbound NAT into hybrid mode, save and add a rule:
interface: LAN
source: any
destination: network, enter 10.10.240.93, mask /32
translation: select 10.10.240.1 -
@viragomann said in Confusion on subnets and local port forwarding:
No, you need an outbound NAT rule.
Set the outbound NAT into hybrid mode, save and add a rule:
interface: LAN
source: any
destination: network, enter 10.10.240.93, mask /32
translation: select 10.10.240.1This worked, thank you very much.
-
I tried asking this over on the r/pfsense subreddit, but was told that my setup looks fine
Really - you brought up that your running vips for different networks and nobody said anything other than that looks fine?
@viragomann is correct this is a pretty odd ball setup.. I would never do it like that, its one way I guess to get around not knowing what IPs these devices will be - but don't you have an actual different interface you can use on your pfsense - so you could actually create whatever network you needed for these other devices you have to work with? And just put whatever network they are on that interface while you work with them, then you wouldn't have to do any natting or vips, etc. Or even port forwards, etc.
it's a network located in a test vehicle at work.
Even if you don't have more interfaces on pfsense - a cheap say 8 port vlan capable switch would allow you to proper bring up any networks you wanted to, and then could route off pfsense via vlans, etc. so you in theory could bring up like 6 different networks on a 8 port switch, 1 connection to pfsense, one connection to your server/laptop whatever you working with - and then 6 other ports for other devices/networks. You can pick up a 8 port gig vlan capable switch for like 40$ or so.. I would think this would be a requirement in your test vehicle networking kit..
-
@johnpoz said in Confusion on subnets and local port forwarding:
I tried asking this over on the r/pfsense subreddit, but was told that my setup looks fine
Really - you brought up that your running vips for different networks and nobody said anything other than that looks fine?
Well to be fair, "looks fine" weren't the exact words, more like "that's gross but should work I think".
@viragomann is correct this is a pretty odd ball setup..
I agree, and believe me I would love to avoid it by just having access to the devices under test, but working with big companies is a nightmare.
I would never do it like that, its one way I guess to get around not knowing what IPs these devices will be - but don't you have an actual different interface you can use on your pfsense - so you could actually create whatever network you needed for these other devices you have to work with? And just put whatever network they are on that interface while you work with them, then you wouldn't have to do any natting or vips, etc. Or even port forwards, etc.
That would work for now, since we only have two subnets involved, but I wanted to make a more future-proof setup for when we inevitably get more subnets going.
it's a network located in a test vehicle at work.
Even if you don't have more interfaces on pfsense - a cheap say 8 port vlan capable switch would allow you to proper bring up any networks you wanted to, and then could route off pfsense via vlans, etc. so you in theory could bring up like 6 different networks on a 8 port switch, 1 connection to pfsense, one connection to your server/laptop whatever you working with - and then 6 other ports for other devices/networks. You can pick up a 8 port gig vlan capable switch for like 40$ or so.. I would think this would be a requirement in your test vehicle networking kit..
Yeah this is what I'll probably do going forward, I only found out that this would be an option like an hour ago. Again, I'm not a networking guy by any means.
-
@jblackburn yeah a vlan capable switch which can be very cheap and small would make a great addition to your abilities of what you can do and how easy it is to do without having to jump through odd hoops to get something to work ;)
Would allow you to bring up any network you want on your pfsense without having to really have lots of interfaces on the pfsense device itself.. Add a AP that can also do vlans - and man you would really be cooking with gas ;)
