Adding a new subnet to server almost stops file transfers - why?
-
I have a box with an Asus mini-ITX mobo that has been running pfSense totally flawless for about half a year now, with mostly standard stuff, only a few extras (some DNS host overrides). It has the mobo's network port as WAN and I had a two port Intel server NIC for my network (192.168.1.x) and an OPT1 network to the part of my house that I rent out (10.0.0.x). On my network pfSense feeds into a Windows Server 2016 with 6 network ports, and I was only using two of them - WAN in from pfSense 192.168.1.4 and my internal network 192.168.0.x, with the Windows Server as DNS, DHCP and RAS server.
But I wanted to separate some home automation stuff (four Rasberry Pi running Node-RED and Home Assistant and communicating with MQTT messages, one running an AirPlay receiver for a room that I couldn't get to with speaker cables and the four receivers that feed the rest of my house with audio) from the 192.168.1.x network, since internet use sometimes created dropouts in AirPlay and I didn't want to have the home automation stuff accessible from any computers on my internal 192.168.0.x network, only from the server itself and the VM I have running home automation software.
So I replaced the two port card in the pfSense box with a four port Intel Server NIC. No problem, up and running with that card and my net and the OPT1 net within ten minutes. Then I created the OPT2 home automation network as 192.168.10.x and set up the standard firewall rules to allow the Pis internet access to update and send mail. I am planning to block anything that I don't need later.
But that's when the fun started. Because with this setup file transfers to and from the Windows server slow to a crawl from devices on my internal 192.168.0.x network. If I disable the NIC from 192.168.10.x network on the server, file transfers work as they should. They also work as they should if I untick all protocols except for VMWare in the NIC settings in Windows Server 2016. But I would like to keep the TCP/IP V4 protocol because I use a program on the server that monitors all my home automation (the MQTT messages to and from the VM and the Pi's and if stuff is turned on as it should) and reboots the Pi's or the VM if anything stops.
I have tried to set a manual metric to 9999 for the 192.168.10.x NIC on the server (lowest allowed value), but that didn't help. Does anybody know why this is happening and what I can do to fix it?
-
You might have accidentally introduced some asymmetric routing. It is easy to do with anything dual-homed.
Client in subnet A tries to reach the IP address of a device in subnet B, if that device is dual homed it will respond from its NIC in subnet A, which would at worse fail to work or at best work for a few moments then drop once the state dies since pfSense doesn't see both parts of the conversation.
It could also be that the file transfers are running through the firewall now instead of directly, which would also slow things down.
Check how the clients resolve the name of the server in local DNS and also check out the flow of traffic with packet captures, and see what the connection states look like.
-
Thanks for answering! The thing is that this should not go through pfSense at all. I'm trying to copy files from the server, which is connected to pfSense in one main in (192.168.1.x) and the secondary for home automation (192.168.10.x), while the computers I'm trying to copy files to is on the insie of the server (192.168.0.x). So in theory it should never even touch the networks on the outside of the server. When I to route print on the client on the inside there's no mentioning of the 192.168.1.x or the 192.168.10.x network. When doing it on the server I see both, but I don't see any wrong references to them. The 10.x network is in 0.0.0.0, but I think it will be when it's connected to the server. The metric is correct too, very high on the 10.x network. This is the route-print that has the 192.168.10.x network in it on the server:
192.168.10.0 255.255.255.0 On-link 192.168.10.4 10255
192.168.10.4 255.255.255.255 On-link 192.168.10.4 10255
192.168.10.255 255.255.255.255 On-link 192.168.10.4 10255
255.255.255.255 255.255.255.255 On-link 192.168.10.4 10255I also see that when I activate this interface it will send all internet traffic from the server through it even if the metric is over 10 000. But this seems to be a Windows Server issue, not a pfSense issue, so maybe I should take it to Windows board instead.
-
Please break out a napkin and some crayons if need be an draw this... So you have server that is multi homed with a 192.168.0 network hanging off it? What are the masks on these networks.. If you use say /16 or even /23 192.168.0.x and 192.168.1.x become the same network..
-
I am embarassingly bad at drawing, but I have tried to create a (simplified) chart of my network. Simplified because I have removed a few VM's and other stuff that does not come in to this at all. Is this at all understandable?
-
Oh, and just to repeat, the problem is that file transfers from server to physical clients is slow to almost stopping. Everything else is working as it should.
-
@mastiff said in Adding a new subnet to server almost stops file transfers - why?:
file transfers from server to physical clients is slow to almost stopping.
Please use IP addresses, not descriptions such as that to avoid interpretation mistakes..
If you were to add the default gateways on all the parts behind the firewall to your diagram I'm sure you would see the asymmetric routing you have almost certainly created.
-
The default gateway for the 192.168.0.150 and 192.168.0.151 are both 192.168.0.1, which is the server that I'm trying to copy the files from. Is this more like what you mean?
-
Maybe it's clearer with some color to the three different nets?
-
Is pink connected to pfSense directly or does it just look like it?
-
It is. I have 192.168.1x on LAN, a network for another part of the house (not shown here, because it has no other connection points) on Opt 1, and pink/192.168.10.x on opt 2.
-
Your server cannot have more than one default gateway in that case. It can have one active default gateway.
-
You have created a real mess there.
-
I probably have... But tidying up the mess, what do I do as a default gateway on the pink? I have tried to do a static IP on the NIC and set it to 192.168.1.1, but that didn't change anything.
Edit: There's actually only one thing I need the server to have access to from the pink network, and that's mosquitto messages. Would that be better served by removing that from the server all together and create a rule between 192.168.10.x and 192.168.1.x that only lets MQTT through on port 8883?
Edit 2: Oh wait, that won't work! I need access to Airplay too there, and that needs a local connection, I believe.
-
Why is the internal network (192.168.0.0/24) apparently being "routed" by the server?
Your problem is whatever "server" is is also being asked to be a router.
You might be able to get airplay working with one router hop in between using the avahi package. Last time I looked at it, Airplay sent its "hellos" with a TTL of 2 so you could have one router hop in between. There are also DNS tricks you can do.
The trouble is these "home automation" companies do not care about/support networks with more than one subnet.
What is the IP address of the device making the connections you are having trouble with?
What is the IP address of the server it is making connections to?
What is the port/protocol/etc of the connection it is trying to make?
-
The internal network is being routed by the server because it should be. This is a Windows Server 2016 that treats the input 192.168.1.4 from the pfSense box as any regular WAN connection. So there's actually three levels of routing here, the DSL modem, the pfSense box and the Windows Server 2016. That way I have separated different networks for different needs in layers. I would prefer to have only two layers and use the DSL modem in bridge mode, but for some weird reason that stops voice over WiFi from working, and we need that because even if we live in the middle of town, the cell signal is lousy.
I will look at that avahi package and see if that works, it would be nice if that solves the problem.
Oh, and almost all the home automation stuff is mostly on the one subnet. I have no problems with that at all.
As I said in the first post, the only problem is that regular file transfers (so Windows file sharing, which I guess is Samba in reality) from the Server to the clients on the internal network (from 192.168.0.1 to 192.168.0.150 and .151 in this example drawing) is slowing to a crawl almost stopping when the 192.168.10.x network is connected from the pfSense box to a separate network port on the server (I mean separate from the 192.168.1.4 network port). Everything else works.
-
Your problem is entirely on the Windows server there because if clients on 192.168.0.0/24 are talking to the server with a connected interface of 192.168.0.x/24 traffic between them should never leave that segment. If it is, it's the server doing it.
No idea why you are trying to use a windows server as a router so I'm probably not going to be able to help.
-
I'm not just using it as a router, I'm using it as a full server, as any SOHO server. File server, routing and remote access, DNS, DHCP, print server, media server, Softether tunnel to a second site, virtual machine host and so on. But I think you're right, there's something happening on the server that I don't understand. I'll ask the question in a Windows Server oriented forum.
-
Right but there is no reason to "route" your internal subnet through it in that case.
Nobody does that.
You would just put it and all of your hosts on 192.168.1.0/24.
-
I see where something may have been unclear. The pink network only goes INTO the Windows Server, there are no arrows on the network lines. So it doesn't go out from the Windows Server, there's no DHCP or anything running on the server that feeds the pink network, all that comes from the pfSense box. The only network going out from the server is 192.168.0.1.
So I'm not routing my internal network through it, I am RUNNING a separate internal 0.x network with 1.x used strictly as a WAN. I have stuff on the 1.x network that I don't want to be visible on the internal 0.x network, which is why I have an extra layer. That's not accessible from the clients on the internal netwrok if they don't know the precise IP address of them. I just lef those out because they didn't really have anything to do with this particular question.
