Adding a new subnet to server almost stops file transfers - why?

  • I have a box with an Asus mini-ITX mobo that has been running pfSense totally flawless for about half a year now, with mostly standard stuff, only a few extras (some DNS host overrides). It has the mobo's network port as WAN and I had a two port Intel server NIC for my network (192.168.1.x) and an OPT1 network to the part of my house that I rent out (10.0.0.x). On my network pfSense feeds into a Windows Server 2016 with 6 network ports, and I was only using two of them - WAN in from pfSense and my internal network 192.168.0.x, with the Windows Server as DNS, DHCP and RAS server.

    But I wanted to separate some home automation stuff (four Rasberry Pi running Node-RED and Home Assistant and communicating with MQTT messages, one running an AirPlay receiver for a room that I couldn't get to with speaker cables and the four receivers that feed the rest of my house with audio) from the 192.168.1.x network, since internet use sometimes created dropouts in AirPlay and I didn't want to have the home automation stuff accessible from any computers on my internal 192.168.0.x network, only from the server itself and the VM I have running home automation software.

    So I replaced the two port card in the pfSense box with a four port Intel Server NIC. No problem, up and running with that card and my net and the OPT1 net within ten minutes. Then I created the OPT2 home automation network as 192.168.10.x and set up the standard firewall rules to allow the Pis internet access to update and send mail. I am planning to block anything that I don't need later.

    But that's when the fun started. Because with this setup file transfers to and from the Windows server slow to a crawl from devices on my internal 192.168.0.x network. If I disable the NIC from 192.168.10.x network on the server, file transfers work as they should. They also work as they should if I untick all protocols except for VMWare in the NIC settings in Windows Server 2016. But I would like to keep the TCP/IP V4 protocol because I use a program on the server that monitors all my home automation (the MQTT messages to and from the VM and the Pi's and if stuff is turned on as it should) and reboots the Pi's or the VM if anything stops.

    I have tried to set a manual metric to 9999 for the 192.168.10.x NIC on the server (lowest allowed value), but that didn't help. Does anybody know why this is happening and what I can do to fix it?

  • Rebel Alliance Developer Netgate

    You might have accidentally introduced some asymmetric routing. It is easy to do with anything dual-homed.

    Client in subnet A tries to reach the IP address of a device in subnet B, if that device is dual homed it will respond from its NIC in subnet A, which would at worse fail to work or at best work for a few moments then drop once the state dies since pfSense doesn't see both parts of the conversation.

    It could also be that the file transfers are running through the firewall now instead of directly, which would also slow things down.

    Check how the clients resolve the name of the server in local DNS and also check out the flow of traffic with packet captures, and see what the connection states look like.

  • Thanks for answering! The thing is that this should not go through pfSense at all. I'm trying to copy files from the server, which is connected to pfSense in one main in (192.168.1.x) and the secondary for home automation (192.168.10.x), while the computers I'm trying to copy files to is on the insie of the server (192.168.0.x). So in theory it should never even touch the networks on the outside of the server. When I to route print on the client on the inside there's no mentioning of the 192.168.1.x or the 192.168.10.x network. When doing it on the server I see both, but I don't see any wrong references to them. The 10.x network is in, but I think it will be when it's connected to the server. The metric is correct too, very high on the 10.x network. This is the route-print that has the 192.168.10.x network in it on the server: On-link 10255 On-link 10255 On-link 10255 On-link 10255

    I also see that when I activate this interface it will send all internet traffic from the server through it even if the metric is over 10 000. But this seems to be a Windows Server issue, not a pfSense issue, so maybe I should take it to Windows board instead.

  • LAYER 8 Global Moderator

    Please break out a napkin and some crayons if need be an draw this... So you have server that is multi homed with a 192.168.0 network hanging off it? What are the masks on these networks.. If you use say /16 or even /23 192.168.0.x and 192.168.1.x become the same network..

  • I am embarassingly bad at drawing, but I have tried to create a (simplified) chart of my network. Simplified because I have removed a few VM's and other stuff that does not come in to this at all. Is this at all understandable?

    0_1541957538895_Mastiff's network.jpg

  • Oh, and just to repeat, the problem is that file transfers from server to physical clients is slow to almost stopping. Everything else is working as it should.

  • LAYER 8 Netgate

    @mastiff said in Adding a new subnet to server almost stops file transfers - why?:

    file transfers from server to physical clients is slow to almost stopping.

    Please use IP addresses, not descriptions such as that to avoid interpretation mistakes..

    If you were to add the default gateways on all the parts behind the firewall to your diagram I'm sure you would see the asymmetric routing you have almost certainly created.

  • The default gateway for the and are both, which is the server that I'm trying to copy the files from. Is this more like what you mean? 0_1541961038018_8c01b2a2-8c70-4162-ad5e-1a7dbe964e4c-bilde.png

  • Maybe it's clearer with some color to the three different nets? 0_1541962325008_7013da00-84f3-416e-a255-3a04a73fc03a-bilde.png

  • LAYER 8 Netgate

    Is pink connected to pfSense directly or does it just look like it?

  • It is. I have 192.168.1x on LAN, a network for another part of the house (not shown here, because it has no other connection points) on Opt 1, and pink/192.168.10.x on opt 2.

  • LAYER 8 Netgate

    Your server cannot have more than one default gateway in that case. It can have one active default gateway.

  • LAYER 8 Netgate

    You have created a real mess there.

  • I probably have... But tidying up the mess, what do I do as a default gateway on the pink? I have tried to do a static IP on the NIC and set it to, but that didn't change anything.

    Edit: There's actually only one thing I need the server to have access to from the pink network, and that's mosquitto messages. Would that be better served by removing that from the server all together and create a rule between 192.168.10.x and 192.168.1.x that only lets MQTT through on port 8883?

    Edit 2: Oh wait, that won't work! I need access to Airplay too there, and that needs a local connection, I believe.

  • LAYER 8 Netgate

    Why is the internal network ( apparently being "routed" by the server?

    Your problem is whatever "server" is is also being asked to be a router.

    You might be able to get airplay working with one router hop in between using the avahi package. Last time I looked at it, Airplay sent its "hellos" with a TTL of 2 so you could have one router hop in between. There are also DNS tricks you can do.

    The trouble is these "home automation" companies do not care about/support networks with more than one subnet.

    What is the IP address of the device making the connections you are having trouble with?

    What is the IP address of the server it is making connections to?

    What is the port/protocol/etc of the connection it is trying to make?

  • The internal network is being routed by the server because it should be. This is a Windows Server 2016 that treats the input from the pfSense box as any regular WAN connection. So there's actually three levels of routing here, the DSL modem, the pfSense box and the Windows Server 2016. That way I have separated different networks for different needs in layers. I would prefer to have only two layers and use the DSL modem in bridge mode, but for some weird reason that stops voice over WiFi from working, and we need that because even if we live in the middle of town, the cell signal is lousy.

    I will look at that avahi package and see if that works, it would be nice if that solves the problem.

    Oh, and almost all the home automation stuff is mostly on the one subnet. I have no problems with that at all.

    As I said in the first post, the only problem is that regular file transfers (so Windows file sharing, which I guess is Samba in reality) from the Server to the clients on the internal network (from to and .151 in this example drawing) is slowing to a crawl almost stopping when the 192.168.10.x network is connected from the pfSense box to a separate network port on the server (I mean separate from the network port). Everything else works.

  • LAYER 8 Netgate

    Your problem is entirely on the Windows server there because if clients on are talking to the server with a connected interface of 192.168.0.x/24 traffic between them should never leave that segment. If it is, it's the server doing it.

    No idea why you are trying to use a windows server as a router so I'm probably not going to be able to help.

  • I'm not just using it as a router, I'm using it as a full server, as any SOHO server. File server, routing and remote access, DNS, DHCP, print server, media server, Softether tunnel to a second site, virtual machine host and so on. But I think you're right, there's something happening on the server that I don't understand. I'll ask the question in a Windows Server oriented forum.

  • LAYER 8 Netgate

    Right but there is no reason to "route" your internal subnet through it in that case.

    Nobody does that.

    You would just put it and all of your hosts on

  • I see where something may have been unclear. The pink network only goes INTO the Windows Server, there are no arrows on the network lines. So it doesn't go out from the Windows Server, there's no DHCP or anything running on the server that feeds the pink network, all that comes from the pfSense box. The only network going out from the server is

    So I'm not routing my internal network through it, I am RUNNING a separate internal 0.x network with 1.x used strictly as a WAN. I have stuff on the 1.x network that I don't want to be visible on the internal 0.x network, which is why I have an extra layer. That's not accessible from the clients on the internal netwrok if they don't know the precise IP address of them. I just lef those out because they didn't really have anything to do with this particular question.

  • LAYER 8 Netgate

    Right. And segmenting your networks is what people use routers and firewalls for, not windows server 2016.

    I am RUNNING a separate internal 0.x network with 1.x used strictly as a WAN.

    I have no idea what this even means.

    Maybe someone else can decipher it.

  • It depends on how you want to have your network, I guess. I look on pfSense as the army standing between the dangerous internet and my server, and my server is the friendly policeman who keeps my little town (internal network) in line and sends the cars (data packets) and trucks (files, media and so on) where they need to be. I have been running Windows Servers in my home since NT and I have no need or wish to arrange my network with anything but a Windows server as the central do it all in my network.

    By running as opposed to routing I mean that for the internal network the server does everything in one box, I don't split the server roles as it seems you want me to do, with one box for routing and one box for everything else the server does, like VM hosting, file server, media server, AirPlay server with TuneBlade, FTP server and so on.

    So basically I wouldn't need pfSense or any other router if the internet was secure, I could just plug my Windows Server straight to it. But it isn't. And since I now have it here I thought I would use the extra optional networks compartmentalize stuff even a bit more, with the home automation and media stuff going straight to receivers and active speakers in the 14 zones in the house on a single network, since AirPlay is a bit unstable if there's a lot of small packages going fast through the network, like several people browsing or gaming at the same time.

  • And I may have misunderstood you, but as far as I know the server isn't segmenting the network the way I understand segmenting. It's taking 192.168.1.x from the pfSense box in as a WAN and is then serving the internal network with the 192.168.0.x segment and nothing else. The pfSense box does the segmenting into 192.168.1x and 192.168.10.x (and more segments that don't even touch my sever). I would just like the server to take in an extra feed from the segment 192.168.10.x and use it for programs running on the server itself, not share that out to anything. The automation server VM running on the physical server host gets both 192.168.1.x and 192.168.10.x directly from bridging the actual network ports, not via the server OS.

  • Netgate Administrator

    It does seem like there is a high possibility of asymmetric routing here. However you would not expect that between clients and the server in the same subnet unless something is misconfigured there.
    Some packet captures of the transfer should show whats happening.

    I would probably disconnect the home automation subnet from the server directly and just add a static route to it through pfSense.


  • LAYER 8 Global Moderator

    @mastiff said in Adding a new subnet to server almost stops file transfers - why?:

    It's taking 192.168.1.x from the pfSense box in as a WAN and is then serving the internal network with the 192.168.0.x segment and nothing else

    Then 192.168.0/24 can not get to the internet? Or any other segment..

    If your "server" nats this traffic or routes this traffic then it is acting a router/firewall? be it you think it is or not..

    If you want to use a "server" as a downstream router have at it... But you also have this other VM on the left multihomed in every segment?? Why?? Makes zero sense.

    You have a very powerful router and firewall with pfsense... Have at and segment up your network into how ever many you want/need.. I am currently running 9 different segments. All routed and firewall thru pfsense.. And also have a downstream copy of pfsense running as VM testing when users have messes and they need help in how to do a downstream router via transit network... Which is the ONLY proper way to do a downstream router, or your going to run into asymmetrical problems..

    Devices your going to multihome, ie put in more than 1 network should only have a default gateway on the network that gets them out to other networks (other than the one they are directly attached too).. When you start doing multihomed with multiple gateways it gets complicated really quickly... Which unless you really really have a reason for this and your different gateways can both get to any other network that your box might need to get to its going to run you into problems. If you need to use a gateway on some other network of these multihomed box then you need to create routes on that box and it should only have 1 default gateway, etc.

    Again - makes no sense to do such a complicated setup with what amounts to a very basic network with 3 vlans.. All of which can be done on pfsense and a simple cheap vlan capable switch.

  • Stephenw10, thanks! No, it confused the heck out of me too. I just can't figure out why it affects the wrong subnet. But I will try to remember how I use WireShark (I think it's been ten years since I needed to do a packet capture, and then I found out that there was a bad switch in the house I had then).

    The problem with using the same subnet is AirPlay, which I want to move to another net than everything else (except for the MQTT messages, which are so small that they don't affect anything) because surfing and gaming makes AirPlay stutter at times.

    Johnpoz, I understood the splitting of subnets to mean that he thought I used the Windows server to split 192.168.1.x and 192.168.10.x, which I don't. But yes, I take 192.168.1.x into the server as WAN, and I use the Routing and remote access, DNS server and DHCP Windows Server roles to give the internal network clients access to Internet. But almost without NAT or firewalling, I think the only thing I have NAT'ed is the FTP server. The rest that should be accessible from outside is on VM's, with the NAT in the pfSense box.

    The VM is for home automation and has some parts available directly from the Internet 192.168.1.x (which I don't want to carry over to the internal network 192.168.0.x), some parts controlling stuff on the home automation network 192.168.10.x and some parts that are only available from the internal network 192.168.0.x (like home secuerity control). That's why it has so many connections (it actually has six more, but they are from VM pfSense for very special purposes, I remember that you really loved that setup when I asked for a few advices some months ago... 😈 And surprise, surprise... It has worked completely without a hitch with the six VM pfSense feeding different DynDNS web hops into the correct part of the VM home automation (which is necessary because the only way the home automation software can separate isolated web pages is to have them come from separate IP's).

    But nothing of this can easily be done the way you suggest because my home automation setup has very specific requirements. And it works extremely stable, I only reboot for updates, and everything's available from where it should be and nowhere else.

    As for my pfSense box I'm using that to split into three regular segments and one VPN for watching Netflix like an american. ;)

    But I will set up wireshark and see if I can see something wonky here.

  • LAYER 8 Global Moderator

    @mastiff said in Adding a new subnet to server almost stops file transfers - why?:

    ut nothing of this can easily be done the way you suggest because my home automation setup has very specific requirements.

    So... Pfsense is a ROUTER and a firewall.. Any sort of segmentation you need can be done with a couple of clicks and a 30$ switch that understands vlans... Or if you want a bunch of dumb switches..

    Needing things on the same layer 2 for stuff like airplay could all be solved with the avhai package since all airplay needs the multicast discovery to be passed from 1 layer 2 to the other... Multihome stuff so airplay works is just nonsense sorry..

    When I see such messes I just scream inside my head... I do this for a living and such nonsense is freaking painful!! You do understand you could do simple policy route for your devices to access whatever you want via vpn to circumvent geoip restrictions.. OMG this thread just makes my head hurt..

    Where did I ever say I loved such a mess - point me to this thread... There is NO Possible way I would of said anything nice about such a setup.. Yes I like segmentation of network for security and control - not multihomed nonsense..

    Are you talking about the proxy and url nonsense thread where your interface looks like a dancing baby should be in the corner its so horrific looking?? That thread - nowhere did I ever say anything was done right.. It was like pulling teeth with a pair of tweezers even getting any info from you...

    I have to call it quits that thread was bad enough I can not go through another one of those nightmares stephen can have fun with this one if he wants... Im done - sorry... But I can only deal with so much crazy..

  • Sorry, I thought you would understand the sarcasm when I put your brother after the sentence. I was obviously wrong. Of course you hated it. But it works for me. And it seems like you have an unhealthy amount of feelings attached to networking, even when it's somebody else's network. For me this does exactly what I want to do, and it's 100 % stable. So why does it bother you so much?

  • Yep, according to Wireshark it's trying to copy the file from the interface even if I use the server's IP to access the shares. So it's something strange happening inside the Windows server. Because when I check the routes (route print) on the client I'm trying to copy the files to, there's no mentioning of the 192.168.10.x network at all there.

  • Netgate Administrator

    The problem with this sort of setup, in general, is that it's far too easy to introduce some small config error that causes loops or asymmetry or worse, bypasses security.

    Anyway it looks like some Windows routing issue given it's sending internet traffic the wrong way and the problem only happens when that interface is connected. I don't think this is a pfSense config issue from what we've seen so far.


  • No, me neither. I think it's all Windows. Oh, and it's not even internet traffic, it's local file copies from files hosted on the server to the client. I'll just drop looking at it from the pfSense point of view and try to limit the NIC as much as I can with Routing and Remote Access, DHCP and DNS, perhaps I'll stumble over something. Thanks for trying to help, Steve! :)

  • LAYER 8 Global Moderator

    @mastiff said in Adding a new subnet to server almost stops file transfers - why?:

    perhaps I'll stumble over something.

    OMG - that is not how you setup a network... Or for that matter troubleshoot something --- Arrrggghhhh..


  • As I said, too much feelings for networking, but I guess you're the angry chef in Hell's Kitchen, while I'm the naked chef. More experimenting, but usually the final result is edible. 😂 Anyway I came back to say that I managed to stumble over it (by stumble over I mean methodically testing setting after setting until something changes, but since I don't actually know what setting will do it I call it stumble over) and fixed it.

    Turned out that it came from Routing and Remote Access on the server, which I found out by turning that off. That fixed it, so then it was only a matter of finding out how to block the card from RAS. After a bunch of tries I found that too: I had to add the card under IPv4, General, open the properties and then Enable IP router manager on the interface and set up filters that blocked everything from the 192.168.0.x network from being routed. So now it works as it should, and I am now listening to Galactic Empire's second album, of course called "Episode II", over AirPlay and there are no dropouts.

  • Netgate Administrator

    Good result in the end then. Just don't look behind the curtain! 😉


  • Exactly! If the cat catches mice, then who cares what it looks like!