GigE between 2 subnets = 50Mbit limit?

  • Currently experiencing a strange problem.

    Network Diagram

    I have 2 private subnets (servers and desktops), a PFsense firewall for each and a public /27 in this network. There is GigE everywhere between the 2 machines I'm testing with.

    I have a Windows 7 machine on the "Servers" network, and another on the "Desktops" network. Copying a file from Servers to Desktops via Windows share is only capable 3.5MByte/s. But if I put a Desktop machine on the /27 Netgear switch (see diagram) and try retrieving the same file from the W7 machine on the "servers" network, I get near 100MByte/s

    I then thought perhaps its some strange Windows Share oddity, so I tried using iperf (jperf in Windows) and between Servers and Desktops it brought back about 6MBytes = 48Mbit.

    The 2 server firewalls are about 6 months old, and the desktop firewall is a few weeks old. They are running i3 processors, but hardly use any CPU. All machines have Intel GigE PCI-e cards, aside from the CARP sync opt interface using the onboard Realtek 100mbit. I've also checked all the interfaces in PFsense and they all show 1000 Full Duplex.

    Any ideas?

    Edit: 1 thing to mention, the config on the desktop firewall was restored, was not entered in manually.

  • Did you test the desktop firewall by putting the server on the netgear /27 and retesting? Did you also traceroute to make sure you are not involving any other equipment?

  • Thanks for your reply. I may have been wrong in the initial post with the 100mbyte speed, the below may have been more like it.

    I did some more testing with jperf.
    -Servers to /27 = 160Mbit
    -Desktops to /27 = 195Mbit

    There may be 5Mbit overhead in each case due to the office using the network.

    The thing is, even if I got these speeds with Windows share between the 2 networks I'd be happy. Right now 48Mbit between the 2 networks isn't good enough.

  • What is the speed of your internet?
    When you have them in their separate subnets, did you do a traceroute to make sure traffic is hitting the correct gateways?

  • Internet is 40Mbit, but that sits on the other side of the network in question.

      1    <1 ms    <1 ms    <1 ms
      2    1 ms    2 ms    1 ms
      3    <1 ms    1 ms    1 ms

  • The trace looks incorrect. Are you trying to use the internal address of the servers? If that is the case, each FW will have to consult its gateway (the cisco 2800). If you have a 2801 or a 2811, there is only a 10/100 connection. Try instead to setup a NAT and associated rules on the server's firewall and access it by that IP address ( perhaps). Or, I would imagine that you could setup the routes on the pfSense machines directly instead of the Cisco. IE, GW on the server's pfSense and GW on the desktop's pfSense machine. You just have to make sure that your rules also allow for private IPs to pass.

  • That was a trace from servers to desktops.

    The firewalls shouldn't be consulting the default gateway because they have a specific route learnt through RIP pointing directly at each other. There's also no-NAT rules for servers to desktops and vice-versa.

  • Well we are trying to figure out the problem. Let us eliminate RIP and set a perm route to make sure.