GigE between 2 subnets = 50Mbit limit?

argon0011

Currently experiencing a strange problem.

Network Diagram http://imgur.com/syGg0

I have 2 private subnets (servers and desktops), a PFsense firewall for each and a public /27 in this network. There is GigE everywhere between the 2 machines I'm testing with.

I have a Windows 7 machine on the "Servers" network, and another on the "Desktops" network. Copying a file from Servers to Desktops via Windows share is only capable 3.5MByte/s. But if I put a Desktop machine on the /27 Netgear switch (see diagram) and try retrieving the same file from the W7 machine on the "servers" network, I get near 100MByte/s

I then thought perhaps its some strange Windows Share oddity, so I tried using iperf (jperf in Windows) and between Servers and Desktops it brought back about 6MBytes = 48Mbit.

The 2 server firewalls are about 6 months old, and the desktop firewall is a few weeks old. They are running i3 processors, but hardly use any CPU. All machines have Intel GigE PCI-e cards, aside from the CARP sync opt interface using the onboard Realtek 100mbit. I've also checked all the interfaces in PFsense and they all show 1000 Full Duplex.

Any ideas?

Edit: 1 thing to mention, the config on the desktop firewall was restored, was not entered in manually.

podilarius

Did you test the desktop firewall by putting the server on the netgear /27 and retesting? Did you also traceroute to make sure you are not involving any other equipment?

argon0011

Thanks for your reply. I may have been wrong in the initial post with the 100mbyte speed, the below may have been more like it.

I did some more testing with jperf.
-Servers to /27 = 160Mbit
-Desktops to /27 = 195Mbit

There may be 5Mbit overhead in each case due to the office using the network.

The thing is, even if I got these speeds with Windows share between the 2 networks I'd be happy. Right now 48Mbit between the 2 networks isn't good enough.

podilarius

What is the speed of your internet?
When you have them in their separate subnets, did you do a traceroute to make sure traffic is hitting the correct gateways?

argon0011

Internet is 40Mbit, but that sits on the other side of the network in question.

Tracert:
1 <1 ms <1 ms <1 ms 192.168.10.2
2 1 ms 2 ms 1 ms 203.91.90.98
3 <1 ms 1 ms 1 ms 192.168.1.171

podilarius

The trace looks incorrect. Are you trying to use the internal address of the servers? If that is the case, each FW will have to consult its gateway (the cisco 2800). If you have a 2801 or a 2811, there is only a 10/100 connection. Try instead to setup a NAT and associated rules on the server's firewall and access it by that IP address (203.91.90.101 perhaps). Or, I would imagine that you could setup the routes on the pfSense machines directly instead of the Cisco. IE, 192.168.10.0/24 GW 203.91.90.98 on the server's pfSense and 192.168.1.0/24 GW 203.91.90.100 on the desktop's pfSense machine. You just have to make sure that your rules also allow for private IPs to pass.

argon0011

That was a trace from servers to desktops.

The firewalls shouldn't be consulting the default gateway because they have a specific route learnt through RIP pointing directly at each other. There's also no-NAT rules for servers to desktops and vice-versa.

podilarius

Well we are trying to figure out the problem. Let us eliminate RIP and set a perm route to make sure.