Router on a stick problems: Double Bandwidth and OpenVPN chokes

[NetBandit]

Hi everyone.

Previously I was using pfSense 2.0.x 32bit on a system with multiple network cards.  It worked great.  However, those network cards were old, and frankly, I didn't trust them.

When pfSense 2.1 was released, I decided to switch to 64bit (server has 4GB RAM), and use a 'router on a stick' setup in conjunction with my Cisco SG300-52 switch instead of using those old network cards.

Anyway, it looks like it go it working… but I discovered problems after I had been using it a while:

1:  My bandwidth graphs report double the speed in both directions
2:  OpenVPN connections will choke when using most of the bandwidth.  By that I mean that the remotely connected client will not be able to ping the pfSense router for a period of 10 or more seconds, but it will resume working again.  This happens when first establishing a Remote Desktop connection, or trying to transfer large amounts of data between the local and remote site (anything that maxes out the bandwidth).

Neither of these were a problem when I was using my previous configuration.

A little more about my configuration:

3 WAN (1 cable, 2 DSL)(gateway grouping not yet implemented... so it's like just a main WAN, and the VPN uses the secondary WAN)
2 LAN (main, and a guest/wifi, but not yet isolated - I can route between networks)
1 VPN

Since I'm new to routing on a stick, I think I may have goofed up my port/VLAN configuration on the switch.
Here is the VLAN configuration:

switch#show vlan

VLAN      Name                  Ports                Type    Authorization
–-- ----------------- --------------------------- ------------ -------------
1          1                gi52,Po1-8            Default      Required
10        Cable                gi49,gi52          permanent    Required
11        DSL1                gi50,gi52          permanent    Required
12        DSL2                  gi51-52            permanent    Required
128  Guest+WiFi.128          gi46-48,gi52          permanent    Required
1200      Main.0              gi1-45,gi52          permanent    Required

Port 52 is my 'trunk' port which is the only connection between the switch and the pfSense router.
VLAN 10 is my main WAN connection
VLAN 11 is my DSL1 WAN connection (used by Open VPN)
VLAN 12 is my DSL2 WAN connection (not used by anything right now)
VLAN 128 is my Guest/WiFi VLAN
VLAN 1200 is my main VLAN

I've attached a screen shot of my VPN bandwidth graph.  As you can see the bandwidth goes to maximum, then drops off completely, then repeats the cycle. (DSL1 is the interface used by OpenVPN.)  The attached picture shows both problems:  Double bandwidth (this connection is 768kbps/7.5mbps, and it's showing 1.74Mbps outbound), as well as the bandwidth going from max to nothing, and back again.

Like I said before I think I might have goofed up something on my switchport and VLAN configuration.  Possibly related to the trunkport and the assigned VLANs… but I'm not sure.  I didn't catch the problem during implementation, and it's too late for me to revert to my old config (2.0.x, 32bit, multiple NICs)

Can anyone see what I'm doing wrong?
Thanks!
-nb

[cmb]

Traffic double: https://redmine.pfsense.org/issues/3314

Can you define "when using most of the bandwidth"? Most of what bandwidth, your Internet, traffic routed between internal VLANs at gigabit speed, or? Establishing a RDP session is a trivial amount of bandwidth, that suggests that part at least is either a coincidence with something else going on at the same time, or it's not specifically bandwidth-related. It's probably not related to your VLAN config, as that tends to be all or nothing, either works or doesn't.

[NetBandit]

@cmb:

Traffic double: https://redmine.pfsense.org/issues/3314

Can you define "when using most of the bandwidth"? Most of what bandwidth, your Internet, traffic routed between internal VLANs at gigabit speed, or? Establishing a RDP session is a trivial amount of bandwidth, that suggests that part at least is either a coincidence with something else going on at the same time, or it's not specifically bandwidth-related. It's probably not related to your VLAN config, as that tends to be all or nothing, either works or doesn't.

Hi, thanks for your thoughtful reply.

You are right.  RDP generally is not a very bandwidth heavy application… except for when first connecting (caching backgrounds, etc.) or when redrawing large amounts of the screen, or transferring files.

The bandwidth you are asking about is the bandwidth of the internet connection, specifically outbound from the pfSense router to the remote system.  When that bandwidth gets maxxed, is when the VPN starts dropping packets.  I have run a ping test on it, and the pings will only stop once I start a file transfer, etc.  The pings will stay stopped for a few more seconds (10+) even after the bandwidth of the application has stopped.

Also, just for clarification, the issue is not just with RDP.  Other application which max out the outbound bandwidth will also cause the same problem.

Also, bug 3314 looks like my issue with regard to the bandwidth graph.

thanks,
-nb

[NetBandit]

@cmb:

Traffic double: https://redmine.pfsense.org/issues/3314

Just to follow up:
According to the post in the bug listing, the RRD graph is correct, which is what I see in my environment, so I think we can confirm the issue is indeed 3314.

-nb

[NetBandit]

With respect to the OpenVPN issue, I noticed this in the log at the same time:

May 19 05:42:34 openvpn[13072]: event_wait : Interrupted system call (code=4)
May 19 05:42:34 openvpn[13072]: /usr/local/sbin/ovpn-linkdown ovpns1 1500 1558 192.168.129.1 192.168.129.2 init
May 19 05:42:34 openvpn[13072]: SIGTERM[hard,] received, process exiting
May 19 05:42:34 openvpn[88542]: OpenVPN 2.3.2 amd64-portbld-freebsd8.3 [SSL (OpenSSL)] [LZO] [eurephia] [MH] [IPv6] built on Mar 27 2014
May 19 05:42:34 openvpn[88542]: NOTE: the current –script-security setting may allow this configuration to call user-defined scripts
May 19 05:42:34 openvpn[88542]: Control Channel Authentication: using '/var/etc/openvpn/server1.tls-auth' as a OpenVPN static key file
May 19 05:42:34 openvpn[88542]: TUN/TAP device ovpns1 exists previously, keep at program end
May 19 05:42:34 openvpn[88542]: TUN/TAP device /dev/tun1 opened
May 19 05:42:34 openvpn[88542]: do_ifconfig, tt->ipv6=1, tt->did_ifconfig_ipv6_setup=0
May 19 05:42:34 openvpn[88542]: /sbin/ifconfig ovpns1 192.168.129.1 192.168.129.2 mtu 1500 netmask 255.255.255.255 up
May 19 05:42:34 openvpn[88542]: /usr/local/sbin/ovpn-linkup ovpns1 1500 1558 192.168.129.1 192.168.129.2 init
May 19 05:42:34 openvpn[90295]: UDPv4 link local (bound): [AF_INET][REDACTED]:1194
May 19 05:42:34 openvpn[90295]: UDPv4 link remote: [undef]
May 19 05:42:34 openvpn[90295]: Initialization Sequence Completed
May 19 05:42:56 openvpn[90295]: event_wait : Interrupted system call (code=4)
May 19 05:42:56 openvpn[90295]: /usr/local/sbin/ovpn-linkdown ovpns1 1500 1558 192.168.129.1 192.168.129.2 init
May 19 05:42:56 openvpn[90295]: SIGTERM[hard,] received, process exiting
May 19 05:42:57 openvpn[3516]: OpenVPN 2.3.2 amd64-portbld-freebsd8.3 [SSL (OpenSSL)] [LZO] [eurephia] [MH] [IPv6] built on Mar 27 2014
May 19 05:42:57 openvpn[3516]: NOTE: the current –script-security setting may allow this configuration to call user-defined scripts
May 19 05:42:57 openvpn[3516]: Control Channel Authentication: using '/var/etc/openvpn/server1.tls-auth' as a OpenVPN static key file
May 19 05:42:57 openvpn[3516]: TUN/TAP device ovpns1 exists previously, keep at program end
May 19 05:42:57 openvpn[3516]: TUN/TAP device /dev/tun1 opened
May 19 05:42:57 openvpn[3516]: do_ifconfig, tt->ipv6=1, tt->did_ifconfig_ipv6_setup=0
May 19 05:42:57 openvpn[3516]: /sbin/ifconfig ovpns1 192.168.129.1 192.168.129.2 mtu 1500 netmask 255.255.255.255 up
May 19 05:42:57 openvpn[3516]: /usr/local/sbin/ovpn-linkup ovpns1 1500 1558 192.168.129.1 192.168.129.2 init
May 19 05:42:57 openvpn[4565]: UDPv4 link local (bound): [AF_INET][REDACTED]:1194
May 19 05:42:57 openvpn[4565]: UDPv4 link remote: [undef]
May 19 05:42:57 openvpn[4565]: Initialization Sequence Completed

It looks like OpenVPN is crashing and restarting… which would explain the lack of throughput and the inability to ping the pfSense server from remote.

Is there a configuration I can modify to correct this problem?
Thanks,
-nb

[NetBandit]

Since this thread is still the top result on searching for double wan bandwidth, I'm posting another pic from my 2.2.2 64bit system.
It appears that this problem has been re-identified… but just for posterity and clarity, see the attached pic.