Slow internet with pfsense



  • Hi,

    Do not know if it's the right category, please move if not :)

    I run pfSense virtual on a XenServer and 1Gbit internet and internet is slow.
    It's a little different too how slow it is, but between 10 and 50 MBits down, upload speed is always good at 1gbit.

    If I rebooted pfSense internet is about 1Gbit/1Gbit for about 5 minutes until it becomes slow again. Reboot and its fast again for 5 minutes.

    What I've done with XenServer for both WAN and all LAN:

    xe vif-param-set uuid=VIFUUID other-config:ethtool-tx="off"
    xe vif-param-set uuid=VIFUUID other-config:ethtool-rx="off"

    pkg install xe-guest-utilities

    echo "xenguest_enable="YES"" >> /etc/rc.conf.local
    ln -s /usr/local/etc/rc.d/xenguest /usr/local/etc/rc.d/xenguest.sh

    service xenguest start

    What i have done with pfSense:

    • Diabled all DHCP Servers

    • Created 4 Lan

    • Under Wan, IPv6 configuration type = none

    • Under wan, IPv3 Configuration type = static IPv4

    • Under wan, added Virtual IP/32

    • Under wan, selected gateway

    • Under system->routing, addd gateway default and checked: Use non-local gateway

    • Under Firewall rules, copied LAN default rule to every interface

    • Under Firewall rules, removed ipv6 rule

    • Under all 4 LAN interfaces deactivated ipv6

    • Under DNS Resolver, deaktivated

    • Under DNS Forwarder, activated and selected all lan under interfaces

    • Under Firewall Virtual IP, added 25 Virtual IPs

    • Under advanced ->network: Disabled Hardware Checksum Offloading

    And I know that there issnt something wrong with my Internet line as if I try to add a virtual IP on another vm and then internet is perfect.

    I also tried the same setup with vmware on another server, had the same problem then.

    Here is OVH network configuration guide:
    http://help.ovh.com/BridgeClient



  • I'd really appreciate some help, have tried to fix this for 3 months now.

    Tell If any screenshots or more info any tests I can do.



  • i have 1 interface for every LAN and i have testing copying a file of 1gb from vlan to vlan and i get speed of 200mb pr sek. so i belive network is ok?



  • Do you see any errors with tcpdump, or in the firewall logs? And what about iperf tests?



  • I cant se any errors…

    iperf also gets good speed:
    [2.3.2-RELEASE][root@fw-pfSense-01.localdomain]/root: iperf -c ping.online.net
    –----------------------------------------------------------
    Client connecting to ping.online.net, TCP port 5001
    TCP window size: 65.0 KByte (default)

    [  3] local 87.98.128.127 port 5109 connected with 62.210.18.40 port 5001
    [ ID] Interval      Transfer    Bandwidth
    [  3]  0.0-10.0 sec  654 MBytes  548 Mbits/sec

    And from a vm behind pfsense:
    ^Croot@isp1:~# iperf -c ping.online.net
    –----------------------------------------------------------
    Client connecting to ping.online.net, TCP port 5001
    TCP window size: 85.0 KByte (default)

    [  3] local 192.168.99.101 port 45274 connected with 62.210.18.40 port 5001
    [ ID] Interval      Transfer    Bandwidth
    [  3]  0.0-10.0 sec  1.09 GBytes  935 Mbits/sec

    So i get good speed with iperf. but when i try to download http://proof.ovh.net/files/10Gb.dat i get only get between 10 and 100mbit pr sec



  • also when downloading http://ping.online.net/10000Mo.dat i only get 20mbit



  • How big is the TCP window size and how big is the MTU during HTTP?



  • I do not know what MTU is? And TCP window size: 85.0 KByte (default) ?



  • What is the current VIFUUID of your VM interface



  • This is the list:

    uuid ( RO)                  : 5c40178e-8998-dce9-b2d0-4311a03cdfb0
            vm-name-label ( RO): fw-pfSense-01
                    device ( RO): 0
                      MAC ( RO): 00:50:56:0b:f2:ae
              network-uuid ( RO): fbbe8cbd-b8c7-961f-34cc-94d890116d65
        network-name-label ( RO): WAN

    uuid ( RO)                  : 5b921ac5-e297-60dd-83dd-3607f053f2fb
            vm-name-label ( RO): fw-pfSense-01
                    device ( RO): 4
                      MAC ( RO): 0a:87:13:7d:bb:eb
              network-uuid ( RO): ce166d3a-613a-b364-174f-95f47496b284
        network-name-label ( RO): ISP

    uuid ( RO)                  : d3679b9b-4960-7ec8-76d7-48fb7e2fcf22
            vm-name-label ( RO): fw-pfSense-01
                    device ( RO): 2
                      MAC ( RO): a6:0c:62:9d:4b:06
              network-uuid ( RO): 426b2541-0140-41dc-ac92-1fb55cecf6d0
        network-name-label ( RO): BACKUP

    uuid ( RO)                  : 78c01939-9395-b040-d6ba-ef81c9e51195
            vm-name-label ( RO): fw-pfSense-01
                    device ( RO): 1
                      MAC ( RO): 9a:b7:cc:e6:eb:26
              network-uuid ( RO): 26c32f02-31d4-ab1f-a6d6-83a823ce8607
        network-name-label ( RO): ADMIN

    uuid ( RO)                  : 9b4da5fe-06d2-8b2f-08f5-e531a6e209cf
            vm-name-label ( RO): fw-pfSense-01
                    device ( RO): 3
                      MAC ( RO): 4e:20:0d:fa:c7:1e
              network-uuid ( RO): 84dfa988-3c82-1832-1343-3bcc02944eb5
        network-name-label ( RO): HOME

    And also On host server:

    [root@ns3044318 tmp]# wget -O /dev/null http://proof.ovh.net/files/1Gio.dat
    –2016-11-23 20:15:57--  http://proof.ovh.net/files/1Gio.dat
    Resolving proof.ovh.net... 188.165.12.106, 2001:41d0:2:876a::1
    Connecting to proof.ovh.net|188.165.12.106|:80... connected.
    HTTP request sent, awaiting response... 200 OK
    Length: 1073741824 (1.0G) [application/octet-stream]
    Saving to: `/dev/null'

    100%[=================================================================================================================================================================================================>] 1,073,741,824  112M/s  in 13s

    2016-11-23 20:16:10 (79.2 MB/s) - `/dev/null' saved [1073741824/1073741824]

    On host behind pfsense:

    administrator@isp1:~$  wget -O /dev/null http://proof.ovh.net/files/1Gio.dat
    –2016-11-23 20:16:56--  http://proof.ovh.net/files/1Gio.dat
    Slår opp vertsnavn proof.ovh.net (proof.ovh.net) … 188.165.12.106, 2001:41d0:2:876a::1
    Kobler til proof.ovh.net (proof.ovh.net)|188.165.12.106|:80 … tilkoblet.
    HTTP-forespørsel sendt. Venter på svar … 200 OK
    Lengde: 1073741824 (1,0G) [application/octet-stream]
    Lagrer til: «/dev/null»

    /dev/null                                                  100%[========================================================================================================================================>]  1,00G  5,07MB/s    om 3m 42s



  • Strange, it looks like packets are discarded or the MTU/Window/Fragmentation sizes are all wrong to cause this. Is MTR installed? Try MTR'ing to that DL server. Oh, and if you simply ping for 100 times, how are the latency/drops?



  • Running mtr  -w -c 10 -i 1 ping.online.net:

    Start: Mon Nov 28 20:05:08 2016
    HOST: fw-pfSense-01.localdomain          Loss%  Snt  Last  Avg  Best  Wrst StDev
      1.|– 51.255.92.253                      0.0%    10  180.6 182.0 165.6 194.7  9.2
      2.|-- po110.gra-g2-a75.fr.eu              0.0%    10    0.3  0.3  0.3  0.4  0.0
      3.|-- ???                                100.0    10    0.0  0.0  0.0  0.0  0.0
      4.|-- be99-1110.th2-1-a9.fr.eu            0.0%    10    5.5  5.0  4.7  5.5  0.0
      5.|-- ???                                100.0    10    0.0  0.0  0.0  0.0  0.0
      6.|-- 45x-s44-2-a9k2.dc3.poneytelecom.eu  0.0%    10    6.9  6.1  5.4  9.4  0.9
      7.|-- ping.online.net                    0.0%    10    4.9  5.0  4.9  5.4  0.0

    Host Server

    –- www.google.com ping statistics ---
    592 packets transmitted, 592 received, 0% packet loss, time 591891ms
    rtt min/avg/max/mdev = 4.495/4.598/9.068/0.238 ms

    Local VM behind pfsense

    –- www.google.com ping statistics ---
    619 packets transmitted, 614 packets received, 0.8% packet loss
    round-trip min/avg/max/stddev = 4.561/4.739/11.052/0.336 ms

    sometimes i get alot of pageloss on pfsense and 0 on the host



  • Sounds like you may still be having non-ICMP traffic issues, maybe your offloading settings still aren't right. Can you print the ethtool output on the hypervisor side for the VIF's?



  • i am not sure what command i have to run to find this:

    but i run this script right now and it did not help
    https://github.com/cloudnull/XenServer-Offloading-Off/blob/master/offloadingoff.sh



  • I also have to say i had the same issue on vmware



  • Have you tried pfSense 2.4 Beta just to see if the newer FreeBSD base makes a difference?



  • Trying to set up tomorrow. but do you think it might be a problem with pfSense or configuration error or problem with OVH?



  • @telvenes:

    Trying to set up tomorrow. but do you think it might be a problem with pfSense or configuration error or problem with OVH?

    Others have a working setup on OVH, but it's somewhat tricky with FreeBSD on mass-virtualisation, so it could be an issue with the underlying platform. It's hard to say at this point, but everything still points towards packets being discarded.



  • i am now running on 2.4, was thinking if i should try without xen-tools first and se how that goes.



  • Did not help. installed xen tools and still same error. could it bee a config error?



  • I have read over the whole internet on guides to get this working.

    And almost everywhere i see this lines:

    route add -net 2.3.4.254/32 -iface vmx1
    route add default 2.3.4.254

    I do not use them, i have only added default gateway and activated gateway outside subnet, is this correct?



  • @telvenes:

    I have read over the whole internet on guides to get this working.

    And almost everywhere i see this lines:

    route add -net 2.3.4.254/32 -iface vmx1
    route add default 2.3.4.254

    I do not use them, i have only added default gateway and activated gateway outside subnet, is this correct?

    No, that's a somewhat random suggestion and probably has nothing to do with your network.
    If you have vmx interfaces, they are not VT-d PCI interfaces and you may end up with the checksum bug as described in the sticky'ed topic (see the main virt forum).



  • no they are xn, but i dont know what more i can check to get this working correctly.

    Anyway, if i reboot my pfsense internet is fast for 10 minutes or something before it goes slower.



  • Did some pacage filtering for 10 seconds while trying to download a file, can anyone see something wrong?

    17:23:19.255593 IP MYIP.3389 > 109.247.144.79.65168: tcp 565
    17:23:19.258132 IP 109.247.144.79.65168 > MYIP.3389: tcp 0
    17:23:19.266947 IP 109.247.144.79.65168 > MYIP.3389: tcp 117
    17:23:19.267531 IP 191.181.216.88.4243 > 149.202.114.11.23: tcp 0
    17:23:19.286667 IP MYIP.3389 > 109.247.144.79.65168: tcp 0
    17:23:19.286736 IP MYIP.3389 > 109.247.144.79.65168: tcp 85
    17:23:19.305069 IP 109.247.144.79.65168 > MYIP.3389: tcp 117
    17:23:19.331952 IP6 fe80::2ff:ffff:feff:fffd > ff02::1:ff00:0: ICMP6, neighbor solicitation, who has 2001:41d0:1004:20a7::, length 32
    17:23:19.333436 IP MYIP.3389 > 109.247.144.79.65168: tcp 0
    17:23:19.338452 IP MYIP2 > 51.255.92.254: ICMP echo request, id 14876, seq 22429, length 8
    17:23:19.340620 IP 51.255.92.254 > MYIP2: ICMP echo reply, id 14876, seq 22429, length 8
    17:23:19.349256 IP MYIP.3389 > 109.247.144.79.65168: tcp 101
    17:23:19.349268 IP MYIP.3389 > 109.247.144.79.65168: tcp 1460
    17:23:19.349271 IP MYIP.3389 > 109.247.144.79.65168: tcp 241
    17:23:19.349274 IP MYIP.3389 > 109.247.144.79.65168: tcp 1460
    17:23:19.349278 IP MYIP.3389 > 109.247.144.79.65168: tcp 241
    17:23:19.349287 IP MYIP.3389 > 109.247.144.79.65168: tcp 1460
    17:23:19.349290 IP MYIP.3389 > 109.247.144.79.65168: tcp 97
    17:23:19.349365 IP MYIP.3389 > 109.247.144.79.65168: tcp 1460
    17:23:19.349370 IP MYIP.3389 > 109.247.144.79.65168: tcp 241
    17:23:19.349395 IP MYIP.3389 > 109.247.144.79.65168: tcp 1460
    17:23:19.349411 IP MYIP.3389 > 109.247.144.79.65168: tcp 241
    17:23:19.349450 IP MYIP.3389 > 109.247.144.79.65168: tcp 1460
    17:23:19.349456 IP MYIP.3389 > 109.247.144.79.65168: tcp 241
    17:23:19.349516 IP MYIP.3389 > 109.247.144.79.65168: tcp 1460
    17:23:19.349522 IP MYIP.3389 > 109.247.144.79.65168: tcp 241
    17:23:19.349557 IP MYIP.3389 > 109.247.144.79.65168: tcp 1460
    17:23:19.349563 IP MYIP.3389 > 109.247.144.79.65168: tcp 241
    17:23:19.349609 IP MYIP.3389 > 109.247.144.79.65168: tcp 1460
    17:23:19.349616 IP MYIP.3389 > 109.247.144.79.65168: tcp 241
    17:23:19.349620 IP MYIP.3389 > 109.247.144.79.65168: tcp 485
    17:23:19.349683 IP MYIP.3389 > 109.247.144.79.65168: tcp 1460
    17:23:19.349688 IP MYIP.3389 > 109.247.144.79.65168: tcp 241
    17:23:19.349718 IP MYIP.3389 > 109.247.144.79.65168: tcp 101
    17:23:19.349770 IP MYIP.3389 > 109.247.144.79.65168: tcp 1460
    17:23:19.349776 IP MYIP.3389 > 109.247.144.79.65168: tcp 241
    17:23:19.349796 IP MYIP.3389 > 109.247.144.79.65168: tcp 1460
    17:23:19.349801 IP MYIP.3389 > 109.247.144.79.65168: tcp 81
    17:23:19.349814 IP MYIP.3389 > 109.247.144.79.65168: tcp 101
    17:23:19.363405 IP 109.247.144.79.65168 > MYIP.3389: tcp 0
    17:23:19.395919 IP MYIP.3389 > 109.247.144.79.65168: tcp 853
    17:23:19.400052 IP 109.247.144.79.65168 > MYIP.3389: tcp 0
    17:23:19.400921 IP 109.247.144.79.65168 > MYIP.3389: tcp 0
    17:23:19.403148 IP 109.247.144.79.65168 > MYIP.3389: tcp 0
    17:23:19.403156 IP 109.247.144.79.65168 > MYIP.3389: tcp 0
    17:23:19.403195 IP 109.247.144.79.65168 > MYIP.3389: tcp 0
    17:23:19.403202 IP 109.247.144.79.65168 > MYIP.3389: tcp 0
    17:23:19.422843 IP 109.247.144.79.65168 > MYIP.3389: tcp 0
    17:23:19.423399 IP 109.247.144.79.65168 > MYIP.3389: tcp 0
    17:23:19.423408 IP 109.247.144.79.65168 > MYIP.3389: tcp 0
    17:23:19.423413 IP 109.247.144.79.65168 > MYIP.3389: tcp 0
    17:23:19.423416 IP 109.247.144.79.65168 > MYIP.3389: tcp 0
    17:23:19.423452 IP 109.247.144.79.65168 > MYIP.3389: tcp 0
    17:23:19.423485 IP 109.247.144.79.65168 > MYIP.3389: tcp 0
    17:23:19.423493 IP 109.247.144.79.65168 > MYIP.3389: tcp 0
    17:23:19.427759 IP 109.247.144.79.65168 > MYIP.3389: tcp 101
    17:23:19.427827 IP 109.247.144.79.65168 > MYIP.3389: tcp 117
    17:23:19.427944 IP MYIP.3389 > 109.247.144.79.65168: tcp 0
    17:23:19.436158 IP 109.247.144.79.65168 > MYIP.3389: tcp 117
    17:23:19.442717 IP MYIP.3389 > 109.247.144.79.65168: tcp 1077
    17:23:19.472448 IP 109.247.144.79.65168 > MYIP.3389: tcp 0
    17:23:19.489523 IP MYIP.3389 > 109.247.144.79.65168: tcp 645
    17:23:19.518900 IP 109.247.144.79.65168 > MYIP.3389: tcp 0
    17:23:19.565971 IP 109.247.144.79.65168 > MYIP.3389: tcp 0
    17:23:19.578572 IP 92.222.185.1 > 149.202.114.9: ICMP echo request, id 22793, seq 1, length 12
    17:23:19.618643 ARP, Request who-has 51.255.92.251 tell 51.255.92.253, length 46
    17:23:19.675378 ARP, Request who-has 51.255.92.246 tell 51.255.92.253, length 46
    17:23:19.832092 ARP, Request who-has 51.255.92.246 tell 51.255.92.253, length 46
    17:23:19.840443 IP MYIP2 > 51.255.92.254: ICMP echo request, id 14876, seq 22430, length 8
    17:23:19.844237 IP 51.255.92.254 > MYIP2: ICMP echo reply, id 14876, seq 22430, length 8
    17:23:20.037704 ARP, Request who-has 51.255.92.242 tell 51.255.92.253, length 46
    17:23:20.061806 IP 51.255.92.253.1985 > 224.0.0.2.1985: UDP, length 20
    17:23:20.075734 ARP, Request who-has 51.255.92.1 tell 51.255.92.253, length 46
    17:23:20.115356 ARP, Request who-has 51.255.92.249 tell 51.255.92.253, length 46
    17:23:20.264026 IP 92.222.184.1 > 149.202.114.13: ICMP echo request, id 50886, seq 1, length 12
    17:23:20.264177 IP 149.202.114.13 > 92.222.184.1: ICMP echo reply, id 50886, seq 1, length 12
    17:23:20.276486 IP 92.222.184.1 > 149.202.114.8: ICMP echo request, id 50886, seq 1, length 12
    17:23:20.310848 IP6 fe80::2ff:ffff:feff:fffe > ff02::1:ff00:0: ICMP6, neighbor solicitation, who has 2001:41d0:1004:20a5::, length 32
    17:23:20.342422 IP MYIP2 > 51.255.92.254: ICMP echo request, id 14876, seq 22431, length 8
    17:23:20.344993 IP 51.255.92.254 > MYIP2: ICMP echo reply, id 14876, seq 22431, length 8
    17:23:20.381787 IP 109.247.144.79.65168 > MYIP.3389: tcp 85
    17:23:20.389901 IP 109.247.144.79.65168 > MYIP.3389: tcp 85
    17:23:20.390052 IP MYIP.3389 > 109.247.144.79.65168: tcp 0
    17:23:20.394264 IP MYIP.3389 > 109.247.144.79.65168: tcp 85
    17:23:20.397723 IP 109.247.144.79.65168 > MYIP.3389: tcp 85
    17:23:20.409800 IP MYIP.3389 > 109.247.144.79.65168: tcp 0
    17:23:20.413599 IP 109.247.144.79.65168 > MYIP.3389: tcp 101
    17:23:20.429884 IP 109.247.144.79.65168 > MYIP.3389: tcp 101
    17:23:20.429979 IP MYIP.3389 > 109.247.144.79.65168: tcp 0
    17:23:20.445900 IP 109.247.144.79.65168 > MYIP.3389: tcp 85
    17:23:20.463383 IP 109.247.144.79.65168 > MYIP.3389: tcp 101
    17:23:20.463520 IP MYIP.3389 > 109.247.144.79.65168: tcp 0
    17:23:20.477848 IP 109.247.144.79.65168 > MYIP.3389: tcp 101
    17:23:20.487830 IP MYIP.3389 > 109.247.144.79.65168: tcp 85
    17:23:20.493827 IP 109.247.144.79.65168 > MYIP.3389: tcp 101
    17:23:20.509964 IP 109.247.144.79.65168 > MYIP.3389: tcp 101
    17:23:20.510076 IP MYIP.3389 > 109.247.144.79.65168: tcp 0
    17:23:20.525818 IP 109.247.144.79.65168 > MYIP.3389: tcp 101
    17:23:20.527893 ARP, Request who-has 51.255.92.246 tell 51.255.92.253, length 46
    17:23:20.538488 ARP, Request who-has 51.255.92.251 tell 51.255.92.253, length 46



  • I have contacted the internet provider, but its i think its strange if its something wrong with the internet? since it works ok with E.G Windows?