IPv6 packet loss on host machine



  • Hi,

    I have my pfSense installed as a VM on Linux KVM and everything is working fine. I have configured IPv6 through a HE.net tunnel and works like a charm from any computer on my network or from the pfSense box itself.

    But apparently something is not fine on the host itself, as the machine hosting the pfSense VM experiences packet loss, but ONLY when using IPv6 (so it's not a hardware problem).

    I've read this about this very same problem https://github.com/projectcalico/felix/issues/14 and seems to be exactly my problem, but I do not have any clue on how to solve this. They've mentioned a fix to dnsmasq in the post, but I think it's a change to the dnsmasq code itself.

    I Have made a test like this:

    ping6 ipv6.google.com -c 1000
    

    and this the "packet loss" stats…. (made with the script found on the post I've mentioned before)

    At 9, missed 8 packets
    At 18, missed 8 packets
    At 33, missed 8 packets
    At 47, missed 9 packets
    At 62, missed 8 packets
    At 82, missed 18 packets
    At 98, missed 8 packets
    At 107, missed 8 packets
    At 116, missed 8 packets
    At 125, missed 8 packets
    At 141, missed 8 packets
    At 147, missed 1 packets
    At 177, missed 27 packets
    At 192, missed 8 packets
    At 208, missed 8 packets
    At 217, missed 8 packets
    At 234, missed 8 packets
    At 239, missed 1 packets
    At 249, missed 8 packets
    At 259, missed 8 packets
    At 275, missed 9 packets
    At 285, missed 9 packets
    At 301, missed 9 packets
    At 311, missed 9 packets
    At 321, missed 9 packets
    At 336, missed 8 packets
    At 380, missed 36 packets
    At 396, missed 8 packets
    At 412, missed 8 packets
    At 428, missed 8 packets
    At 444, missed 8 packets
    At 459, missed 8 packets
    At 470, missed 9 packets
    At 485, missed 9 packets
    At 501, missed 9 packets
    At 518, missed 8 packets
    At 534, missed 8 packets
    At 560, missed 18 packets
    At 564, missed 1 packets
    At 577, missed 8 packets
    At 588, missed 9 packets
    At 604, missed 8 packets
    At 615, missed 9 packets
    At 631, missed 9 packets
    At 647, missed 8 packets
    At 663, missed 8 packets
    At 673, missed 8 packets
    At 684, missed 9 packets
    At 700, missed 8 packets
    At 709, missed 8 packets
    At 725, missed 8 packets
    At 741, missed 8 packets
    At 750, missed 8 packets
    At 755, missed 1 packets
    At 768, missed 8 packets
    At 777, missed 8 packets
    At 788, missed 9 packets
    At 803, missed 9 packets
    At 819, missed 8 packets
    At 836, missed 8 packets
    At 852, missed 8 packets
    At 869, missed 9 packets
    At 878, missed 8 packets
    At 887, missed 8 packets
    At 905, missed 8 packets
    At 918, missed 8 packets
    At 953, missed 27 packets
    At 962, missed 8 packets
    At 971, missed 8 packets
    At 989, missed 17 packets
    At 998, missed 8 packets
    

    Have anyone of you faced this problem and know how to solve it?

    Thanks,
    Pablo



  • Sorry to spam, but nobody experienced this problem?, perhaps have some wrong config of my VMs…. I have used bridge networking... is that ok?



  • No, but then I don't run pfSense in a virtual machine.

    –- ipv6.google.com ping statistics ---
    1000 packets transmitted, 1000 received, 0% packet loss, time 1000244ms


  • LAYER 8 Global Moderator

    I run pfsense on VM (esxi 6.5) I run HE.. never seen this problem…

    Just ran you ping -c 1000 test..

    That fix was done back in 2014..  And you are NOT seeing anything like the problem they discussed which is miss a few packets.. At a repeated interval..

    Your test doesn't seem valid for counting your packet loss

    At 989, missed 17 packets
    At 998, missed 8 packets

    How do you have 17 packets missing at 989 if your only counting to 1000, and then 8 at 998...  I didn't want to wait so long so set 200ms between pings..

    64 bytes from ord37s09-in-x0e.1e100.net: icmp_seq=996 ttl=56 time=11.2 ms
    64 bytes from ord37s09-in-x0e.1e100.net: icmp_seq=997 ttl=56 time=12.8 ms
    64 bytes from ord37s09-in-x0e.1e100.net: icmp_seq=998 ttl=56 time=12.8 ms
    64 bytes from ord37s09-in-x0e.1e100.net: icmp_seq=999 ttl=56 time=15.0 ms
    64 bytes from ord37s09-in-x0e.1e100.net: icmp_seq=1000 ttl=56 time=11.6 ms

    --- ipv6.google.com ping statistics ---
    1000 packets transmitted, 1000 received, 0% packet loss, time 200513ms
    rtt min/avg/max/mdev = 10.568/15.925/353.653/17.353 ms, pipe 2



  • Thanks for your reply, perhaps the script is not counting correctly, but the problem still exists, have a look at this ping output:

    PING ipv6.google.com(atl14s80-in-x0e.1e100.net) 56 data bytes
    64 bytes from atl14s80-in-x0e.1e100.net: icmp_seq=19 ttl=56 time=160 ms
    64 bytes from atl14s80-in-x0e.1e100.net: icmp_seq=20 ttl=56 time=160 ms
    64 bytes from atl14s80-in-x0e.1e100.net: icmp_seq=21 ttl=56 time=160 ms
    64 bytes from atl14s80-in-x0e.1e100.net: icmp_seq=22 ttl=56 time=160 ms
    64 bytes from atl14s80-in-x0e.1e100.net: icmp_seq=32 ttl=56 time=159 ms
    64 bytes from atl14s80-in-x0e.1e100.net: icmp_seq=51 ttl=56 time=159 ms
    64 bytes from atl14s80-in-x0e.1e100.net: icmp_seq=52 ttl=56 time=159 ms
    64 bytes from atl14s80-in-x0e.1e100.net: icmp_seq=53 ttl=56 time=160 ms
    64 bytes from atl14s80-in-x0e.1e100.net: icmp_seq=54 ttl=56 time=160 ms
    64 bytes from atl14s80-in-x0e.1e100.net: icmp_seq=55 ttl=56 time=167 ms
    64 bytes from atl14s80-in-x0e.1e100.net: icmp_seq=56 ttl=56 time=159 ms
    64 bytes from atl14s80-in-x0e.1e100.net: icmp_seq=57 ttl=56 time=160 ms
    64 bytes from atl14s80-in-x0e.1e100.net: icmp_seq=58 ttl=56 time=160 ms
    64 bytes from atl14s80-in-x0e.1e100.net: icmp_seq=59 ttl=56 time=160 ms
    64 bytes from atl14s80-in-x0e.1e100.net: icmp_seq=68 ttl=56 time=159 ms
    64 bytes from atl14s80-in-x0e.1e100.net: icmp_seq=69 ttl=56 time=160 ms
    64 bytes from atl14s80-in-x0e.1e100.net: icmp_seq=70 ttl=56 time=160 ms
    64 bytes from atl14s80-in-x0e.1e100.net: icmp_seq=71 ttl=56 time=160 ms
    64 bytes from atl14s80-in-x0e.1e100.net: icmp_seq=80 ttl=56 time=159 ms
    64 bytes from atl14s80-in-x0e.1e100.net: icmp_seq=89 ttl=56 time=159 ms
    64 bytes from atl14s80-in-x0e.1e100.net: icmp_seq=98 ttl=56 time=161 ms
    64 bytes from atl14s80-in-x0e.1e100.net: icmp_seq=99 ttl=56 time=160 ms
    64 bytes from atl14s80-in-x0e.1e100.net: icmp_seq=108 ttl=56 time=159 ms
    64 bytes from atl14s80-in-x0e.1e100.net: icmp_seq=109 ttl=56 time=159 ms
    ^C
    --- ipv6.google.com ping statistics ---
    109 packets transmitted, 24 received, 77% packet loss, time 108327ms
    rtt min/avg/max/mdev = 159.288/160.524/167.526/1.647 ms
    
    

    I'm not using esxi so perhaps this is not a problem when using esxi but only when using KVM, I'm really clueless about how I can solve this….

    The problem shows only in the host machine, as any other machines on the network works without missing any packets, also other VMs on the same machine and on the same NICs works ok with zero packet loss.

    Any kernel parameters, KVM configurations...?


  • LAYER 8 Global Moderator

    There is a sticky about KVM and pfsense.. Suggest you go through that.

    https://forum.pfsense.org/index.php?topic=88467.0

    You have no such issues with ipv4?



  • @johnpoz:

    There is a sticky about KVM and pfsense.. Suggest you go through that.

    https://forum.pfsense.org/index.php?topic=88467.0

    You have no such issues with ipv4?

    I've experienced that problem some time ago and done the offloading "trick", but that seems to be another different problem as that is with IPv4 and on the other guest machines.

    My guests VMs are working perfectly on both IPv6 and IPv4, only host machine is having problems with IPv6.


  • LAYER 8 Global Moderator

    How exactly are you bringing this HE tunnel into the host if the tunnel endpoints on your VM?



  • @johnpoz:

    How exactly are you bringing this HE tunnel into the host if the tunnel endpoints on your VM?

    I'm not sure what your question is, but the tunnel is from and to the pfSense VM (followed HE intructions), and the host is just another machine on the network as the other VMs or "real" machines that work ok.


  • LAYER 8 Global Moderator

    The host is not another machine on the network.. IT has a interface - that is most likely sharing the connection?

    Please explain how your physically connected and how you have your virtual network setup.. My esxi host gets IPv6 connectivity to the internet through my pfsense VM, which uses an HE tunnel.  But it gets this connectivity through its vmkern interface that is connected to the vswitch that is the "lan" of pfsense via physical interface that is connected to the same network…  So when my esxi host wants to talk to the internet be it ipv4 or ipv6 it goes out its vmkern interface that goes out the phy nic to the phy switch.  Then back in from different port on the switch to different phy nic to the "lan" vswitch that is connected to pfsense lan then out the HE tunnel..

    So vmkern or the host is connected via a phy port on a switch, that is connected to a different phy port on the host that is pfsense "lan" then in the host it routes via the pfsense vm that is also connected to a different phy nic and then connected to the modem and then the internet.  See attached if that helps..  So what I am asking is how exactly does your "host" have connectivity to your pfsense vm that hosts the HE tunnel.




  • @johnpoz:

    The host is not another machine on the network.. IT has a interface - that is most likely sharing the connection?

    Please explain how your physically connected and how you have your virtual network setup…

    Ok, sorry for being so "short" in my answer. I'll try to be as detailed as possible.

    You're right, the host is not like other machines. My host have 3 "real" NICs, one for LAN and two for two ISPs (one cable and one aDSL). Those 3 NICs are defined as bridges on my host machine:

    
    pablot@deathstar2:~$ cat /etc/network/interfaces
    # This file describes the network interfaces available on your system
    # and how to activate them. For more information, see interfaces(5).
    
    source /etc/network/interfaces.d/*
    
    # The loopback network interface
    auto lo
    iface lo inet loopback
    
    # El bridge es ahora la interface principal par ala lan
    auto br0
    iface br0 inet static
            address 192.168.2.1
            netmask 255.255.255.0
            network 192.168.2.0
            broadcast 192.168.2.255
            gateway 192.168.2.13
            bridge_ports enp2s0
            bridge_stp off
            bridge_fd 0
            bridge_maxwait 5
            # dns-* options are implemented by the resolvconf package, if installed
            dns-nameservers 192.168.2.13 8.8.8.8
            dns-search localdomain
    
    auto br1
    iface br1 inet manual
            bridge_ports enp4s0
            bridge_stp off
            bridge_fd 0
            bridge_maxwait 0
    
    auto br2
    iface br2 inet manual
            bridge_ports enp5s0
            bridge_stp off
            bridge_fd 0
            bridge_maxwait 0
    
    

    br0 -> LAN 192.168.2.1: connected to a Netgear R7000 with Xwrt firmware. Ethernet clients are connected to its 4 port switch.
    br1 -> cable ISP: connected to a cable router configured as a bridge
    br2 -> aDSL ISP: connected to an adsl router configured as a bridge

    This are the interfaces defined on this host:

    
    pablot@deathstar2:~$ ifconfig
    br0       Link encap:Ethernet  HWaddr e0:3f:49:0e:5e:c0
              inet addr:192.168.2.1  Bcast:192.168.2.255  Mask:255.255.255.0
              inet6 addr: 2001:---------:7346/64 Scope:Global
              inet6 addr: 2001:---------:5ec0/64 Scope:Global
              inet6 addr: fe80::e23f:49ff:fe0e:5ec0/64 Scope:Link
              UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
              RX packets:92808 errors:0 dropped:146 overruns:0 frame:0
              TX packets:40968 errors:0 dropped:0 overruns:0 carrier:0
              collisions:0 txqueuelen:1000
              RX bytes:13286698 (13.2 MB)  TX bytes:11077212 (11.0 MB)
    
    br1       Link encap:Ethernet  HWaddr 18:d6:c7:05:13:33
              inet6 addr: fe80::1ad6:c7ff:fe05:1333/64 Scope:Link
              UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
              RX packets:13561 errors:0 dropped:0 overruns:0 frame:0
              TX packets:102 errors:0 dropped:0 overruns:0 carrier:0
              collisions:0 txqueuelen:1000
              RX bytes:1986311 (1.9 MB)  TX bytes:9164 (9.1 KB)
    
    br2       Link encap:Ethernet  HWaddr 18:d6:c7:05:b9:99
              inet6 addr: 2001:----------:6efc/64 Scope:Global
              inet6 addr: fe80::1ad6:c7ff:fe05:b999/64 Scope:Link
              inet6 addr: 2001:----------:b999/64 Scope:Global
              UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
              RX packets:53687 errors:0 dropped:146 overruns:0 frame:0
              TX packets:1090 errors:0 dropped:0 overruns:0 carrier:0
              collisions:0 txqueuelen:1000
              RX bytes:10265924 (10.2 MB)  TX bytes:93660 (93.6 KB)
    
    enp2s0    Link encap:Ethernet  HWaddr e0:3f:49:0e:5e:c0
              inet6 addr: fe80::e23f:49ff:fe0e:5ec0/64 Scope:Link
              UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
              RX packets:9920669 errors:0 dropped:0 overruns:0 frame:0
              TX packets:12343885 errors:0 dropped:0 overruns:0 carrier:0
              collisions:0 txqueuelen:1000
              RX bytes:4935037983 (4.9 GB)  TX bytes:9476588488 (9.4 GB)
    
    enp4s0    Link encap:Ethernet  HWaddr 18:d6:c7:05:13:33
              inet6 addr: fe80::1ad6:c7ff:fe05:1333/64 Scope:Link
              UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
              RX packets:9472337 errors:0 dropped:0 overruns:0 frame:0
              TX packets:7692095 errors:0 dropped:0 overruns:0 carrier:0
              collisions:0 txqueuelen:1000
              RX bytes:7600631722 (7.6 GB)  TX bytes:3626500190 (3.6 GB)
    
    enp5s0    Link encap:Ethernet  HWaddr 18:d6:c7:05:b9:99
              inet6 addr: fe80::1ad6:c7ff:fe05:b999/64 Scope:Link
              UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
              RX packets:3473420 errors:0 dropped:0 overruns:0 frame:0
              TX packets:2616932 errors:0 dropped:0 overruns:0 carrier:0
              collisions:0 txqueuelen:1000
              RX bytes:2311752848 (2.3 GB)  TX bytes:1382280998 (1.3 GB)
    
    lo        Link encap:Local Loopback
              inet addr:127.0.0.1  Mask:255.0.0.0
              inet6 addr: ::1/128 Scope:Host
              UP LOOPBACK RUNNING  MTU:65536  Metric:1
              RX packets:541 errors:0 dropped:0 overruns:0 frame:0
              TX packets:541 errors:0 dropped:0 overruns:0 carrier:0
              collisions:0 txqueuelen:1
              RX bytes:43448 (43.4 KB)  TX bytes:43448 (43.4 KB)
    
    virbr0    Link encap:Ethernet  HWaddr 52:54:00:4f:28:e6
              inet addr:192.168.122.1  Bcast:192.168.122.255  Mask:255.255.255.0
              UP BROADCAST MULTICAST  MTU:1500  Metric:1
              RX packets:0 errors:0 dropped:0 overruns:0 frame:0
              TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
              collisions:0 txqueuelen:1000
              RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
    
    vnet0     Link encap:Ethernet  HWaddr fe:54:00:57:2e:07
              inet6 addr: fe80::fc54:ff:fe57:2e07/64 Scope:Link
              UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
              RX packets:10118 errors:0 dropped:0 overruns:0 frame:0
              TX packets:23981 errors:0 dropped:0 overruns:0 carrier:0
              collisions:0 txqueuelen:1000
              RX bytes:1859305 (1.8 MB)  TX bytes:5408313 (5.4 MB)
    
    vnet1     Link encap:Ethernet  HWaddr fe:54:00:0a:74:e6
              inet6 addr: fe80::fc54:ff:fe0a:74e6/64 Scope:Link
              UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
              RX packets:823860 errors:0 dropped:0 overruns:0 frame:0
              TX packets:592780 errors:0 dropped:0 overruns:0 carrier:0
              collisions:0 txqueuelen:1000
              RX bytes:901727626 (901.7 MB)  TX bytes:89828043 (89.8 MB)
    
    vnet2     Link encap:Ethernet  HWaddr fe:54:00:3e:1b:ef
              inet6 addr: fe80::fc54:ff:fe3e:1bef/64 Scope:Link
              UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
              RX packets:553930 errors:0 dropped:0 overruns:0 frame:0
              TX packets:795473 errors:0 dropped:0 overruns:0 carrier:0
              collisions:0 txqueuelen:1000
              RX bytes:82024196 (82.0 MB)  TX bytes:827958514 (827.9 MB)
    
    vnet3     Link encap:Ethernet  HWaddr fe:54:00:15:57:0b
              inet6 addr: fe80::fc54:ff:fe15:570b/64 Scope:Link
              UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
              RX packets:177061 errors:0 dropped:0 overruns:0 frame:0
              TX packets:190347 errors:0 dropped:0 overruns:0 carrier:0
              collisions:0 txqueuelen:1000
              RX bytes:23176258 (23.1 MB)  TX bytes:89195814 (89.1 MB)
    
    vnet4     Link encap:Ethernet  HWaddr fe:54:00:2f:88:6d
              inet6 addr: fe80::fc54:ff:fe2f:886d/64 Scope:Link
              UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
              RX packets:224468 errors:0 dropped:0 overruns:0 frame:0
              TX packets:274035 errors:0 dropped:0 overruns:0 carrier:0
              collisions:0 txqueuelen:1000
              RX bytes:25030471 (25.0 MB)  TX bytes:173332703 (173.3 MB)
    
    

    The vnetX interfaces are created by KVM/QEMU to be assigned to the VMs so I have assigned interfaces as this:

    vnet0 -> pfSense LAN - IP: 192.168.2.13
    vnet2 -> pfSense FIBERTEL  (ISP1 cable)
    vnet3 -> pfSense ARNET (ISP2 aDSL)
    

    (See attached NICs config below - pf1.png and pf2.png)

    I also have other VMs wich also use the br0 bridge through the host device vnet0 and works perfectly both on IPv4 and IPv6 (see attached NIC config below - ubuntu.png)

    My HE tunnel goes through the cable ISP (will config another tunnel through the aDSL ISP in the future)

    My host gets IPv6 connectivity to the internet through my pfsense VM, which uses an HE tunnel. It gets it's connectivity through the br0 interface. That same phisical connection is also the the LAN of pfSense, as the vtnet0 is assigned to the LAN interface on pfSense VM.

    (I'm attaching my network diagram, there are more things, like VLANs, etc.)









    ![Network pablot - VLANs.png](/public/imported_attachments/1/Network pablot - VLANs.png)
    ![Network pablot - VLANs.png_thumb](/public/imported_attachments/1/Network pablot - VLANs.png_thumb)


  • LAYER 8 Global Moderator

    Both of bridge interfaces on your host have ipv6 on them.. So why would it be going through pfsense HE tunnel??



  • @johnpoz:

    Both of bridge interfaces on your host have ipv6 on them.. So why would it be going through pfsense HE tunnel??

    I'm sorry, but is that wrong?, do I have to remove ipv6 addresses from them?. I guess that the br2 address is not supposed to be there, but what about br0? is that wrong?



  • I've checked disabling completely IPv6 on the br1 and br2 interfaces and still teh same problem, so I'm not sure what else to do.


Log in to reply