IPv6 packet loss on host machine
-
Hi,
I have my pfSense installed as a VM on Linux KVM and everything is working fine. I have configured IPv6 through a HE.net tunnel and works like a charm from any computer on my network or from the pfSense box itself.
But apparently something is not fine on the host itself, as the machine hosting the pfSense VM experiences packet loss, but ONLY when using IPv6 (so it's not a hardware problem).
I've read this about this very same problem https://github.com/projectcalico/felix/issues/14 and seems to be exactly my problem, but I do not have any clue on how to solve this. They've mentioned a fix to dnsmasq in the post, but I think it's a change to the dnsmasq code itself.
I Have made a test like this:
ping6 ipv6.google.com -c 1000
and this the "packet loss" stats…. (made with the script found on the post I've mentioned before)
At 9, missed 8 packets At 18, missed 8 packets At 33, missed 8 packets At 47, missed 9 packets At 62, missed 8 packets At 82, missed 18 packets At 98, missed 8 packets At 107, missed 8 packets At 116, missed 8 packets At 125, missed 8 packets At 141, missed 8 packets At 147, missed 1 packets At 177, missed 27 packets At 192, missed 8 packets At 208, missed 8 packets At 217, missed 8 packets At 234, missed 8 packets At 239, missed 1 packets At 249, missed 8 packets At 259, missed 8 packets At 275, missed 9 packets At 285, missed 9 packets At 301, missed 9 packets At 311, missed 9 packets At 321, missed 9 packets At 336, missed 8 packets At 380, missed 36 packets At 396, missed 8 packets At 412, missed 8 packets At 428, missed 8 packets At 444, missed 8 packets At 459, missed 8 packets At 470, missed 9 packets At 485, missed 9 packets At 501, missed 9 packets At 518, missed 8 packets At 534, missed 8 packets At 560, missed 18 packets At 564, missed 1 packets At 577, missed 8 packets At 588, missed 9 packets At 604, missed 8 packets At 615, missed 9 packets At 631, missed 9 packets At 647, missed 8 packets At 663, missed 8 packets At 673, missed 8 packets At 684, missed 9 packets At 700, missed 8 packets At 709, missed 8 packets At 725, missed 8 packets At 741, missed 8 packets At 750, missed 8 packets At 755, missed 1 packets At 768, missed 8 packets At 777, missed 8 packets At 788, missed 9 packets At 803, missed 9 packets At 819, missed 8 packets At 836, missed 8 packets At 852, missed 8 packets At 869, missed 9 packets At 878, missed 8 packets At 887, missed 8 packets At 905, missed 8 packets At 918, missed 8 packets At 953, missed 27 packets At 962, missed 8 packets At 971, missed 8 packets At 989, missed 17 packets At 998, missed 8 packets
Have anyone of you faced this problem and know how to solve it?
Thanks,
Pablo -
Sorry to spam, but nobody experienced this problem?, perhaps have some wrong config of my VMs…. I have used bridge networking... is that ok?
-
No, but then I don't run pfSense in a virtual machine.
–- ipv6.google.com ping statistics ---
1000 packets transmitted, 1000 received, 0% packet loss, time 1000244ms -
I run pfsense on VM (esxi 6.5) I run HE.. never seen this problem…
Just ran you ping -c 1000 test..
That fix was done back in 2014.. And you are NOT seeing anything like the problem they discussed which is miss a few packets.. At a repeated interval..
Your test doesn't seem valid for counting your packet loss
At 989, missed 17 packets
At 998, missed 8 packetsHow do you have 17 packets missing at 989 if your only counting to 1000, and then 8 at 998... I didn't want to wait so long so set 200ms between pings..
64 bytes from ord37s09-in-x0e.1e100.net: icmp_seq=996 ttl=56 time=11.2 ms
64 bytes from ord37s09-in-x0e.1e100.net: icmp_seq=997 ttl=56 time=12.8 ms
64 bytes from ord37s09-in-x0e.1e100.net: icmp_seq=998 ttl=56 time=12.8 ms
64 bytes from ord37s09-in-x0e.1e100.net: icmp_seq=999 ttl=56 time=15.0 ms
64 bytes from ord37s09-in-x0e.1e100.net: icmp_seq=1000 ttl=56 time=11.6 ms--- ipv6.google.com ping statistics ---
1000 packets transmitted, 1000 received, 0% packet loss, time 200513ms
rtt min/avg/max/mdev = 10.568/15.925/353.653/17.353 ms, pipe 2 -
Thanks for your reply, perhaps the script is not counting correctly, but the problem still exists, have a look at this ping output:
PING ipv6.google.com(atl14s80-in-x0e.1e100.net) 56 data bytes 64 bytes from atl14s80-in-x0e.1e100.net: icmp_seq=19 ttl=56 time=160 ms 64 bytes from atl14s80-in-x0e.1e100.net: icmp_seq=20 ttl=56 time=160 ms 64 bytes from atl14s80-in-x0e.1e100.net: icmp_seq=21 ttl=56 time=160 ms 64 bytes from atl14s80-in-x0e.1e100.net: icmp_seq=22 ttl=56 time=160 ms 64 bytes from atl14s80-in-x0e.1e100.net: icmp_seq=32 ttl=56 time=159 ms 64 bytes from atl14s80-in-x0e.1e100.net: icmp_seq=51 ttl=56 time=159 ms 64 bytes from atl14s80-in-x0e.1e100.net: icmp_seq=52 ttl=56 time=159 ms 64 bytes from atl14s80-in-x0e.1e100.net: icmp_seq=53 ttl=56 time=160 ms 64 bytes from atl14s80-in-x0e.1e100.net: icmp_seq=54 ttl=56 time=160 ms 64 bytes from atl14s80-in-x0e.1e100.net: icmp_seq=55 ttl=56 time=167 ms 64 bytes from atl14s80-in-x0e.1e100.net: icmp_seq=56 ttl=56 time=159 ms 64 bytes from atl14s80-in-x0e.1e100.net: icmp_seq=57 ttl=56 time=160 ms 64 bytes from atl14s80-in-x0e.1e100.net: icmp_seq=58 ttl=56 time=160 ms 64 bytes from atl14s80-in-x0e.1e100.net: icmp_seq=59 ttl=56 time=160 ms 64 bytes from atl14s80-in-x0e.1e100.net: icmp_seq=68 ttl=56 time=159 ms 64 bytes from atl14s80-in-x0e.1e100.net: icmp_seq=69 ttl=56 time=160 ms 64 bytes from atl14s80-in-x0e.1e100.net: icmp_seq=70 ttl=56 time=160 ms 64 bytes from atl14s80-in-x0e.1e100.net: icmp_seq=71 ttl=56 time=160 ms 64 bytes from atl14s80-in-x0e.1e100.net: icmp_seq=80 ttl=56 time=159 ms 64 bytes from atl14s80-in-x0e.1e100.net: icmp_seq=89 ttl=56 time=159 ms 64 bytes from atl14s80-in-x0e.1e100.net: icmp_seq=98 ttl=56 time=161 ms 64 bytes from atl14s80-in-x0e.1e100.net: icmp_seq=99 ttl=56 time=160 ms 64 bytes from atl14s80-in-x0e.1e100.net: icmp_seq=108 ttl=56 time=159 ms 64 bytes from atl14s80-in-x0e.1e100.net: icmp_seq=109 ttl=56 time=159 ms ^C --- ipv6.google.com ping statistics --- 109 packets transmitted, 24 received, 77% packet loss, time 108327ms rtt min/avg/max/mdev = 159.288/160.524/167.526/1.647 ms
I'm not using esxi so perhaps this is not a problem when using esxi but only when using KVM, I'm really clueless about how I can solve this….
The problem shows only in the host machine, as any other machines on the network works without missing any packets, also other VMs on the same machine and on the same NICs works ok with zero packet loss.
Any kernel parameters, KVM configurations...?
-
There is a sticky about KVM and pfsense.. Suggest you go through that.
https://forum.pfsense.org/index.php?topic=88467.0
You have no such issues with ipv4?
-
There is a sticky about KVM and pfsense.. Suggest you go through that.
https://forum.pfsense.org/index.php?topic=88467.0
You have no such issues with ipv4?
I've experienced that problem some time ago and done the offloading "trick", but that seems to be another different problem as that is with IPv4 and on the other guest machines.
My guests VMs are working perfectly on both IPv6 and IPv4, only host machine is having problems with IPv6.
-
How exactly are you bringing this HE tunnel into the host if the tunnel endpoints on your VM?
-
How exactly are you bringing this HE tunnel into the host if the tunnel endpoints on your VM?
I'm not sure what your question is, but the tunnel is from and to the pfSense VM (followed HE intructions), and the host is just another machine on the network as the other VMs or "real" machines that work ok.
-
The host is not another machine on the network.. IT has a interface - that is most likely sharing the connection?
Please explain how your physically connected and how you have your virtual network setup.. My esxi host gets IPv6 connectivity to the internet through my pfsense VM, which uses an HE tunnel. But it gets this connectivity through its vmkern interface that is connected to the vswitch that is the "lan" of pfsense via physical interface that is connected to the same network… So when my esxi host wants to talk to the internet be it ipv4 or ipv6 it goes out its vmkern interface that goes out the phy nic to the phy switch. Then back in from different port on the switch to different phy nic to the "lan" vswitch that is connected to pfsense lan then out the HE tunnel..
So vmkern or the host is connected via a phy port on a switch, that is connected to a different phy port on the host that is pfsense "lan" then in the host it routes via the pfsense vm that is also connected to a different phy nic and then connected to the modem and then the internet. See attached if that helps.. So what I am asking is how exactly does your "host" have connectivity to your pfsense vm that hosts the HE tunnel.
-
The host is not another machine on the network.. IT has a interface - that is most likely sharing the connection?
Please explain how your physically connected and how you have your virtual network setup…
Ok, sorry for being so "short" in my answer. I'll try to be as detailed as possible.
You're right, the host is not like other machines. My host have 3 "real" NICs, one for LAN and two for two ISPs (one cable and one aDSL). Those 3 NICs are defined as bridges on my host machine:
pablot@deathstar2:~$ cat /etc/network/interfaces # This file describes the network interfaces available on your system # and how to activate them. For more information, see interfaces(5). source /etc/network/interfaces.d/* # The loopback network interface auto lo iface lo inet loopback # El bridge es ahora la interface principal par ala lan auto br0 iface br0 inet static address 192.168.2.1 netmask 255.255.255.0 network 192.168.2.0 broadcast 192.168.2.255 gateway 192.168.2.13 bridge_ports enp2s0 bridge_stp off bridge_fd 0 bridge_maxwait 5 # dns-* options are implemented by the resolvconf package, if installed dns-nameservers 192.168.2.13 8.8.8.8 dns-search localdomain auto br1 iface br1 inet manual bridge_ports enp4s0 bridge_stp off bridge_fd 0 bridge_maxwait 0 auto br2 iface br2 inet manual bridge_ports enp5s0 bridge_stp off bridge_fd 0 bridge_maxwait 0
br0 -> LAN 192.168.2.1: connected to a Netgear R7000 with Xwrt firmware. Ethernet clients are connected to its 4 port switch.
br1 -> cable ISP: connected to a cable router configured as a bridge
br2 -> aDSL ISP: connected to an adsl router configured as a bridgeThis are the interfaces defined on this host:
pablot@deathstar2:~$ ifconfig br0 Link encap:Ethernet HWaddr e0:3f:49:0e:5e:c0 inet addr:192.168.2.1 Bcast:192.168.2.255 Mask:255.255.255.0 inet6 addr: 2001:---------:7346/64 Scope:Global inet6 addr: 2001:---------:5ec0/64 Scope:Global inet6 addr: fe80::e23f:49ff:fe0e:5ec0/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:92808 errors:0 dropped:146 overruns:0 frame:0 TX packets:40968 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:13286698 (13.2 MB) TX bytes:11077212 (11.0 MB) br1 Link encap:Ethernet HWaddr 18:d6:c7:05:13:33 inet6 addr: fe80::1ad6:c7ff:fe05:1333/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:13561 errors:0 dropped:0 overruns:0 frame:0 TX packets:102 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:1986311 (1.9 MB) TX bytes:9164 (9.1 KB) br2 Link encap:Ethernet HWaddr 18:d6:c7:05:b9:99 inet6 addr: 2001:----------:6efc/64 Scope:Global inet6 addr: fe80::1ad6:c7ff:fe05:b999/64 Scope:Link inet6 addr: 2001:----------:b999/64 Scope:Global UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:53687 errors:0 dropped:146 overruns:0 frame:0 TX packets:1090 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:10265924 (10.2 MB) TX bytes:93660 (93.6 KB) enp2s0 Link encap:Ethernet HWaddr e0:3f:49:0e:5e:c0 inet6 addr: fe80::e23f:49ff:fe0e:5ec0/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:9920669 errors:0 dropped:0 overruns:0 frame:0 TX packets:12343885 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:4935037983 (4.9 GB) TX bytes:9476588488 (9.4 GB) enp4s0 Link encap:Ethernet HWaddr 18:d6:c7:05:13:33 inet6 addr: fe80::1ad6:c7ff:fe05:1333/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:9472337 errors:0 dropped:0 overruns:0 frame:0 TX packets:7692095 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:7600631722 (7.6 GB) TX bytes:3626500190 (3.6 GB) enp5s0 Link encap:Ethernet HWaddr 18:d6:c7:05:b9:99 inet6 addr: fe80::1ad6:c7ff:fe05:b999/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:3473420 errors:0 dropped:0 overruns:0 frame:0 TX packets:2616932 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:2311752848 (2.3 GB) TX bytes:1382280998 (1.3 GB) lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:65536 Metric:1 RX packets:541 errors:0 dropped:0 overruns:0 frame:0 TX packets:541 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1 RX bytes:43448 (43.4 KB) TX bytes:43448 (43.4 KB) virbr0 Link encap:Ethernet HWaddr 52:54:00:4f:28:e6 inet addr:192.168.122.1 Bcast:192.168.122.255 Mask:255.255.255.0 UP BROADCAST MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) vnet0 Link encap:Ethernet HWaddr fe:54:00:57:2e:07 inet6 addr: fe80::fc54:ff:fe57:2e07/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:10118 errors:0 dropped:0 overruns:0 frame:0 TX packets:23981 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:1859305 (1.8 MB) TX bytes:5408313 (5.4 MB) vnet1 Link encap:Ethernet HWaddr fe:54:00:0a:74:e6 inet6 addr: fe80::fc54:ff:fe0a:74e6/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:823860 errors:0 dropped:0 overruns:0 frame:0 TX packets:592780 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:901727626 (901.7 MB) TX bytes:89828043 (89.8 MB) vnet2 Link encap:Ethernet HWaddr fe:54:00:3e:1b:ef inet6 addr: fe80::fc54:ff:fe3e:1bef/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:553930 errors:0 dropped:0 overruns:0 frame:0 TX packets:795473 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:82024196 (82.0 MB) TX bytes:827958514 (827.9 MB) vnet3 Link encap:Ethernet HWaddr fe:54:00:15:57:0b inet6 addr: fe80::fc54:ff:fe15:570b/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:177061 errors:0 dropped:0 overruns:0 frame:0 TX packets:190347 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:23176258 (23.1 MB) TX bytes:89195814 (89.1 MB) vnet4 Link encap:Ethernet HWaddr fe:54:00:2f:88:6d inet6 addr: fe80::fc54:ff:fe2f:886d/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:224468 errors:0 dropped:0 overruns:0 frame:0 TX packets:274035 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:25030471 (25.0 MB) TX bytes:173332703 (173.3 MB)
The vnetX interfaces are created by KVM/QEMU to be assigned to the VMs so I have assigned interfaces as this:
vnet0 -> pfSense LAN - IP: 192.168.2.13 vnet2 -> pfSense FIBERTEL (ISP1 cable) vnet3 -> pfSense ARNET (ISP2 aDSL)
(See attached NICs config below - pf1.png and pf2.png)
I also have other VMs wich also use the br0 bridge through the host device vnet0 and works perfectly both on IPv4 and IPv6 (see attached NIC config below - ubuntu.png)
My HE tunnel goes through the cable ISP (will config another tunnel through the aDSL ISP in the future)
My host gets IPv6 connectivity to the internet through my pfsense VM, which uses an HE tunnel. It gets it's connectivity through the br0 interface. That same phisical connection is also the the LAN of pfSense, as the vtnet0 is assigned to the LAN interface on pfSense VM.
(I'm attaching my network diagram, there are more things, like VLANs, etc.)
![Network pablot - VLANs.png](/public/imported_attachments/1/Network pablot - VLANs.png)
![Network pablot - VLANs.png_thumb](/public/imported_attachments/1/Network pablot - VLANs.png_thumb) -
Both of bridge interfaces on your host have ipv6 on them.. So why would it be going through pfsense HE tunnel??
-
Both of bridge interfaces on your host have ipv6 on them.. So why would it be going through pfsense HE tunnel??
I'm sorry, but is that wrong?, do I have to remove ipv6 addresses from them?. I guess that the br2 address is not supposed to be there, but what about br0? is that wrong?
-
I've checked disabling completely IPv6 on the br1 and br2 interfaces and still teh same problem, so I'm not sure what else to do.