Lagg + vlans
-
Hi,
Today i was reinstalling my pfsense with the latest version (pfSense-2.0-BETA1-20091227-1613.iso).
Im currently 4 hours into the process.I can get the machine up and running,
i can assign a single interface and get dhcp and working routing.
But when i put my intel dual gbit nic in an lagg and use vlans it won't work.
This setup was running previously (i must add that was with freebsd 7 build)When i remove the vlans, and tell my switch to use LACP on 2 ports and untagged in an vlan everything works fine again.
Except i need the vlans.I cannot get any error message to appear, i checked the switch (says lacp is ok), the syslog which 1 time said that ' lagg0_vlan1' does not exist. but could not get the system to repeat that. And the firewall logs, which show some blocked traffic from the lagg0_vlan1 interface but i cannot generate these errors by trying to ping pfsense or something like that.
Ok, i got some errors out of it:
Dec 28 16:26:54 kernel: em2: link state changed to DOWN Dec 28 16:26:54 kernel: em3: link state changed to DOWN Dec 28 16:26:54 kernel: lagg0: link state changed to DOWN Dec 28 16:26:54 kernel: lagg0_vlan1: link state changed to DOWN Dec 28 16:26:54 kernel: lagg0_vlan2: link state changed to DOWN Dec 28 16:26:54 kernel: lagg0_vlan3: link state changed to DOWN Dec 28 16:27:17 kernel: em2: link state changed to UP Dec 28 16:27:17 kernel: lagg0: link state changed to UP Dec 28 16:27:17 kernel: lagg0_vlan1: link state changed to UP Dec 28 16:27:17 kernel: lagg0_vlan2: link state changed to UP Dec 28 16:27:17 kernel: lagg0_vlan3: link state changed to UP Dec 28 16:27:17 kernel: em3: link state changed to UP Dec 28 16:27:20 check_reload_status: rc.linkup starting Dec 28 16:27:21 php: : Hotplug event detected for opt1 but ignoring since interface is configured with static IP (172.16.10.254) Dec 28 16:27:52 kernel: ifa_del_loopback_route: deletion failed Dec 28 16:27:56 check_reload_status: syncing firewall Dec 28 16:28:10 login: login on ttyv0 as root Dec 28 16:28:02 check_reload_status: syncing firewall Dec 28 16:31:43 php: /interfaces_vlan_edit.php: The command '/sbin/ifconfig lagg0 vlanhwtag' returned exit code '1', the output was 'ifconfig: vlanhwtag: Invalid argument' Dec 28 16:31:43 php: /interfaces_vlan_edit.php: The command '/sbin/ifconfig lagg0 vlanmtu' returned exit code '1', the output was 'ifconfig: vlanmtu: Invalid argument' Dec 28 16:31:43 php: /interfaces_vlan_edit.php: The command '/sbin/ifconfig lagg0 vlanhwfilter' returned exit code '1', the output was 'ifconfig: vlanhwfilter: Invalid argument' Dec 28 16:31:43 kernel: vlan0: changing name to 'lagg0_vlan2' Dec 28 16:31:48 check_reload_status: syncing firewall Dec 28 16:32:17 php: /interfaces_vlan_edit.php: The command '/sbin/ifconfig lagg0 vlanhwtag' returned exit code '1', the output was 'ifconfig: vlanhwtag: Invalid argument' Dec 28 16:32:17 php: /interfaces_vlan_edit.php: The command '/sbin/ifconfig lagg0 vlanmtu' returned exit code '1', the output was 'ifconfig: vlanmtu: Invalid argument' Dec 28 16:32:17 php: /interfaces_vlan_edit.php: The command '/sbin/ifconfig lagg0 vlanhwfilter' returned exit code '1', the output was 'ifconfig: vlanhwfilter: Invalid argument' Dec 28 16:32:18 kernel: vlan1: changing name to 'lagg0_vlan1' Dec 28 16:32:18 check_reload_status: syncing firewall Dec 28 16:33:00 php: /interfaces_assign.php: SYSTEM: We do not have a gateways array in our XML. Is this configuration damaged? Dec 28 16:33:02 dnsmasq[15737]: exiting on receipt of SIGTERM Dec 28 16:33:03 dnsmasq[19891]: started, version 2.51 cachesize 10000 Dec 28 16:33:03 dnsmasq[19891]: compile time options: no-IPv6 GNU-getopt no-DBus I18N DHCP TFTP Dec 28 16:33:03 dnsmasq[19891]: reading /etc/resolv.conf Dec 28 16:33:03 dnsmasq[19891]: using nameserver 62.212.65.123#53 Dec 28 16:33:03 dnsmasq[19891]: using nameserver 83.149.80.123#53 Dec 28 16:33:03 dnsmasq[19891]: using nameserver 85.17.96.69#53 Dec 28 16:33:03 dnsmasq[19891]: read /etc/hosts - 2 addresses Dec 28 16:33:04 dhcpd: Internet Systems Consortium DHCP Server V3.0.7 Dec 28 16:33:04 dhcpd: Copyright 2004-2008 Internet Systems Consortium. Dec 28 16:33:04 dhcpd: All rights reserved. Dec 28 16:33:04 dhcpd: For info, please visit http://www.isc.org/sw/dhcp/ Dec 28 16:33:04 dhcpd: Multiple interfaces match the same subnet: lagg0 lagg0_vlan1 Dec 28 16:33:04 dhcpd: Multiple interfaces match the same shared network: lagg0 lagg0_vlan1 Dec 28 16:33:04 php: /interfaces_assign.php: Creating rrd update script Dec 28 16:33:04 check_reload_status: reloading filter Dec 28 16:33:05 check_reload_status: syncing firewall Dec 28 16:33:06 check_reload_status: reloading interfaces Dec 28 16:33:07 php: : WARNING: Recursive call to interfaces sync! Dec 28 16:33:12 check_reload_status: reloading filter Dec 28 16:33:13 check_reload_status: syncing firewallI produced them by adding and removing clans to lagg0
I hope someone can tell me whats going on :-)
Greetings, Marcus
-
Provide ifconfig command output, netstat -rn output.
Your config.xml and your system log. -
Ok,
I created OPT1 for my laptop to stay in touch with pfsense when i make the vlan1 on lagg0.
You requested:
ifconfig
# ifconfig em0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500 options=9b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum>ether 00:0b:db:90:e6:6f inet6 fe80::20b:dbff:fe90:e66f%em0 prefixlen 64 scopeid 0x1 inet 82.192.91.115 netmask 0xffffff00 broadcast 82.192.91.255 inet 172.16.1.6 netmask 0xfffffff8 broadcast 172.16.1.7 media: Ethernet autoselect (100baseTX <full-duplex>) status: active em1: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500 options=9b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum>ether 00:0b:db:90:e6:70 inet6 fe80::20b:dbff:fe90:e670%em1 prefixlen 64 scopeid 0x2 inet 172.16.50.254 netmask 0xffffff00 broadcast 172.16.50.255 media: Ethernet autoselect (1000baseT <full-duplex>) status: active em2: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500 options=9b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum>ether 00:02:a5:4e:ac:ba inet6 fe80::202:a5ff:fe4e:acba%em2 prefixlen 64 scopeid 0x3 media: Ethernet autoselect (1000baseT <full-duplex>) status: active em3: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500 options=9b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum>ether 00:02:a5:4e:ac:ba inet6 fe80::202:a5ff:fe4e:acbb%em3 prefixlen 64 scopeid 0x4 media: Ethernet autoselect (1000baseT <full-duplex>) status: active lo0: flags=8049 <up,loopback,running,multicast>metric 0 mtu 16384 options=3 <rxcsum,txcsum>inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5 inet 127.0.0.1 netmask 0xff000000 pfsync0: flags=0<> metric 0 mtu 1460 syncpeer: 224.0.0.240 maxupd: 128 pflog0: flags=100 <promisc>metric 0 mtu 33200 enc0: flags=0<> metric 0 mtu 1536 lagg0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500 options=9b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum>ether 00:02:a5:4e:ac:ba inet6 fe80::202:a5ff:fe4e:acba%lagg0 prefixlen 64 scopeid 0x9 inet 172.16.10.254 netmask 0xffffff00 broadcast 172.16.10.255 media: Ethernet autoselect status: active laggproto lacp laggport: em3 flags=1c <active,collecting,distributing>laggport: em2 flags=1c <active,collecting,distributing>lagg0_vlan1: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500 options=3 <rxcsum,txcsum>ether 00:02:a5:4e:ac:ba inet6 fe80::20b:dbff:fe90:e66f%lagg0_vlan1 prefixlen 64 scopeid 0xa inet 172.16.10.254 netmask 0xffffff00 broadcast 172.16.10.255 media: Ethernet autoselect status: active vlan: 1 parent interface: lagg0 lagg0_vlan2: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500 options=3 <rxcsum,txcsum>ether 00:02:a5:4e:ac:ba inet6 fe80::20b:dbff:fe90:e66f%lagg0_vlan2 prefixlen 64 scopeid 0xb media: Ethernet autoselect status: active vlan: 2 parent interface: lagg0 #</rxcsum,txcsum></up,broadcast,running,simplex,multicast></rxcsum,txcsum></up,broadcast,running,simplex,multicast></active,collecting,distributing></active,collecting,distributing></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum></up,broadcast,running,simplex,multicast></promisc></rxcsum,txcsum></up,loopback,running,multicast></full-duplex></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum></up,broadcast,running,simplex,multicast></full-duplex></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum></up,broadcast,running,simplex,multicast></full-duplex></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum></up,broadcast,running,simplex,multicast></full-duplex></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum></up,broadcast,running,simplex,multicast>netstat -rn
# netstat -rn Routing tables Internet: Destination Gateway Flags Refs Use Netif Expire default 82.192.91.254 UGS 1 16263 em0 82.192.91.0/24 link#1 U 0 3 em0 82.192.91.115 link#1 UHS 0 0 lo0 127.0.0.1 link#5 UH 0 0 lo0 127.0.0.2 127.0.0.1 UHS 0 0 lo0 172.16.1.0/29 link#1 U 0 0 em0 172.16.1.6 link#1 UHS 0 0 lo0 172.16.10.0/24 link#9 U 2 34591 lagg0 172.16.10.254 link#10 UHS 0 0 lo0 172.16.50.0/24 link#2 U 2 4999 em1 172.16.50.254 link#2 UHS 0 0 lo0 Internet6: Destination Gateway Flags Netif Expire ::1 ::1 UH lo0 fe80::%em0/64 link#1 U em0 fe80::20b:dbff:fe90:e66f%em0 link#1 UHS lo0 fe80::%em1/64 link#2 U em1 fe80::20b:dbff:fe90:e670%em1 link#2 UHS lo0 fe80::%em2/64 link#3 U em2 fe80::202:a5ff:fe4e:acba%em2 link#3 UHS lo0 fe80::%em3/64 link#4 U em3 fe80::202:a5ff:fe4e:acbb%em3 link#4 UHS lo0 fe80::%lo0/64 link#5 U lo0 fe80::1%lo0 link#5 UHS lo0 fe80::%lagg0/64 link#9 U lagg0 fe80::202:a5ff:fe4e:acba%lagg0 link#9 UHS lo0 fe80::%lagg0_vlan1/64 link#10 U lagg0_vl fe80::20b:dbff:fe90:e66f%lagg0_vlan1 link#10 UHS lo0 fe80::%lagg0_vlan2/64 link#11 U lagg0_vl fe80::20b:dbff:fe90:e66f%lagg0_vlan2 link#11 UHS lo0 ff01:1::/32 fe80::20b:dbff:fe90:e66f%em0 U em0 ff01:2::/32 fe80::20b:dbff:fe90:e670%em1 U em1 ff01:3::/32 fe80::202:a5ff:fe4e:acba%em2 U em2 ff01:4::/32 fe80::202:a5ff:fe4e:acbb%em3 U em3 ff01:5::/32 ::1 U lo0 ff01:9::/32 fe80::202:a5ff:fe4e:acba%lagg0 U lagg0 ff01:a::/32 fe80::20b:dbff:fe90:e66f%lagg0_vlan1 U lagg0_vl ff01:b::/32 fe80::20b:dbff:fe90:e66f%lagg0_vlan2 U lagg0_vl ff02::%em0/32 fe80::20b:dbff:fe90:e66f%em0 U em0 ff02::%em1/32 fe80::20b:dbff:fe90:e670%em1 U em1 ff02::%em2/32 fe80::202:a5ff:fe4e:acba%em2 U em2 ff02::%em3/32 fe80::202:a5ff:fe4e:acbb%em3 U em3 ff02::%lo0/32 ::1 U lo0 ff02::%lagg0/32 fe80::202:a5ff:fe4e:acba%lagg0 U lagg0 ff02::%lagg0_vlan1/32 fe80::20b:dbff:fe90:e66f%lagg0_vlan1 U lagg0_vl ff02::%lagg0_vlan2/32 fe80::20b:dbff:fe90:e66f%lagg0_vlan2 U lagg0_vl #[edit] One second, ill attack the config and syslog. I had them pasted here, but they are gone now. [/edit]
[edit2]
Ok here they are, they are to large to attach, ill need to link:
http://dump.marcusvandam.nl/ph!135/pfsense/config.xml
http://dump.marcusvandam.nl/ph!135/pfsense/system.log
[/edit2]I hope this helps.
Hoping to hear from you guys, and kind regards.
Marcus van Dam -
Dec 28 18:51:16 pfsense dhcpd: Multiple interfaces match the same subnet: lagg0 lagg0_vlan1 Dec 28 18:51:16 pfsense dhcpd: Multiple interfaces match the same shared network: lagg0 lagg0_vlan1This seems to be your problem.
-
And how can that be?
If you check my config, you will see that i switched it over to lagg0_vlan1. lagg0 is not used in this situation.Greets, Marcus
-
Looks like when you switched off lagg0 it didn't unassign that IP, I'll check into that. If you reboot is everything brought up correctly?
-
No, i am afraid not.
It didn't after my upgrade and didn't after a fresh install. -
I just tried to add a vlan on lagg0, no dhcp this time.
Just a vlan, added an interface gave it the 172.16.20.254 ip.
Now i still cant ping that address form a machine within that vlan.
And again i get errors in syslog about ifconfig, and an interesting one, the mac change. the machine in vlan2 has 2 interfaces and 2 ips on the vlan, and pfsense notices the machine but does not respond to his ping. nor routes its traffic.I have here the syslog part and the netstat -rn:
Syslog:
Jan 1 17:01:45 pfsense php: /interfaces_vlan_edit.php: The command '/sbin/ifconfig lagg0 vlanhwtag' returned exit code '1', the output was 'ifconfig: vlanhwtag: Invalid argument' Jan 1 17:01:45 pfsense php: /interfaces_vlan_edit.php: The command '/sbin/ifconfig lagg0 vlanmtu' returned exit code '1', the output was 'ifconfig: vlanmtu: Invalid argument' Jan 1 17:01:45 pfsense php: /interfaces_vlan_edit.php: The command '/sbin/ifconfig lagg0 vlanhwfilter' returned exit code '1', the output was 'ifconfig: vlanhwfilter: Invalid argument' Jan 1 17:01:46 pfsense kernel: vlan0: changing name to 'lagg0_vlan2' Jan 1 17:01:48 pfsense check_reload_status: syncing firewall Jan 1 17:02:08 pfsense check_reload_status: syncing firewall Jan 1 17:02:10 pfsense php: /interfaces_assign.php: Creating rrd update script Jan 1 17:02:14 pfsense check_reload_status: syncing firewall Jan 1 17:02:20 pfsense check_reload_status: reloading filter Jan 1 17:02:21 pfsense check_reload_status: syncing firewall Jan 1 17:02:51 pfsense check_reload_status: syncing firewall Jan 1 17:02:53 pfsense apinger: Exiting on signal 15. Jan 1 17:02:53 pfsense apinger: Starting Alarm Pinger, apinger(32229) Jan 1 17:02:53 pfsense apinger: No usable targets found, exiting Jan 1 17:02:53 pfsense php: /interfaces.php: Creating rrd update script Jan 1 17:02:57 pfsense check_reload_status: reloading filter Jan 1 17:05:28 pfsense check_reload_status: syncing firewall Jan 1 17:05:34 pfsense check_reload_status: reloading filter Jan 1 17:07:23 pfsense kernel: arp: 172.16.20.161 moved from 00:0c:29:88:6a:e0 to 00:0c:29:88:6a:d6 on lagg0_vlan2 Jan 1 17:07:48 pfsense sshd[34134]: Accepted keyboard-interactive/pam for root from 172.16.10.96 port 55486 ssh2netstat -rn:
Routing tables Internet: Destination Gateway Flags Refs Use Netif Expire default 82.192.91.254 UGS 2 24884628 em0 82.192.91.0/24 link#1 U 0 337691 em0 82.192.91.115 link#1 UHS 0 0 lo0 127.0.0.1 link#5 UH 0 18 lo0 127.0.0.2 127.0.0.1 UHS 0 0 lo0 172.16.1.0/29 link#1 U 0 2 em0 172.16.1.6 link#1 UHS 0 0 lo0 172.16.10.0/24 link#9 U 5 42901917 lagg0 172.16.10.254 link#9 UHS 0 0 lo0 172.16.20.0/24 link#10 U 0 0 lagg0_ 172.16.20.254 link#10 UHS 0 0 lo0 172.16.50.0/24 link#2 U 0 1562 em1 172.16.50.254 link#2 UHS 0 0 lo0 Internet6: Destination Gateway Flags Netif Expire ::1 ::1 UH lo0 fe80::%em0/64 link#1 U em0 fe80::20b:dbff:fe90:e66f%em0 link#1 UHS lo0 fe80::%em1/64 link#2 U em1 fe80::20b:dbff:fe90:e670%em1 link#2 UHS lo0 fe80::%em2/64 link#3 U em2 fe80::202:a5ff:fe4e:acba%em2 link#3 UHS lo0 fe80::%em3/64 link#4 U em3 fe80::202:a5ff:fe4e:acbb%em3 link#4 UHS lo0 fe80::%lo0/64 link#5 U lo0 fe80::1%lo0 link#5 UHS lo0 fe80::%lagg0/64 link#9 U lagg0 fe80::202:a5ff:fe4e:acba%lagg0 link#9 UHS lo0 fe80::%lagg0_vlan2/64 link#10 U lagg0_vl fe80::20b:dbff:fe90:e66f%lagg0_vlan2 link#10 UHS lo0 ff01:1::/32 fe80::20b:dbff:fe90:e66f%em0 U em0 ff01:2::/32 fe80::20b:dbff:fe90:e670%em1 U em1 ff01:3::/32 fe80::202:a5ff:fe4e:acba%em2 U em2 ff01:4::/32 fe80::202:a5ff:fe4e:acbb%em3 U em3 ff01:5::/32 ::1 U lo0 ff01:9::/32 fe80::202:a5ff:fe4e:acba%lagg0 U lagg0 ff01:a::/32 fe80::20b:dbff:fe90:e66f%lagg0_vlan2 U lagg0_vl ff02::%em0/32 fe80::20b:dbff:fe90:e66f%em0 U em0 ff02::%em1/32 fe80::20b:dbff:fe90:e670%em1 U em1 ff02::%em2/32 fe80::202:a5ff:fe4e:acba%em2 U em2 ff02::%em3/32 fe80::202:a5ff:fe4e:acbb%em3 U em3 ff02::%lo0/32 ::1 U lo0 ff02::%lagg0/32 fe80::202:a5ff:fe4e:acba%lagg0 U lagg0 ff02::%lagg0_vlan2/32 fe80::20b:dbff:fe90:e66f%lagg0_vlan2 U lagg0_vl[edit]
the following line bothers me:172.16.20.0/24 link#10 U 0 0 lagg0_Shouldn't it be somethink more then lagg_ ?
[/edit]I hope this sheds more light on the problem and someone can help me :-)
Kind regards, Marcus
-
I know…. I should not bump.
But now the vacation is over. I would bring the topic under the attention of the experts.
I am trying t provide as much info as possible. I cannot be the only one with the problem ???Kind Regards, Marcus van Dam
-
See the same here. So I made a bug report http://redmine.pfsense.org/issues/show/296#change-829
-
Can you please edit interfaces.inc in /etc/inc and remove a line that has hwvlantag in it?!
It should fix your problem.
-
Well, not really.
I can create the vlan, and it shows up in ifconfig,
But when i try to add an OPT2 interface and assign it to the vlan the webinterface hangs, the console still works until you press enter. You can press 8 to start and shell, but as soon as you run "ifconfig" the process freezes, you have to reboot with CRTL-ALT-DEL.I hope this helps.
-
I also found some interesting links about this problem:
Same problem but then in the freebsd treads
http://forums.freebsd.org/showthread.php?t=7668Kernel bug that might explain it al
http://www.freebsd.org/cgi/query-pr.cgi?pr=141646 -
@ermal:
Can you please edit interfaces.inc in /etc/inc and remove a line that has hwvlantag in it?!
It should fix your problem.
Deleting the line does not change anything, hw tagging are still shown on ifconfig. If I do ifconfig emX -vlanhwtag for every em nic I can get as fare as receiving a IP on my static failover trunk but even pinging the gateway fails.
As DJ wrote freezing often happens when assigning nics. -
I cannot seem to get an interface assigned with vlanhwtag turned off. It worked when turned on.
Now it just crashes on every commando you give the machine after assigning. -
Is everyone here trying using Intel em cards? Curious what driver(s) have issues, as that seems to be the cause.
I did get lagg + VLAN trunk working fine with fxp cards, haven't had a chance to test em yet but will, as well as a few other NICs I have available.
-
@cmb:
Is everyone here trying using Intel em cards? Curious what driver(s) have issues, as that seems to be the cause.
I did get lagg + VLAN trunk working fine with fxp cards, haven't had a chance to test em yet but will, as well as a few other NICs I have available.
I am using the Intel
PRO/1000 MT Dual Port Server Adapter (http://www.intel.com/products/server/adapters/pro1000mt-dualport/pro1000mt-dualport-overview.htm) in an Dell 1650 machine.
unfortunately i do not have a second setup to test with different cards.It seems to be related to the em(4) driver.