Unable to add more vlans, but don't know why

frater

I have a multi LAN / multi WAN setup in a campus network where I have 5 VLANs for my WAN-connections and 51 VLANs for my LAN-connections.
Recently I wanted to add a house which needs 2 additional VLANs, but when I add them, my pfsense becomes unavailable…

I don't know the exact reason for this.
Just before making this post I needed to know how many VLANs there were in my system, so I used this command

[2.1-RC0][root@/root(10): ifconfig | grep em1_vlan | grep mtu | wc -l
       5
[2.1-RC0][root@]/root(11): ifconfig | grep em0_vlan | grep mtu | wc -l
      51

But when I do this:
[2.1-RC0][root@]/root(12): ifconfig | grep em0_vlan | grep mtu

em0_vlan100: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
em0_vlan101: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
em0_vlan102: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
em0_vlan103: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
em0_vlan3: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
em0_vlan200: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
em0_vlan104: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
em0_vlan105: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
em0_vlan106: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
em0_vlan150: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
em0_vlan151: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
em0_vlan152: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
em0_vlan153: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
em0_vlan154: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
em0_vlan155: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
em0_vlan180: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
em0_vlan181: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
em0_vlan182: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
em0_vlan183: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
em0_vlan184: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
em0_vlan185: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
em0_vlan210: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
em0_vlan211: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
em0_vlan212: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
em0_vlan213: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
em0_vlan110: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
em0_vlan111: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
em0_vlan112: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
em0_vlan113: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
em0_vlan114: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
em0_vlan115: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
em0_vlan107: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
em0_vlan120: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
em0_vlan121: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
em0_vlan122: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
em0_vlan123: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
em0_vlan124: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
em0_vlan125: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
em0_vlan130: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
em0_vlan131: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
em0_vlan132: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
em0_vlan133: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
em0_vlan134: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
em0_vlan135: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
em0_vlan136: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
em0_vlan140: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
em0_vlan141: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
em0_vlan142: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
em0_vlan143: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
em0_vlan108: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
em0_vlan109: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500</up,broadcast,running,simplex,multicast></up,broadcast,running,simplex,multicast></up,broadcast,running,simplex,multicast></up,broadcast,running,simplex,multicast></up,broadcast,running,simplex,multicast></up,broadcast,running,simplex,multicast></up,broadcast,running,simplex,multicast></up,broadcast,running,simplex,multicast></up,broadcast,running,simplex,multicast></up,broadcast,running,simplex,multicast></up,broadcast,running,simplex,multicast></up,broadcast,running,simplex,multicast></up,broadcast,running,simplex,multicast></up,broadcast,running,simplex,multicast></up,broadcast,running,simplex,multicast></up,broadcast,running,simplex,multicast></up,broadcast,running,simplex,multicast></up,broadcast,running,simplex,multicast></up,broadcast,running,simplex,multicast></up,broadcast,running,promisc,simplex,multicast></up,broadcast,running,promisc,simplex,multicast></up,broadcast,running,promisc,simplex,multicast></up,broadcast,running,promisc,simplex,multicast></up,broadcast,running,promisc,simplex,multicast></up,broadcast,running,promisc,simplex,multicast></up,broadcast,running,promisc,simplex,multicast></up,broadcast,running,promisc,simplex,multicast></up,broadcast,running,promisc,simplex,multicast></up,broadcast,running,promisc,simplex,multicast></up,broadcast,running,promisc,simplex,multicast></up,broadcast,running,promisc,simplex,multicast></up,broadcast,running,promisc,simplex,multicast></up,broadcast,running,promisc,simplex,multicast></up,broadcast,running,promisc,simplex,multicast></up,broadcast,running,promisc,simplex,multicast></up,broadcast,running,promisc,simplex,multicast></up,broadcast,running,promisc,simplex,multicast></up,broadcast,running,promisc,simplex,multicast></up,broadcast,running,promisc,simplex,multicast></up,broadcast,running,promisc,simplex,multicast></up,broadcast,running,promisc,simplex,multicast></up,broadcast,running,promisc,simplex,multicast></up,broadcast,running,promisc,simplex,multicast></up,broadcast,running,promisc,simplex,multicast></up,broadcast,running,promisc,simplex,multicast></up,broadcast,running,promisc,simplex,multicast></up,broadcast,running,promisc,simplex,multicast></up,broadcast,running,promisc,simplex,multicast></up,broadcast,running,promisc,simplex,multicast></up,broadcast,running,promisc,simplex,multicast></up,broadcast,running,promisc,simplex,multicast>

I notice that some have PROMISCUOUS and some don't…
I have no idea why, but it could be a lead...

Because my VLANs wouldn't work otherwise, I do have this cronjob running

grep vlanhw /etc/crontab
*/3     *       *       *       *       root    ifconfig em0 | grep -q VLAN_HWTAG && ifconfig em0 -vlanhwtag
*/3     *       *       *       *       root    ifconfig em1 | grep -q VLAN_HWTAG && ifconfig em1 -vlanhwtag

dhatz

@frater:

I notice that some have PROMISCUOUS and some don't…
I have no idea why, but it could be a lead...

Are you running any other software (e.g. packages) besides stock 2.1 ?
E.g. snort does put in promiscuous mode the interface(s) it listens on …

Are you using any "advanced" config, e.g. lagg or bridging ?

frater

arpwatch maybe?

I've also been thinking about my problem with the 2 Intel cards…
No-one else is reporting any problems, but probably no-one else is using that many VLANS and if they do they probably have other NICs

Recently I had to install Windows Server on an Intel Desktop board and noticed there were no drivers for that specific network card.
It turned out Intel deliberately isn't giving those drivers because those NICs are not fully capable and it has something to do with VLANs (or maybe HyperV).
I was able to get that board going by patching an inf-file and loading the drivers of that same family.

When one machine suddenly broke down a year ago we replaced it with an Intel Atom D2500CC motherboard with 2 NICs.
We bought 2 of them to make sure we have a spare when 1 breaks down.
A working config is stored on a remote server every hour so we can bring that machine quickly up to date..

I think there's something wrong with those NICs and that's why I need to run "ifconfig em0 -vlanhwtag" and "ifconfig em1 -vlanhwtag"
It's possible of course this error only occurs after loading up more than, let's say, 15 VLANs

I asked for a setting in pfsense to disable "vlan hardware tagging".
That way the setting is only applied.
It would probably make system workable....

mikeisfly

What kind of switch are you using? Even though there is a possible 4096 possible vans some switches don't allow anymore than 32. Do you have any switching loops?

cmb

@frater:

No-one else is reporting any problems, but probably no-one else is using that many VLANS and if they do they probably have other NICs

A lot of people run a lot more VLANs than that, many of them on em NICs. The fact you have a hack in there turning off vlanhwtag makes me think something in that specific implementation of the NIC is broken. I can't recall seeing any em cards with problems with VLAN hardware tagging and I probably touch a dozen diff customer boxes every week running multiple VLANs on em NICs. If they're onboard NICs, a BIOS update would be a good idea.

frater

After one of my NIC died in my previous pfsense setup and we quickly needed to replace this device.

I thought I made a good choice by taking 2 Intel D2500CC with dual NIC and an Atom processor.
If one device would break down I could quickly replace it with the other.
Every hour a backup is made of the config on a remote server using curl.

When I installed pfsense for the first time I noticed my working config wasn't working (I already replaced the system temporarily with a PIII-machine).
Finally I could get it to work by disabling the VLAN_HWTAG

Because the whole system is operational and I can't afford much down time it's hard to do experiments on it.
It's also difficult to create a 2nd setup due to the complex LAN behind it.

Only because I'm using Zabbix to check if all the nodes on my LAN are pingable I'm able to see if the system is working properly.

If you have any influence on the development I would really like to have that option to disable VLAN_HWTAG.
I put that command in a cronjob because I'm afraid the VLAN_HWTAG comes back after I change something in my VLAN setup.

BTW… When I chose to create that many VLANs for my LAN I read that it's probably not that wise to do so.

@mikeisfly

I'm using a Netgear GS724Tv3 as my main switch and about 8 Netgear GS108T
If I knew there were loops I would of course eliminate them.
The whole setup is rock solid as long as I don't add or remove interfaces....

frater

I don't think it's the amount, really…
Yesterday I removed a VLAN which I wasn't using. The system has been stable since.
Today I added a VLAN and added a new interface...

Immediately some of the interfaces didn't came up...
By going into the webif and saving those interfaces I could get those missing back....
Maybe it's some performance problem and pfsense is too fast in adding all these interfaces????

*** Welcome to pfSense 2.1-RC0-pfSense (i386) on pfsense ***

 WAN1_DSD_VLAN10 (wan) -> em1_vlan10 -> v4: 89.250.179.117/24
 LAN_VLAN100_PORT3 (lan) -> em0_vlan100 -> v4: 10.0.0.138/24
 LAN0_VLAN1 (opt1) -> em0        -> v4: 10.250.250.129/25
 WAN0_VLAN1 (opt2) -> em1        -> v4: 10.250.250.1/25
 LAN_VLAN101_PORT4 (opt3) -> em0_vlan101 -> v4: 10.0.101.1/24
 LAN_VLAN102 (opt4) -> em0_vlan102 -> v4: 10.0.102.1/24
 WAN2_XS4ALL_VLAN11_PORT5 (opt5) -> em1_vlan11 -> v4: 192.168.188.10/24
 LAN_VLAN120 (opt6) -> em0_vlan120 -> v4: 10.0.120.1/24
 WAN4_DSD_ODILE_VLAN13 (opt7) -> em1_vlan13 -> v4/DHCP4: 89.250.180.164/24
 WAN5_SCARLET_VLAN14 (opt8) -> em1_vlan14 -> v4: 192.168.168.10/24
 LAN_VLAN103 (opt10) -> em0_vlan103 -> v4: 10.0.103.1/24
 LAN_VLAN200 (opt11) -> em0_vlan200 -> v4: 10.0.200.1/24
 LAN_VLAN104 (opt12) -> em0_vlan104 -> v4: 10.0.104.1/24
 LAN_VLAN105 (opt13) -> em0_vlan105 -> v4: 10.0.105.1/24
 LAN_VLAN106 (opt14) -> em0_vlan106 -> v4: 10.0.106.1/24
 LAN_VLAN150 (opt15) -> em0_vlan150 -> v4: 10.0.150.1/24
 LAN_VLAN151 (opt16) -> em0_vlan151 -> v4: 10.0.151.1/24
 LAN_VLAN152 (opt17) -> em0_vlan152 -> v4: 10.0.152.1/24
 LAN_VLAN153 (opt18) -> em0_vlan153 -> v4: 10.0.153.1/24
 LAN_VLAN154 (opt19) -> em0_vlan154 -> v4: 10.0.154.1/24
 LAN_VLAN155 (opt20) -> em0_vlan155 -> v4: 10.0.155.1/24
 LAN_VLAN180 (opt21) -> em0_vlan180 -> v4: 10.0.180.1/24
 LAN_VLAN181 (opt22) -> em0_vlan181 -> v4: 10.0.181.1/24
 LAN_VLAN182 (opt23) -> em0_vlan182 -> v4: 10.0.182.1/24
 LAN_VLAN183 (opt24) -> em0_vlan183 ->
 LAN_VLAN184 (opt25) -> em0_vlan184 -> v4: 10.0.184.1/24
 LAN_VLAN185 (opt26) -> em0_vlan185 -> v4: 10.0.185.1/24
 LAN_VLAN210 (opt27) -> em0_vlan210 -> v4: 10.0.210.1/24
 LAN_VLAN211 (opt28) -> em0_vlan211 -> v4: 10.0.211.1/24
 LAN_VLAN212 (opt29) -> em0_vlan212 -> v4: 10.0.212.1/24
 LAN_VLAN213 (opt30) -> em0_vlan213 ->
 LAN_VLAN110 (opt31) -> em0_vlan110 -> v4: 10.0.110.1/24
 LAN_VLAN111 (opt32) -> em0_vlan111 -> v4: 10.0.111.1/24
 DDS_1_VLAN12 (opt33) -> em1_vlan12 -> v4/DHCP4: 82.170.121.57/23
 LAN_VLAN113 (opt34) -> em0_vlan113 -> v4: 10.0.113.1/24
 LAN_VLAN114 (opt35) -> em0_vlan114 -> v4: 10.0.114.1/24
 LAN_VLAN115 (opt36) -> em0_vlan115 -> v4: 10.0.115.1/24
 LAN_VLAN112 (opt37) -> em0_vlan112 -> v4: 10.0.112.1/24
 LAN_VLAN107 (opt38) -> em0_vlan107 -> v4: 10.0.107.1/24
 LAN_VLAN121 (opt39) -> em0_vlan121 -> v4: 10.0.121.1/24
 LAN_VLAN122 (opt40) -> em0_vlan122 -> v4: 10.0.122.1/24
 LAN_VLAN123 (opt41) -> em0_vlan123 -> v4: 10.0.123.1/24
 LAN_VLAN124 (opt42) -> em0_vlan124 -> v4: 10.0.124.1/24
 LAN_VLAN224 (opt43) -> em0_vlan224 -> v4: 10.0.224.1/24
 LAN_VLAN130 (opt44) -> em0_vlan130 -> v4: 10.0.130.1/24
 LAN_VLAN131 (opt45) -> em0_vlan131 -> v4: 10.0.131.1/24
 LAN_VLAN132 (opt46) -> em0_vlan132 -> v4: 10.0.132.1/24
 LAN_VLAN133 (opt47) -> em0_vlan133 -> v4: 10.0.133.1/24
 LAN_VLAN134 (opt48) -> em0_vlan134 -> v4: 10.0.134.1/24
 LAN_VLAN135 (opt49) -> em0_vlan135 -> v4: 10.0.135.1/24
 LAN_VLAN136 (opt50) -> em0_vlan136 -> v4: 10.0.136.1/24
 LAN_VLAN140 (opt51) -> em0_vlan140 -> v4: 10.0.140.1/24
 LAN_VLAN141 (opt52) -> em0_vlan141 -> v4: 10.0.141.1/24
 LAN_VLAN142 (opt53) -> em0_vlan142 -> v4: 10.0.142.1/24
 LAN_VLAN143 (opt54) -> em0_vlan143 -> v4: 10.0.143.1/24
 LAN_VLAN108 (opt55) -> em0_vlan108 -> v4: 10.0.108.1/24
 LAN_VLAN109 (opt56) -> em0_vlan109 -> v4: 10.0.109.1/24

mikeisfly

Just researched your switch on www.newegg.com and according to them your switch supports up to 64 static van groups, and 24 port based van groups. So looks like your good there. I also noticed that this switch can support up to 8000 mac addresses. Could this be a limit that you are running up against? Just trying to think of the simplest reason to your problem. If your network is very large this could be an issue, don't know if there is a way to show the mac table on the Netgear that you have. That switch has a gui (web server) built into it I think right?

frater

I just had that problem another time and noticed there are actually IP's missing in several LANs
Going through each LAN using pfsense's WebGUI I am able to enable these again…

frater

@mikeisfly:

Just researched your switch on www.newegg.com and according to them your switch supports up to 64 static van groups, and 24 port based van groups. So looks like your good there. I also noticed that this switch can support up to 8000 mac addresses. Could this be a limit that you are running up against? Just trying to think of the simplest reason to your problem. If your network is very large this could be an issue, don't know if there is a way to show the mac table on the Netgear that you have. That switch has a gui (web server) built into it I think right?

I'm way under the 8000 mac addresses.
At this moment I think I'm just having a problem with rebuilding all the interfaces after a mutation….
It turns out they don't all come back.

I wish I knew which script is doing this and would love to see some extra logging there...

I created a Zabbix-item that will count the amount of valid ipv4-interfaces and I will at least get info from there....

frater

I also added a new Zabbix agent item which counts the interfaces without ipv4
It's not as elegant or sturdy as I want it to be, but let's see if this will give me false positives….

ifconfig | grep -A2 'ether ' | grep -c options
0

In Zabbix Agent, they look like this:

UserParameter=net.ipv4.ifs,ifconfig | egrep -c 'inet .*[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+ '
UserParameter=net.ipv4.noip, ifconfig | grep -A2 'ether ' | grep -c options

I noticed that check.reload.status really spikes my processor and I have more than a minute no connection after I add an interface or vlan…
Sometimes not all the interfaces come back with an IP

It would be nice if this didn't happen in the final version....