802.1p/q pfsense setup



  • Hello, I was wondering if anyone had any idea about how to complete any of the following steps on pfsense 2.0.3?

    1. Wan should be on vlan2.
    2. DHCP traffic should have 802.1p bit = 2
    3. IGMP traffic should have 802.1p bit = 6
    4. All other internet traffic 802.1p bit = 3

    Thanks.



  • I'm working on this too.  I'm pretty sure I've got the VLAN side of it figured out.  You probably guessed by my user name. I'm the guy who had this working on his MacBook the other day.  It'll be later tonight before I can take the connection down and test it.  My wife is glued to the TV  :P

    The 802.1p / QoS stuff will be a little less straightforward, but I'll be sure to post up anything I find.  Hopefully someone else can point us in the right direction though.  I'm very much a noob with pfSense.

    I'm really glad you copied that.  Looks like the original post disappeared  :o



  • Yeah, they deleted it lol. I also got the vlan part straight but only get 80-90 down and 10 up without the QoS settings.

    I do not believe there's a way to do it in the webgui, it will probably involve some command line editing.



  • Can't you do your Cos Frame tagging on your switch? What switching platform are you using? As far a VLAN just go to assign under interface and you will see the VLAN tab that is where you can create your VLANs. Once you have the VLANs created then you can assign that VLAN to a interface.



  • I have a Zyxel GS1910-24 switch. I might be able to do it on my switch.



  • Just looked your switch up on New Egg and is does have QoS capabilities. I have no experience with your switch Platform but typically if you are breaking your traffic up on your tagged ports into different Classes then you can give one Class priority over the Other. I believe that is what you are trying to do. CoS is a layer two way to give traffic priority which is what I think you want. PfSense does have QoS capabilities as well but I will let someone who is more knowledgeable in the matter speak on that.  Here http://www.youtube.com/watch?v=EfXImr5q-sw is a video explaining how to setup traffic shaping if you wanted to try to play around with it.



  • I'm feeling like a complete idiot right now.  I can't even get my Watchguard to grab a DHCP address from the network.

    If I put my Macbook on VLAN2, it grabs an IP immediately and I can get out to the net.

    If I put dummy IPs on the Macbook VLAN2 and the WG VLAN2, I can ping from the MacBook to the WG.  Interestingly, I can't ping from the WG to the Macbook.

    I've set my pfSense install back to defaults, I tried setting the MTU to 1496, I've put 'allow any <> any' rules on the WAN interface for both IPv4 and v6, and still no luck.  So I'm dead in the water on testing.

    One thing I did notice when I was messing with the firewall rules.  There's an 802.1p button down near the bottom.  Looks like you could create pass rules that add the 802.1p tags.

    If I can figure out what's up with my DHCP problems I'll get back into this.



  • I don't see any 802.1p settings at the bottom of my firewall rules.



  • I found this image in another, unrelated thread:

    Thread reference: https://forum.pfsense.org/index.php?topic=61002.0

    The above thread basically discusses how it was broken in a previous release.

    If it helps, I'm running 2.1-Release on a Watchguard x5000.    My firewall rule menus look like the ones in the example.  If I were going to try this, I'd set up a pass-all rule for TCP/UDP, and for 802.1p I'd chose match on none and apply CA. (Critical Apps, bit 3)

    I may have found what was broken in my WAN VLAN.  I probably won't be able to test it before tomorrow though.



  • Interesting… 2.0.3 doesn't have that section.



  • Couple of things, remember that most pcs don't deal with tagged traffic. The port going to Pfsense should be tagged with all your vlans. The port going to your mac should be untagged. Some switches due it with the pvid setting others when you assign a vlan to a port make sure its untagged. Lastly I would remind you to make sure you configure dhcp for that vlan.



  • The VLAN'd port is facing the ISP.  The WAN port has to be tagged on VLAN 2 in order for traffic to pass.

    Outgoing traffic to the ISP also needs to have the .1p tags in order to not get dumped into a low speed queue.

    When I talk about testing with my Mac, I'm putting a VLAN on the Thunderbolt GigE interface and plugging it directly into their ONT.



  • I fixed the VLAN and I'm getting out just fine.  I'm pulling ~400 down to Softlayer in Dallas, but uploads are still stuck at 10.

    What's worse is the TV system is not working. The guide is showing, but that could just be cached.  I get a black screen on every channel I try.

    I set up outbound rules from the WAN interface to 'any' to try to apply the tags as provided in the first post.  Nothing seems to help so far.

    I'm starting to wonder if the original info was deleted simply because it was wrong or incomplete, and not because it's some conspiracy to keep 3rd party routers off the network.



  • Still no joy on the uploads.

    I do have some possible insight into the problem with the TV, though I'm no closer to fixing it.  Atlantisman, let me know if you're a TV subscriber or if you're internet-only.  I won't clutter up the thread with TV service details if I'm the only one using it right now.



  • I am also a TV subscriber, and i did notice that if i put the TV equipment behind a different router other than their own that it would just give me black screens. Even if i did this Fiber jack –-> Their router ---> pfsense ---> tv box.



  • I noticed the TV boxes and the storage box send a UDP IPv6 packet to  ff02::1 approximately once per minute.  This is roughly equivalent to IPv4 multicast on 224.0.0.1?  I'm still really green on IPv6.

    Even though my pfSense install had a permit any <> any rule for IPv6, it was still blocking these multicast messages.  I put in a pass rule using the auto-generate tool in the logs.  That let the traffic out, but no replies were coming in.  It seems there's a lot that needs fixing.  This will really test the patience of my wife  ;D



  • Can you guys tunnel your TV service through a vlan on your network keeping the isp router, but then have it supply a public IP to Pfsense so you can use it as your edge router? I think this will give you control over the internet which is what you want and also allow your TV service to work undisturbed. What service are you guys using that you get 400 Mbps down? That is amazing!



  • 400 is slow.  It's supposed to be a gig both directions ;)  Unfortunately, I think my old Watchguard box will be hardware-limited to ~400.  As long as I can fix the upload speeds and get the TV working, I don't really care.  Even 400 is faster than pretty much anything else I can connect to.

    That's not a bad idea on segregating the ISP router. I'm not yet convinced that it's necessary though.  It looks like the TV devices just need to pass certain kinds of IPv6 traffic which pfSense seems to block by default.

    Later this week I'll see about borrowing some hardware from work so I can set up a Wireshark tap between the ONT and router. Then we'll see exactly what's going over the wire.

    I'm also going to set up one of my Adtran routers to do some testing.  I've got a much better understanding of those, and I've got a much easier interface to mess with the .1p tags.



  • You're getting better results than i am without the .1p settings, max i have seen is 85/10, and i know my pfsense build can support the whole gig.


  • Netgate Administrator

    @Jeff:

    Unfortunately, I think my old Watchguard box will be hardware-limited to ~400.

    Are you still running the 2.8GHz P4? My X6000 passes ~365Mbps but it's running at 1.2GHz. I would expect yours to pass well over 400Mbps.

    Steve



  • @stephenw10:

    Are you still running the 2.8GHz P4?

    Yeah.  I haven't done anything to the CPU.  I stuck more RAM in it because it was sitting in a box doing nothing.  But that's the only performance chance I made.  I had to replace the PSU and every capacitor on the motherboard though.  That was not fun.

    I haven't done any throughput testing on it.  The reduced speeds could be due to the .1p situation.  My outbound requests for data are going into the 'best effort' bin, which has the default effect of slowing down a server's response to me.



  • I haven't had a lot of time to do much testing as of now. I have just been trying to research a way of either doing the .1p settings in pfsense or possibly on a switch level before it gets to my pfsense machine.



  • I've made a bit of progress.  It's not strictly pfSense related, but I'm hoping we can use this to bridge the gap.

    I borrowed an Adtran Netvanta 1335 from work.  It's basically a router with some Layer 3 switching capabilities.  There's 24 10/100 ports and 2 gig ports.

    Right off the bat, this old POS looks to be hardware limited to ~120 Mbit/sec even on the gig ports.  I knew they were running out of gas (which is why we're replacing them at work) but I thought it was a CPU/ # of firewall sessions problem.  I guess it's all of the above.

    Also, I have no IPv6 enabled.  I'm not even sure it's supported on this platform.  No IPv6 = IPTV on this system.

    Anyway, I fixed the upload problem. Once I got basic connectivity established, I was pulling 120 down, and only 10 up.  Which is what Atlantis and I were seeing on pfSense.

    After I got a QoS policy in place, upload improved to match the download rates.  I was getting 120 both ways.   I did verify that the gig ports were auto-negotiating at the correct rate and not accumulating errors.

    So here's what I'm hoping for.  Adtran configs are very similar to Cisco.  It's my hope that some of the more knowledgeable folks will read what I did with this Adtran, and then chime in with how we might be able to implement a similar config on pfSense.

    I'll explain the relevant parts of the config, and I'll upload the entire thing as an attachment.  The only changes I've made is to remove the password hashes.  Everything else is line for line identical to my running config.  Please don't critique it too hard.  It's just something I banged together in a few minutes for testing purposes.

    So, to get this working:

    Create interface VLAN 2, and set it to DHCP.
    *Put interface gigabit-switchport 0/1 into VLAN trunking mode.  Verify that VLAN 2 obtains an IP address and you can ping out.
    *Turn up interface gigabit-switchport 0/2 and let it go on the default VLAN.  Add the necessary policies to allow outbound NAT. Verify access.

    Create access list GF-dhcp
    *Set the ACL to match both TCP and UDP port 67.  Probably only needs UDP.  Whatever.

    Create access list GF-default
    *Set this as a permit IP any <-> any

    Create QoS policy GF-QoS

    • On the first policy term, match against the GF-dhcp ACL
    • When packets match the ACL, set the VLAN priority / 802.1p / CoS bit 2
    • On the second policy term, match against GF-default
    • This is the catch-all rule, which applies VLAN priority / 802.1p / CoS bit 3
    • I wanted to do a ACL and QoS term for IGMP, but I couldn't figure out how to enable that.  Maybe later.

    Apply the QoS policy in the outbound direction on VLAN 2.

    All traffic exiting VLAN 2 towards the internet will have the .1p / CoS bits set, and upload speeds should see a dramatic improvement.

    Anyone want to take a crack at interpreting this into a pfSense config?

    GF-NV1335-noPW.txt



  • I was just able to get it working, but the QoS part is done through my switch (two ports are vlan'd off, one for the fiber jack, and one for pfsense). The switch strips off the QoS then passing the clean packets to pfsense. No TV as of now, as i probably need to upgrade to pfsense 2.1 to get full IPv6 support.

    I have a Zyxel GS1910 switch, and i wasn't sure which bit was for IGMP and which was for DHCP (they all just have a number code on my switch) so i set them all to 3.

    The result is a clean 930 down and 934 up.

    UPDATE

    Upgraded to pfsense 2.1 and set allow all IP6 traffic outbound, still no TV service, i get the guide, On-Demand, and DVR functionality, but no live TV.

    UPDATE 2

    The TV service not functioning does not appear to be caused by the firewall, it could be that the TV equipment somehow pairs to the router so you can't just take the TV box over to a buddies house and get the service from it. You'd think that they'd accomplish this task by only allowing IPs from their subnets on their IPTV servers, but who knows.



  • Have you read up on IGMP at all?  I've seen people in other threads about IPTV systems mentioning that they had to run some kind of IGMP proxy in order to get TV working.    I haven't dug too deep into it myself since my focus until now has been getting the data working correctly.

    I really hope it's possible to get the .1p stuff working in pfSense.  I've got a really nice gig switch, but I'd rather not have it managing both internal and external traffic, mainly from a simplicity standpoint. That was a big part of my reason for buying the Watchguard.



  • I haven't really looked at IGMP, i am going to investigate that more tonight when i have some time.



  • I was able to get my TV service working again, but with a cludge-y work-around. I just added another port on my switch to vlan 2, plugged the network box into that, and plugged the TV into the network box. ISP will give you more than one public IP (I have three from them at the moment, running 2 pfsense machines for fun, and 1 for their network box), so one for pfsense for your data network, and one for the their router and their TV equipment. I am still going to be tinkering around with the TV service though to see if i can get anything to work through pfsense entirely.

    Not exactly ideal, as now i cannot use the android app to control the tv equipment (unless i re-activate wireless on the NB and switch wireless networks whenever i want to, which would be stupid.). At least it's working though.

    I also contacted them to see if using our own routers was against their ToS in anyway, and i was assured it wasn't, just that their support will not assist us in getting this to work.



  • I was just able to get the TV equipment to work through my own router as well. Remember i am still doing the QoS at a switch level though, i am going to tinker around with QoS on pfsense though and see if i can get everything working 100% through pfsense.

    Until then my speedtests are right where they should be at about 930mbps download and 930 mbps upload. The IPTV service is also working completely through pfsense, and the TV app is also working great. Life is good.



  • Cool! If you have PfSense do the QoS aren't you taking clocks away from PfSense that could be used for other things that the switch has custom asics just for that purpose? For knowledge sake I say go for it, and please document how you got everything to work. 930 Mbps is awesome, I'm really jealous right now!



  • That is spectacular.  If the occasion ever arises, I'll buy you a beer or two :)

    Can you post the config details that get the TV going?



  • Yes, i am in the process of prepping a full write up/guide. I will post it in a bit.



  • I have completed the guide. Here it is:

    https://www.dropbox.com/s/zg9ju9373t0fnpu/GoogleFiberRouterGuide.pdf

    Have fun!


  • Netgate Administrator

    Ah, Google fibre. I was wondering what provider was giving you such huge bandwidth. Nice.  :)

    Reading through your guide (which I'll never be able to actually use  :() this part seems potentially confusing:

    Access your pfsense’s machine webgui and navigate to Interfaces -> Assign -> VLANs and add
    VLAN 2 to your WAN interface, as shown below:

    At this point, you should now be able to access the internet, though the upload speed will be limited
    to about 10mbps.

    Presumably at that point you actually have to assign the new interface, em1_VLAN2 in your example, as WAN?

    Steve



  • @stephenw10:

    Ah, Google fibre. I was wondering what provider was giving you such huge bandwidth. Nice.  :)

    Steve

    Yeah, the only remotely bad thing about it was the inability to use whatever router you chose. I assume they give you a router and do this to reduce the volume of technical support calls. Otherwise most of their calls would be regarding slow internet speeds, since most routers wouldn't have the ability to support such high bandwidth.

    em1 is my WAN interface (the interfaced directly plugged into port 2 on the switch mentioned earlier in the guide). You would need to tag that interface with VLAN 2.


  • Netgate Administrator

    Right so after you've added the new VLAN interface, em1_VLAN2, you have to re-assign WAN to use the new interface rather than using em1 directly which would still be sending untagged traffic.
    It's just that reading your document it could easily be interpreted as simply adding the VLAN to em1 is sufficient. Now it's highly unlikely that anyone who didn't understand this would be reading the document in the first place.  ;)

    Steve



  • AH, right, thanks for catching that. i will edit the document to explain that. I have actually determinded that the whole vlan step within pfsense is not needed, as the vlans are being set at the switch level. I will modify the document to reflect it.

    EDIT

    i am having a strange problem with this step though, maybe someone can help me figure it out.

    "To complete your IGMP configuration navigate to Firewall -> Rules -> LAN, edit your default
    allow any rule on your LAN network, scroll down to Advanced Features -> Advanced Options
    and check the first box., It should read, “This allows packets with OP options to pass. Otherwise
    they are blocked by default. This is usually only seen with multicast traffic.” Save the rule and
    apply your firewall settings."

    After activating this, it seems like DHCP goes crazy and does assign IP addresses so new clients (mostly wireless) are not able to connect to the network. This seems to be an intermittent issue, but its extremely annoying.



  • The netgear GS108Tv2 switch came today.  Holy number of settings, Batman.  I can't really be sure if I got them right or not, I was a little bit guessing having never really dug into Layer 2 like this.  I ended up pulling the switch entirely out of the picture, but the pfSense box still wasn't able to pick up an IP address via DHCP when plugged into the OTN.  I might be completely wrong, but I thought that was supposed to work - albeit with severely degraded bandwidth.

    Prior to that, at one point I had things all messed up, and the pfSense WAN picked up an IP address from its LAN - I think that was because I had the VLAN mappings in the switch goofed.  At least it tells me the WAN interface is capable of accepting and processing DHCP traffic, acting as a dhcp client.

    The only things I changed on the pfsense were the settings in the doc, mostly the stuff around the IGMP traffic.  The GFNB was able to talk to the OTN immediately through the same port on the patch panel.  I might try a simpler off-the-shelf netgear router tomorrow, just to see what happens.

    I posted a few screenshots of what the switch configuration looks like based on Atlantisman's document: https://www.dropbox.com/sh/ug31k8t6n9618ni/ligIuMmIiQ/gs108t_screenshots?lst.  There is no way to delete or rename the first three VLANs.  I really don't know what impact VLAN 2 being "Voice VLAN" has.  I can disable "Voice VLAN" in another screen, or try to move it to VLAN 3, but it doesn't change anything as far as I can tell.

    edit: corrected bad syntax. sorry, long day.



  • @rhornsby:

    The netgear GS108Tv2 switch came today.  Holy number of settings, Batman.  I can't really be sure if I got them right or not, I was a little bit guessing having never really dug into Layer 2 like this.  I ended up pulling the switch entirely out of the picture, but the pfSense box still wasn't able to pick up an IP address via DHCP when plugged into the OTN.  I might be completely wrong, but I thought that was supposed to work - albeit with severely degraded bandwidth.

    Prior to that, at one point I had things all messed up, and the pfSense WAN picked up an IP address from its LAN - I think that was because I had the VLAN mappings in the switch goofed.  At least it tells me the WAN interface is capable of accepting and processing DHCP traffic, acting as a dhcp client.

    The only things I changed on the pfsense were the settings in the doc, mostly the stuff around the IGMP traffic.  The GFNB was able to talk to the OTN immediately through the same port on the patch panel.  I might try a simpler off-the-shelf netgear router tomorrow, just to see what happens.

    I posted a few screenshots of what the switch configuration looks like based on Atlantisman's document: https://www.dropbox.com/sh/ug31k8t6n9618ni/ligIuMmIiQ/gs108t_screenshots?lst.  There is no way to delete or rename the first three VLANs.  I really don't know what impact VLAN 2 being "Voice VLAN" has.  I can disable "Voice VLAN" in another screen, or try to move it to VLAN 3, but it doesn't change anything as far as I can tell.

    edit: corrected bad syntax. sorry, long day.

    1. Yes it should work without the QoS settings, just highly reduced bandwidth (mine did, at about 930/10).

    2. Are you plugging anything else into the other ports on the switch? Or just pfsense and the OTN?

    3. Are you running pfsense in a VM or anything like that? If so you'd have to configure vlans on the virtual switch in esxi/hyper-v.

    4. It shouldn't matter that it is labelled voice vlan. Port one and two do need to be on VLAN 2 no matter what though.

    5. Some switches come with two different vlan options (private and normal), make sure you are not configuring a private vlan, otherwise the OTN won't send you packets.

    Those screenshots look right to me, though i am not too familiar with that particular switch.



  • Thanks for the info.

    @Atlantisman:

    1. Yes it should work without the QoS settings, just highly reduced bandwidth (mine did, at about 930/10).

    2. Are you plugging anything else into the other ports on the switch? Or just pfsense and the OTN?

    I tried setting up ports 1 (OTN), 2 (pfSense WAN) on VLAN 2 and the rest on VLAN 1 to isolate them.  No luck.  I also just plugged in the OTN and pfSense to the switch (everything else removed), and wiring my laptop into the pfSense LAN port to monitor the pfSense, no luck there either.

    @Atlantisman:

    3. Are you running pfsense in a VM or anything like that? If so you'd have to configure vlans on the virtual switch in esxi/hyper-v.

    No, it is running on metal.  I'm starting to wonder if having switch port 2 tagged is causing an issue.  I think the pfSense WAN interface MTU is 1492 but I'll have to check.

    Would it be appropriate to set switch port 1 to tagged and port 2 to untagged?  Both members of VLAN 2 as your point #4 states, yes.

    @Atlantisman:

    5. Some switches come with two different vlan options (private and normal), make sure you are not configuring a private vlan, otherwise the OTN won't send you packets.

    I don't see any options like those, but I'll keep looking.



  • @rhornsby:

    Thanks for the info.

    @Atlantisman:

    1. Yes it should work without the QoS settings, just highly reduced bandwidth (mine did, at about 930/10).

    2. Are you plugging anything else into the other ports on the switch? Or just pfsense and the OTN?

    I tried setting up ports 1 (OTN), 2 (pfSense WAN) on VLAN 2 and the rest on VLAN 1 to isolate them.  No luck.  I also just plugged in the OTN and pfSense to the switch (everything else removed), and wiring my laptop into the pfSense LAN port to monitor the pfSense, no luck there either.

    @Atlantisman:

    3. Are you running pfsense in a VM or anything like that? If so you'd have to configure vlans on the virtual switch in esxi/hyper-v.

    No, it is running on metal.  I'm starting to wonder if having switch port 2 tagged is causing an issue.  I think the pfSense WAN interface MTU is 1492 but I'll have to check.

    Would it be appropriate to set switch port 1 to tagged and port 2 to untagged?  Both members of VLAN 2 as your point #4 states, yes.

    @Atlantisman:

    5. Some switches come with two different vlan options (private and normal), make sure you are not configuring a private vlan, otherwise the OTN won't send you packets.

    I don't see any options like those, but I'll keep looking.

    I would setup the switch with a different port (3-8). After the switch is setup unplug everything but the OTN and the pfsense box.

    You may also need to setup vlans in pfsense, though i didn't have to. There is no reason why this wouldn't work.