Bridging Troubles



  • So, I finally decided to give pfSense a try on an old desktop PC I wasn't using.  It's an socket AM2 with 8 GB of RAM running an Athlon 64 X2 4600+ (2.4 GHz dual core).  I'm using the onboard Ethernet Nic for my WAN (nfe0) and I grabbed a trio of old PCI Ethernet Nics for my LAN (I can only install two on the motherboard).  I followed a few guides including, https://www.all4os.com/router/bridge-multiple-lan-portsnics-to-act-like-a-router-in-pfsense-2-1.html and http://www.cyberciti.biz/faq/how-to-pfsense-configure-network-interface-as-a-bridge-network-switch/ except I still can't get it working.

    I started out with a card that is listed to work (dc0) and one of the two that I wasn't sure about (rl0).  I was getting internet throught the WAN and to my old router (DHCP off in the router and using a LAN side port) and to all computers attached using the dc0 card.  Both of my LAN ports were showing their status as UP, so I decided to bridge dc0 with rl0 to learn how to get multiple ports on my pfSense router which is where the problems began.  The dc0 card was not showing status as UP even though I had just had internet on it when it was solo.  I was able to get internet through rl0 while in the bridge just fine.  After alot of playing around, I finally rebooted the machine and dc0 was UP and both ports had internet.  A few days later, my power went out during a bad storm and the dc0 port went back to status DOWN.  I assumed the card might be bad, so I powered down and tried the other card.  It showed up as dc0, so I tried it and still got a status of DOWN.  I then confirmed that the dc0 was showing as different Nics by rebooting the system with each of them in and recording the MAC address for each of them.  It appears to be an issue with the computer itself.  I tried different arrangement of the ports with the nfe0 being LAN and dc0 being WAN with dc1 just hanging out, not doing anything and I couldn't get the WAN to work even though the LAN did and all the links showed status UP.  That leads me to believe that I've got IRQ conflicts.  Where could I start looking to confirm that and are there any other options for what my problem could be?

    I could see it being possible that the power outage messed up my PCI Bus, but it's highly unlikely since I had these issues preceding that event.  If you have suggestions on how I might test the PCI Bus, I'll try them, but otherwise, without a testing procedure, I'm not entertaining the idea of my PCI Bus being the culprit until I've ruled out the IRQs and anything else that it might be.


  • Netgate

    I decided to bridge dc0 with rl0 to learn how to get multiple ports on my pfSense router which is where the problems began.



  • So, you're saying I did it wrong?


  • Netgate

    It's a router not a switch just get a switch.



  • Would a switch not slow me down?  If I have separate Nics, don't they each pull at their full speed and bandwidth?  Wouldn't a switch be limited to the bandwidth for the single card?  There are plenty of people doing this and they suggested this route, so I'm a bit confused on why you would tell me not to do this.  How else am I supposed to put wireless in this router (I know I can attach old router's as APs, but what if I wanted to use one of these as an AP?) or link aggregate a server to this?  The functionality is there, so why wouldn't I use it?  Honestly, I'm trying to figure out if I messed something up or if this is hardware related (which I think it is).  If you only use it for a router and use switches from there on, that's fine.  You don't have to be quite so dismissive.  :P

    Seriously, I just need a little bit of help before I decide to go all in with this.  No sense in wasting $100 on this when I could have just bought a $100 router if it won't work using pfSense.  I had a spare computer and wanted to play around with this.  I felt like I might learn a thing or two.

    If it'll make you feel better, then I'll hear your reasoning on why I shouldn't be bridging.  What are the disadvantages to doing it the way I am and what are the advantages to doing it the way you think I should?


  • Netgate

    Any time a switch has to forward a frame using software, performance suffers.

    pfSense bridges forward every frame between ports using software.

    Even the cheapest $19 switch forwards frames using hardware ASICs.

    On a switch, traffic between LAN hosts doesn't go through your "router" It is all local to the switch and it will be MUCH faster than anything going through pfSense. The only traffic that has to hit your router is traffic that has to be routed, like out to the internet.

    There are certain cases where pfSense bridging makes sense. Having more ports on LAN isn't one of them. Get a switch, connect it to your LAN interface, and forget about it.



  • Thanks for the explanation.  That makes sense.  So, it'd be faster to use a single gigabit port on a switch to communicate with my server than it would be to use the software to link aggregate 4 gigabit ports?  Also, the pfSense router will still handle the DHCP assignment for all of my machines and APs?  Everything just goes through a switch and never actually hits the router, even though it routes the information?  Is this true for any router that I might buy, like a Netgear?


  • Netgate

    Bridging is not link aggregation. You are talking about two completely different things.

    The fastest way you are going to talk to a server on LAN from a host on LAN is using two switch ports. And you certainly don't want a router between the two.

    The LAN ports on a "Netgear" will have associated switching ASICs I already mentioned. Nothing like that exists for pfSense that I know of.

    A switching ASIC on a four-port card with FreeBSD support would be an interesting thing.



  • So, basically, I'm wasting my time looking for pfSense to be a SOLID all in one box.  I clearly have ALOT to learn, which is why I made the move to pfSense.

    So, with a consumer grade router like Netgear, Linksys, ASUS, etc., I'm buying a router with a switch interface that's not bridged with the WLAN, but that's kept separate?  With pfSense, then, I need to buy a separate switch to handle all of my traffic.  If that's the case, then I need to buy a SMART Switch in order to perform the link aggregation won't I?  So, what you're telling me is that pfSense will use DHCP to hand out IPs and such and then the computers will never talk to it again for local network traffic?  On that note, is it a bad idea to use wireless on a pfSense machine period?  Based on what you're saying about the wired ports on a consumer router, I'm assuming that the wireless is handled the same way in that the router doesn't actually handle the internal traffic.  On that note, then why are people using pfSense as APs?

    Is there no way other than Bridging that will achieve my goals without using a switch?  You're saying that a 4-port Nic won't act as a hardware switch.  I guess I should give up on that idea for now, then.

    When I was talking about link aggregation, was that not going to have to be bridged in as well in order to be on the same network?  I know the difference between bridging and link aggregation, but I haven't put everything together yet (thankfully), so I don't know how it gets implemented.  I assumed that I would have to link aggregate a 4-port Gigabit Nic and then bridge it in with the rest of the Nics that I wanted on the network, including my wireless N card.

    I don't see why you didn't just tell me that bridging was slower than a switch from the get go instead of quoting me and dismissing my issue.  I still don't understand why my (soon to be removed  XD  ) Bridge didn't work…lol.  Not that it matters now.  I do believe what you're saying.  It makes pretty good sense.  I'm just confused about why there are so many people using it the wrong way if what you're saying is true.


  • Netgate

    I'm just confused about why there are so many people using it the wrong way if what you're saying is true.

    Because so many people are idiots.

    pfSense to be a SOLID all in one box

    This is a mental disorder some people have. Trying to make their FIREWALL do everything "in one box".

    If you want Wi-Fi, get an access point that is DESIGNED to be NOTHING BUT an access point. It will WORK GREAT and you WON'T HAVE TO FUCK WITH IT!

    If you want to switch Layer 2, get a switch that is DESIGNED to be NOTHING BUT a layer 2 switch. It will WORK GREAT and you WON'T HAVE TO FUCK WITH IT!

    Ad freaking nauseum.

    As an aside, your posts are tl;dr.

    LACP/LAGG AND BRIDGING INTERFACES ARE NOT THE SAME THING! Stop talking about them like they are.



  • Using that logic, you wouldn't use pfsense as a router, cause really it's a firewall..

    I digress. A switch will be faster and lower latency unless it is a poor switch (I am looking squarely at you d-link).

    Personally I like mikrotik switches, but I'm weird



  • I don't really understand why this is so upsetting to you.  I'm glad you've pointed me in the right direction as to hardware layout choices.  I guess my thing is that everyone has different usage needs and my needs aren't what some others might need.  I'm pretty flexible and your suggestions are pretty easy for me to go with.  I have two older servers with parts that I had on hand, but before taking them apart I was going to use some other parts I had lying around.  If everything went smoothly, I HAD planned to buy additional parts, but I'll be picking up a switch instead.

    A note on Link Aggregation and Bridging, I never said they were the same.  I know that Link Aggregation is connecting a group of ports so that they share the bandwidth while maintaining their speed.  I understand that Bridging on the other hand, is multiple Nics sharing a Network Segment so that they can communicate with one another using the same Gateway.  I assumed that in order to have the Link Aggregate on the same Network Segment, I would have to bridge it afterward.  Was this a wrong assumption?



  • @Keljian:

    Using that logic, you wouldn't use pfsense as a router, cause really it's a firewall..

    LOL…this is getting great.  So, what is a better router?  Is pfSense my best option for a firewall?


  • Netgate

    Was this a wrong assumption?

    Yes LAGG and bridging are completely different technologies for $^$&'s sake. www.wikipedia.org


  • Rebel Alliance Global Moderator

    "Using that logic, you wouldn't use pfsense as a router, cause really it's a firewall.."

    Yeah to be honest that is correct.. If what you wanted to do was route and not firewall you shouldn't be using pfsense..  Actual Router would do a much better job at that.

    Looking at every packet to determine if meets a rule to be allowed or blocked is not the most efficient routing now is it..

    Using 2 interfaces in a router as a switch when you just want to switch between them makes no sense either..  Now if you had 2 different physical network segments that you wanted to transparently firewall between, then using a bridge on pfsense might make sense.

    If you have a few network segments that want to talk to each other "router" and you also want to control what traffic is allowed "firewall" then yes using your firewall as your router makes sense.  If all you were going to do is have any any rules.. Using a firewall as your router would be pretty stupid as well.

    Because so many people are idiots.

    This needed to be stated again because of how freaking true it is! ;)

    So for example in a decent sized network with many local segments.. Its quite possible you would just use a L3 switch to let your multiple segments talk to each other "route" but where your network talks to the edge/internet/wan you would use pfsense.  You might have different networks that connect into pfsense, lan, dmz, etc.  That you want to talk to each other in limited fashion.  But pfsense is there to route to and from the wan.. Your routing in your internal network where you not doing any firewall stuff pfsense would be a pretty bad choice..




  • A note on Link Aggregation and Bridging, I never said they were the same.

    For sure it isn´t the same.

    I know that Link Aggregation is connecting a group of ports so that they share the bandwidth while maintaining their speed.

    Aggregating from two till eight LAN ports together that is then acting as a big fat pipe will be coming
    nearly that doings. You can go with a static LAG and manual set up and a dynamic LAG using the LACP.

    I understand that Bridging on the other hand, is multiple Nics sharing a Network Segment so that they can communicate with one another using the same Gateway.

    Bridging will be need in some rarely cases where it must be used, and only some very experienced network
    admins know really when and where this is the best option to go with. But for all others of us we should go
    with routing.

    I assumed that in order to have the Link Aggregate on the same Network Segment, I would have to bridge it afterward.  Was this a wrong assumption?

    Yes, it was.

    LOL…this is getting great.  So, what is a better router?

    A router is routing packets from one to another or more networks.
    A firewall is separating one network from another or more networks

    That both are able to route, makes them not being devices of one class or playing in the same league.
    For sure at today this borders will be more and more blurred makes them once more again not to be
    pointed in one class of devices or playing in one and the same league.

    Is pfSense my best option for a firewall?

    This is not really so easy to answer, because we all don´t know your networking skills and the entire
    work filed you are in. Each firewall comes with their own goodies and bad things or less playing nice
    fields, pfSense is a firewall that offers you some options from the most other firewall but combined
    in one fine structure that is even and active maintained and worked on. For sure there are other out
    there you might be have a look on if you are not really impressed by pfSense. but if it comes to the
    actual hardware support and usage of some new things to speed up much, you will be searching
    for a greater while at other distros and also the different stages of the support abilities that comes
    besides with pfsense.

    Firewalls:

    • ZeroShell
    • IPCop
    • IPFire
    • SmoothWall
    • Endian Firewall
    • VyattaOS
    • OPNSense

    UTMs:

    • Untangle UTM
    • Sophos UTM
    • pfSense als UTM
    • GB-OS

    Router:

    • OpenWRT
    • DD-WRT
    • RouterOS
    • fli4l & Eisfair
    • Freetz
    • FreeSCO
    • LEAF Project
    • SME Server
    • Zentyal
    • Tomato

    Große BGP Router:

    • OpenBSD & Quagga
    • OpenBSD & Zebra
    • VyattaOS

    Others:

    • ClearOS
    • CentOS & SoftEtherVPN

    Their all having their pro´s and con´s but something good from each or all of them you will find only
    merged together in pfSense and yes so it will be one of the best options you will be able to deal with.

    The best bet is to go early as you can do it the right way or one step ahead! Don´t use bridging, go
    and search a more powerful hardware and go with routing instead. And so on with all other things,
    likes the LAG, you can easily find some SMB switches with 2 x 10 GbE or SFP+ ports that will be
    changing much more your game and entire throughput as you could do it with building LAGs.

    LAGs are only making sense for me if there are many users or devices and they are also connecting
    to one server or other device.



  • A quick one about link aggregation which isn't usually well explained:
    If you have 2 links aggregated, the maximum speed of any connection to the aggregated pair is the speed of one link.

    Therefore, the highest speed you will see over 2 ports from one single IP is the speed of one port.

    If you have two devices hooked up, the maximum that each will see is the speed of their own link

    Generally speaking, if it is at all possible, you want to go with faster links instead of aggregated links. Firstly for simplicity, but second because you are likely to benefit more from it.



  • @Keljian:

    If you have 2 links aggregated, the maximum speed of any connection to the aggregated pair is the speed of one link.

    Same speed, but increased throughput.  My goal isn't to make things move faster than gigabit, only that I want to maintain my gigabit speed while multiple devices make their transfers.

    @Keljian:

    Generally speaking, if it is at all possible, you want to go with faster links instead of aggregated links. Firstly for simplicity, but second because you are likely to benefit more from it.

    Price and availability at that price.  I'm a home user who'd LOVE to have 10G networking, but justifying it to my wife could be a bit of a problem.

    I appreciate all the input guys.  Thanks to you (more specifically, Derelict) I realize that I was going in the wrong direction.  This thread has become more about the proper layout and less about the issue I was having, so, as much as I'd like to continue racking all of your brains, I think I need to research this new direction more myself and then return and start a thread specific to it (if there isn't already one out there) if I still have more questions.  I've already pissed Derelict off enough I do believe.  ;D  Thanks BlueKobold for specifically answering my question about the assumption that I could Bridge a Link Aggregate into the same Network Segment as the rest of my network.  I'll look more into how I need to go about that as well.  I guess I can't expect to have all my devices on the same Network Segment if I go that route.



  • Ok so $240 for a 10 gig (x2 port) switch
    $40 for 2x fibre transmitters
    $5-10 for the optical fibre
    $20 for the network card on eBay (Mellanox connectx 2)

    $310 all told for a 10 gig trunk plus switch

    Could reduce it by $30 if you use twinax sfp+ passive and another 60 if you only need 8x gigabit ports ($220 all told!)

    It's not that expensive



  • It's not that expensive

    But easy to use, easy to configure and really powerful.

    @csswormy
    As a home user it is also not really wise to use a LAG, pending on the configuration, because the first link
    must be fully rendered or saturated until the next one will be in use. So a home network often is not able
    to produce so much traffic.