Filtering a SPAN session from a switch
-
Hi, I was considering if pfSense can be used in my setup using (maybe) transparent firewalling or other features.
I have a big switch (Cat 6500) and a device that must analyse traffic flowing through it. The problem is the device only has 1 Gbit ports and the traffic is much higher on the VLAN I would like to monitor (cannot split VLANs further than that).
So basically I would like a setup where I receive the SPAN session from the switch on several Etherchannel/LACP/Plain interfaces*, filter them, and put the filtered traffic on the single 1gbit output interface (traffic will be unidirectional, being the source a mirror port).
The obvious answer "put ACLs on the SPAN session on the Cat" is not viable, since the Cat dies trying to filter the amount of traffic().
Also I cannot just put the output on a Gigabit interface and plug it to the device, because there are lots of dropped packets and the analysis does not work. If filtering the not interesting hosts on the source, the 1 gbit traffic will be enough for the analysis.
To sum it up I need a multi gigabit input -> ACLS -> output on a single interface. A sort of "filtered bridging".
(*) When used as an output virtual interface, aggregated interface should not use any aggregation protocol according to Cisco, so it seems they behave just like a plain bunch of interfaces where mirrored traffic is spread. Maybe this helps a bit.
() Yes, I suspect I'll need some serious computing power on my pfSense.Thank you for any suggestion.
-
Dude if your switch dies trying to put a ACL on your span.. Sounds like you need heftier switch;)
If your moving more than 1 gig of traffic that you need to work with… Use 10ge.. So bigger Analysis box as well.. This attempt at some sort of MacGyver hack with lagg ports is just behind nuts!!
-
Dude if your switch dies trying to put a ACL on your span.. Sounds like you need heftier switch;)
If your moving more than 1 gig of traffic that you need to work with… Use 10ge.. So bigger Analysis box as well.. This attempt at some sort of MacGyver hack with lagg ports is just behind nuts!!
I can surely do what you said, just lack half a million Euro to revamp my architecture while blocking production activities. Do you have some spare change ;) ?
I know I'm looking for an hack, and that is why I'm posting here (and yes, I love McGyver :)). In your view I can also replace my pfSense boxes with $BIGBRAND hardware doing transparent filtering on 10g fibers and solve my challenge.
I'm the first to suspect that if a device with ASICs devoted to filtering crashes under this load, the same when done using CPU power will not work very well but hey, that's the sort of things that make IT world interesting.
BTW moving SPAN sessions over aggregated ports is a perfectly valid solution in Cisco parlance and not a stupid one when not using aggregation protocols like LACP. If you have a 1.8 gbit traffic to send to your appliance you can use 2 ports on a low cost 1 gbit module, instead of buying a uber costly 10ge module and 10ge appliance.
For the sake of experimenting, cut all the numbers by 10: can this design work? I'm not sure where to create bridges between what. -
So just to clarify, you've got more than 1 Gbps of traffic, in a SPAN/RSPAN, being mirrored to a 1GBE port?
Even if filtering on the receive side to send to the analysis host in question, you're still going to be missing packets/frames/whatever you're spanning.
In theory, yes. You could filter non interesting host traffic before its sent to the analysis box. In practice, you might as well not even include the pfsense box, and on the (assuming linux) analysis box just tcpdump the traffic and do the filtering you need, then import the packet cap file and run analysis.
IMHO, the better option would be to filter down as much as possible on the cisco side. Remove some of the vlans that you've got set up in your RSPAN, in order to reduce the need for the ACLs, or atleast to the point where the ACLs are able to tackle the problem youre throwing at them.
good luck with the span and let us know how it goes! You're right though, this is what makes IT interesting ;)
-
So just to clarify, you've got more than 1 Gbps of traffic, in a SPAN/RSPAN, being mirrored to a 1GBE port?
Even if filtering on the receive side to send to the analysis host in question, you're still going to be missing packets/frames/whatever you're spanning.
In theory, yes. You could filter non interesting host traffic before its sent to the analysis box. In practice, you might as well not even include the pfsense box, and on the (assuming linux) analysis box just tcpdump the traffic and do the filtering you need, then import the packet cap file and run analysis.
IMHO, the better option would be to filter down as much as possible on the cisco side. Remove some of the vlans that you've got set up in your RSPAN, in order to reduce the need for the ACLs, or atleast to the point where the ACLs are able to tackle the problem youre throwing at them.
good luck with the span and let us know how it goes! You're right though, this is what makes IT interesting ;)
I have more then 1 gbit source traffic but using a portchannel I can direct it to several 1 gbit (R)SPAN ports, so nothing gets lost on the switch side. The problem is the receiving end that only has 1 network adapter at 1 gbit. I cannot dump and analyse the traffic, it is in real time. And as I wrote I cannot split the traffic anymore, it is a single VLAN with >1gbit.
Inside the single VLAN I know there is some traffic that I can filter out to reduce to <1 Gbit. This is where the pfSense comes in. Doing the same on the switch with VACLS just crashes it.
So I'm basically outsourcing the VLAN filtering to the pfSense device. -
My apologies, I misunderstood. You can accomplish this by setting some routing rules so the traffic coming in is routed through, and filter the traffic with firewall rules. The setup would have pfsense plugged into your span port, and your analyzing box plugged into pfsense. Assuming the traffic coming in is from the same subnet this should be pretty easy.
-
My apologies, I misunderstood. You can accomplish this by setting some routing rules so the traffic coming in is routed through, and filter the traffic with firewall rules. The setup would have pfsense plugged into your span port, and your analyzing box plugged into pfsense. Assuming the traffic coming in is from the same subnet this should be pretty easy.
This is where the thinsgs gets tricky and why I asked about the transparent firewall thing. The SPAN ports contains traffic that is basically duplicated production traffic, it is not the kind of traffic that flows through a firewall because it is routed to it. Also subnets are different. Ti give you an example in the mirrored VLAN I have 3 servers A,B and C. Hosts are in the 172.30.2.0/24 subnet (maybe in different subnets also but we'll leave this for another moment) and talk to C.
So I see 3-way handshakes and traffic between the machines, in both directions. My goal is ro receive all of this traffic and block any packet for server B (for example), so that the analysis machines only gets A<->C. Traffic will continue to flow between A<->B in the production VLAN, but the analysis machine will not receive it. Being B an high volume machine, removing that will reduce traffic that goes to the analysis machine via the 1 gbit interface. -
Maybe this Paint masterpiece helps
-
My apologies, I misunderstood. You can accomplish this by setting some routing rules so the traffic coming in is routed through, and filter the traffic with firewall rules. The setup would have pfsense plugged into your span port, and your analyzing box plugged into pfsense. Assuming the traffic coming in is from the same subnet this should be pretty easy.
This is where the thinsgs gets tricky and why I asked about the transparent firewall thing. The SPAN ports contains traffic that is basically duplicated production traffic, it is not the kind of traffic that flows through a firewall because it is routed to it. Also subnets are different. Ti give you an example in the mirrored VLAN I have 3 servers A,B and C. Hosts are in the 172.30.2.0/24 subnet (maybe in different subnets also but we'll leave this for another moment) and talk to C.
So I see 3-way handshakes and traffic between the machines, in both directions. My goal is ro receive all of this traffic and block any packet for server B (for example), so that the analysis machines only gets A<->C. Traffic will continue to flow between A<->B in the production VLAN, but the analysis machine will not receive it. Being B an high volume machine, removing that will reduce traffic that goes to the analysis machine via the 1 gbit interface.You set pfsense to route it. for example, set the WAN interface to go to span and give it a random ip outside of your private range. set a static route to forward traffic to LAN, and have the route encompass the network range for the equipment being tested (next hop to 172.30.2.0/24 is inside LAN), and have your analysis box on the lan port. Set firewall rules to pass ALL traffic on wan, and above that set filter rules to weed out HOST B. Also disable 'block bogon networks'.
this would essentially turn pfsense mostly into a router. PFsense thinks its forwarding traffic correctly. you could even go firther to give analysis box an ip on 172.16.1.0/30 subnet, and pfsense the other. then set that firewall rule to route traffic MATCHING the any rule to gateway "Analysis" and just define the analysis box as an upstream gateway. this would open up policy based routing for example.
-
What paint masterpiece?
I don't know how you think a layer 3 CPU will do better at filtering than a layer 2 switch.
If nothing else you should try filtering though another switch. Maybe something up to the task-at-hand.
-
What paint masterpiece?
I don't know how you think a layer 3 CPU will do better at filtering than a layer 2 switch.
If nothing else you should try filtering though another switch. Maybe something up to the task-at-hand.
he already said it wasnt in budget, hence abandoning my argument of replacing the cisco switch with a newer/more powerful model.
-
Maybe this Paint masterpiece helps
i dont have a fancy picture but youre going to do something like this with the topology:
hosts A/B/C in vlan XYZ get SPAN to 2gbps port -> pfsense wan (or opt interface, doesnt matter) set with static routes -> PFSense LAN connected to anaylisis box.
-
He doesn't need to replace the 6500. Just get something that can trim the traffic he needs trimmed. I mean if a $35 3750 off ebay can't do it, he's going to spend a helluva lot more on Xeons to get pfSense to do it.
-
He doesn't need to replace the 6500. Just get something that can trim the traffic he needs trimmed. I mean if a $35 3750 off ebay can't do it, he's going to spend a helluva lot more on Xeons to get pfSense to do it.
depends on what he has available. he expects his current pfsense hardware can handle it, so why not try it? My current setup is handling 10gbps throughput just fine, and im not sure an old 65k from cisco would be up to the task given the filterin requirements i have. thats besides the point though, point being that its possible he already has hardware able to do it except the said hardware isnt a switch.
EDIT:
wouldnt 166Mpps be well within most pfsense boxes anyway? thats the maximum throughput hes talking about filtering… -
Sigh
-
-
166Mpps. No.
-
so youre saying pfsense CANT handle theoretical throughput of 2gbps?
166 million packets isnt much, even for a firewall on non-asic hardware.
EDIT:
the 'medium business' hardware package being sold on the pfsense webstore can handle that just fine. and assuming its the 2015 cpu id say most pfsense boxes being built in a corporate test lab are going to be that age if not newer. OP is clearly not in a home lab. -
166Mpps at 64-byte packets is enough to saturate about 85 gigabit/sec, bro.
Oh. I get it. You think a packet is a byte. We will have to come to terms so we can communicate effectively. Might I suggest Comer.
-
166Mpps at 64-byte packets is enough to saturate about 85 gigabit/sec, bro.
Oh. I get it. You think a packet is a byte. We will have to come to terms so we can communicate effectively. Might I suggest Comer.
no, a packet isnt a bit. that would be a byte.
2gbps / 8 (bit to byte conversion) = 250MBps (bytes).
MTU of 1500 BYTES (since thats the typical mtu, but can vary.) 250,000,000 / 1500 = 166,666 packets. since most packets arent full, and average 576 bytes, that would be 434,027 packets per second. Since thats the theoretical throughput, you can actually adjust that number depending on actual averages. but if you wanted to run it your way and misinterpret a packet to be a byte, that would be 250Mpps ON A SINGLE LINK. Correlating that with the iperf tests i literally just ran 1gbps at 8% cpu utilization, and im not on top end equipment. So I'd say its fair to say a pfsense install in a corporate lab environment is going to be capable of pushing 166Mpps across its 'backplane'. Since, you know, packets per second are theoretical measurements used to descript the capability of a backplane, and not a sing link, bro.
i do appreciate the snide remarks though. keeps me on my toes :)
-
Dude. The first rule of getting out of a hole is to stop digging.
In no sane networking conversation does packet = byte.
-
Dude. The first rule of getting out of a hole is to stop digging.
In no sane networking conversation does packet = byte.
im failing to see how assuming a packet being a single byte, and link speeds moving at 250 megaBYTES per second, isnt 250Mpps.
ive already explained how pps is measured against a backplane, and not a single link. before you jump to the conclusion i dont know what im talking about you can go educate yourself on this topic with some cisco tech specs and running the numbers yourself.
and in no sane networking conversation is packets per second a single link. thats always across a backplane. measuring links speeds in pps isnt easily understood because of the complexity. links speed being tyically measured in bits per second or bytes per second. Point being, if the PPS on a 2Gbps link is in the hundreds of thousands, the backplane on the hardware being capable to handle in the millions, there isnt a performance issue. Therefore, the hardware can handle it.
-
Because the basic, minimum packet size is 64 BYTES so 250 megabits/sec is 480,000 packets per second, using rudimentary math.
-
Because the basic, minimum packet size is 64 BYTES so 250 megabits/sec is 480,000 packets per second, using rudimentary math.
alright, follow the bouncing ball here.
480,000 pps max on a single link. im saying single link loosely, because its going the be the aggregate of presumably a LAGG interface.
backplane PPS being 166M.
so a link transferring 480Kpps leaves how much left for the rest of the backplane? 165,520,000 packets per second (unconsumed), using rudimentary math.
Therefore:
the hardware can handle it. -
You will never be able to have an intelligent conversation on this subject with anyone until you understand that a byte is not a packet.
-
You will never be able to have an intelligent conversation on this subject with anyone until you understand that a byte is not a packet.
Of course a packet isnt a single byte, bro. which at this time id like to point to where i said:
@isolatedvirus:im failing to see how assuming a packet being a single byte, and link speeds moving at 250 megaBYTES per second, isnt 250Mpps
To which you said the minimum is 64b. Which places a 2gbps link well within the capability of hardware that can handle 166Mpps to filter/route.
I'm really failing to see why youre not understanding 166Mpps is a backplane throughput, and not a 2gbps link throughput.
-
166Mpps is 85 gbit/sec, not 2gbit/sec, dude. Stop saying pps when you mean Bps
-
166Mpps is 85 gbit/sec, not 2gbit/sec, dude. Stop saying pps when you mean Bps
-_- do you even know what a backplane IS??! i mean seriouosly. tell me what a backplane is.
-
Do you even know what a packet is? Tell me what a packet is.
-
Because the basic, minimum packet size is 64 BYTES so 250 megabits/sec is 480,000 packets per second, using rudimentary math.
alright, follow the bouncing ball here.
480,000 pps max on a single link. im saying single link loosely, because its going the be the aggregate of presumably a LAGG interface.
backplane PPS being 166M.
so a link transferring 480Kpps leaves how much left for the rest of the backplane? 165,520,000 packets per second (unconsumed), using rudimentary math.
Therefore:
the hardware can handle it.reread this maybe? im seriously at a loss for words on how else to word this. maybe ignore the numbers for a second? Backplane minus link = leftover resources on the backplane. If the backplane is faster than the link, there is left over resources. Meaning the hardware is capable of handling the throughput.
-
Do you even know what a packet is? Tell me what a packet is.
a packet being layer three encapsulation of a frame, marked typically with tcp/udp as the transport layer protocol but not exclusively.
Standard MTU on a packet being 1500, unless youve changed something.
EDIT:
and by change something i mean mess with the MTU, like enabling jumbo frames to support larger packets. -
maybe this is whats tripping you up since you didnt know pps is a measurement commonly given for a backplane.
so youre saying pfsense CANT handle theoretical throughput of 2gbps?
166 million packets isnt much, even for a firewall on non-asic hardware.
EDIT:
the 'medium business' hardware package being sold on the pfsense webstore can handle that just fine. and assuming its the 2015 cpu id say most pfsense boxes being built in a corporate test lab are going to be that age if not newer. OP is clearly not in a home lab.that 166Mpps is clearly me referencing the backplane. not the individual link speed given i later provided the math for the calculation which came out to 166Kpps. I also clarified packets arent measured in bits but instead are measured in bytes.
166Mpps at 64-byte packets is enough to saturate about 85 gigabit/sec, bro.
Oh. I get it. You think a packet is a byte. We will have to come to terms so we can communicate effectively. Might I suggest Comer.
no, a packet isnt a bit. that would be a byte.
2gbps / 8 (bit to byte conversion) = 250MBps (bytes).
MTU of 1500 BYTES (since thats the typical mtu, but can vary.) 250,000,000 / 1500 = 166,666 packets. since most packets arent full, and average 576 bytes, that would be 434,027 packets per second. Since thats the theoretical throughput, you can actually adjust that number depending on actual averages. but if you wanted to run it your way and misinterpret a packet to be a byte, that would be 250Mpps ON A SINGLE LINK. Correlating that with the iperf tests i literally just ran 1gbps at 8% cpu utilization, and im not on top end equipment. So I'd say its fair to say a pfsense install in a corporate lab environment is going to be capable of pushing 166Mpps across its 'backplane'. Since, you know, packets per second are theoretical measurements used to descript the capability of a backplane, and not a sing link, bro.
i do appreciate the snide remarks though. keeps me on my toes :)
-
id also l ike to point out in my testing i was pushing 166kpps through a 1gbps link and saw a jump from 4% to 8% cpu utilization.
given the cpu utilization, i could push 3.2Mpps (166k * 20) and this would put me at 80% cpu utilization. Again, not on crazy hardware.
-
My apologies, I misunderstood. You can accomplish this by setting some routing rules so the traffic coming in is routed through, and filter the traffic with firewall rules. The setup would have pfsense plugged into your span port, and your analyzing box plugged into pfsense. Assuming the traffic coming in is from the same subnet this should be pretty easy.
This is where the thinsgs gets tricky and why I asked about the transparent firewall thing. The SPAN ports contains traffic that is basically duplicated production traffic, it is not the kind of traffic that flows through a firewall because it is routed to it. Also subnets are different. Ti give you an example in the mirrored VLAN I have 3 servers A,B and C. Hosts are in the 172.30.2.0/24 subnet (maybe in different subnets also but we'll leave this for another moment) and talk to C.
So I see 3-way handshakes and traffic between the machines, in both directions. My goal is ro receive all of this traffic and block any packet for server B (for example), so that the analysis machines only gets A<->C. Traffic will continue to flow between A<->B in the production VLAN, but the analysis machine will not receive it. Being B an high volume machine, removing that will reduce traffic that goes to the analysis machine via the 1 gbit interface.You set pfsense to route it. for example, set the WAN interface to go to span and give it a random ip outside of your private range. set a static route to forward traffic to LAN, and have the route encompass the network range for the equipment being tested (next hop to 172.30.2.0/24 is inside LAN), and have your analysis box on the lan port. Set firewall rules to pass ALL traffic on wan, and above that set filter rules to weed out HOST B. Also disable 'block bogon networks'.
this would essentially turn pfsense mostly into a router. PFsense thinks its forwarding traffic correctly. you could even go firther to give analysis box an ip on 172.16.1.0/30 subnet, and pfsense the other. then set that firewall rule to route traffic MATCHING the any rule to gateway "Analysis" and just define the analysis box as an upstream gateway. this would open up policy based routing for example.
I got your picture and will try. I just don't understand how pfSense will think to route to internal something that is not going to be routed, but I suspect it is in the magic of "trasparent" routing. Don't know how it can behave since all parts of the TCP connection (3w handshake) appear on the wan instead of coming from wan and lan. Worth trying.
Thanks -
By the way guys, cool down! No need to heat up.
I got virus' point on backplane and agree with him. My Cat has spare on that side but the ASIC that do the filtering do badly fail. In the docs they say to not worry about Vlan ACLs since are done in hardware: I tested (not supposed) this is not real even being far from saturating the backplane.
And @derelict: I was the first one to say I'm not sure a standard CPU will cope with that traffic (1st message), just trying.
So, with some hardware coming, I'm going to test this setup and report on the results. -
Get a better HW. End of story.