Exclude CARP traffic from Traffic Shaping

mdima

Hi Namezero,
good question, but unfortunately I don't have an answer.

If you want to be sure that the CARP traffic is not limited by the traffic shaper, I can suggest you the same solution I implemented in my production environment: Create a "gigabit speed queue" and make it the default queue (do this for each interface). Then assign the traffic that you want explicitly to be limited/shaped using the floating rules.
This solution has a higher impact on the configuration, but I can grant you that it totally works (since I implemented it I had no more issues with CARP under heavy load).

You can find the detailed description of the solution in this thread, read my message posted on: "August 24, 2012, 06:03:57 pm".

Ciao,
Michele

adam65535

There is no way to override the pfsync and carp rules going to the default queue. The rules I listed are already listed and hidden before any rules you create. They will always go to the default queue. You must make sure your default queue is high priority or even it's own dedicated queue. Then make sure to classify all other traffic. I have essentially 4 queues. Low, medium, high, very high. I set the default queue to very high for the built in hidden rules that you can not override and I put everything else in different queues on the floating tab with match rules. I first have a catch all to put everything in low priority. I then classify other traffic into the medium and high queues(http, https, VPN, etc). This leaves BitTorrent on the low queue.

namezero111111

Hmm that seems like a bug then.

I was able to provoke the issue by jamming the connection and it started missing heartbeats right away and failed over after approximately 10-15 seconds.
Then I tested by changing the link speed to 950Gpbs and the problem disappeared.
I then set it back to the normal value and I cannot produce the problem since. Maybe a reboot would make the problem reoccur though when the states are cleared.

In that case, I will comment out the code in functions.inc so that I can assign CARP & PFSYNC to the highest available queue in the GUI, since it seems too difficult changing everything going through the default queue to something else.

mdima

Hi Namezero,
well… I did not know about that code:

@namezero111111:

I have dug a little deeper, and it seems that in 2.0.1 release this has been taken into consideration…
In functions.inc, on line 2752 there is a function "function filter_process_carp_rules()" that is called during the rule generation:

but even if in my rules.debug I find them, I had to implement a "gigabit default queue" to solve all the CARP/shaping problems… :S

I update the ticket in redmine...

adam65535

The hidden rules that your are talking about that specify carp do not have a queue specified so they go to the default queue.

mdima

@adam65535:

The hidden rules that your are talking about that specify carp do not have a queue specified so they go to the default queue.

OK, now I understand why I can't "shape" the CARP traffic (assign it to another queue), because there's a quick rule before that assigns the CARP protocol to the default queue… now everything is clearer, thanks! ;)

namezero111111

Thanks for all your replies.
You are correct, Adam. Since those rules catch first they go to the default queue.

Hence my workaround to comment the code out so I can define the rules in the GUI.
I haven't decided whether to do it that way yet or whether I'll path a queue (qAck, qVeryHigh) into functions.inc, but the first way seems cleaner somehow.
Either way should work though.

I suppose those two "hidden" rules were well intentioned, but they do seem to wreak havoc when congested and shaped interfaces come into play.

adam65535

I really wish rules like this were not hidden. Put them in floating and give them a comment that they are built-in system rules for pfsync and carp and maybe allow users to disable them. Hidden rules can become a problem for audits if the user does not know they exist. If there is a rule that allows access to something I honestly feel it should be shown to the user somehow. I feel that even DHCP rules should be shown on the floating tab as a system rule.

For example… the hidden pfsync and carp rules when using carp seem to allow all traffic through the firewall that match because they don't specify an in, out, src, or dst (which users might not know how open those rules are). Immediately I would worry that would allow any traffic to pass not only to your firewall but through your firewall with that protocol. A quick test using hping seems to indicate the traffic is passed through any of the interfaces from any to any ( I modified the builtin rules to log and the logs show pass). If you don't allow any incoming traffic from your WAN or you don't allow outgoing traffic from your DMZ network your security policy would have a hole in it for those protocols if the firewall forwards pfsync protocol or carp protocol packets because of the default rules. I doubt many ISPs will pass that traffic but even with that restriction I doubt many people would really like the idea of someone on their WAN subnet being able to craft packets through their firewall if they use carp features causing those hidden rules to be put in place.

The hidden rules I think really need to be restricted to certain IPs to restrict forwarding.

That is just an example of why I think hidden rules are a bad idea. Out of sight... out of mind.

namezero111111

Hmm yes you're right about that.
This could become problematic, because theoretically someone on a DMZ or WAN on the same subnet would be able to pass malicious CARP / PFSYNC traffic through the firewall to interfere with these protocols on other subnets, no?
Maybe my understanding of PFSYNC is a little too limited, but couldn't someone try to gratuitously inject invalid states into the firewall?

adam65535

I can think of two scenarios right off the top of my head that could potentially be concerning even though they are probably not likely to happen.

Messing with VRRP by someone sending traffic through the firewall to routers using VRRP (not carp) is probably more likely possible since VRRP is not encrypted like CARP. I do not know anything about the encryption with CARP to know about the specifics but I am assuming the encryption done on CARP does help with that for the CARP protocol.

Someone on a subnet that is defined to not have any outbound access could theoretically send data to a device on another subnet or potentially further than that if pfsync or carp traffic gets routed by whatever device it passes through. A custom app could be programmed to tunnel traffic through CARP or PFSYNC protocol to an outside custom service listening on pfsync/carp protocol. A fully restricted subnet/interface is one example where this could be an issue.

eri--

On carp there are protections but also not only on signing the contents but even TTL of the packet.

For pfSync this is a bit more troublesome but pfsync should get rules only for the configured hosts to be allowed to come in.

adam65535

The built in rules allow both pfsync and carp protocol packets from anywhere and to anywhere according to rules.debug. Any user rules which come after that would be ignored from what I understand.

Even though I have dedicated pfsync traffic to a specific interface for security the builtin rules would allow external WAN packets (or any other interface) to add states to the firewall.

There are protections on carp itself as we mentioned. VRRP uses the same protocol number though. If someone can pass traffic from your WAN to your inside networks to mess with your internal Cisco routers VRRP traffic that could still be an issue.

It seems like pfsync and carp would both need to have more restrictive rules to protect from this.

Pfsync should only need to get traffic on the defined sync network. That one seems easiest to fix by restricting that rule.

Psuedocode:

pass pfsync from defined_sync_subnet to defined_sync_subnet
pass pfsync from defined_sync_subnet to 224.0.0.x (assuming this is used)

Carp has to accept traffic from another system on the WAN which exposes it to spoofing but the encryption that you mentioned should protect it from that. If the builtin rule for carp were also changed to put the same restrictions on the carp traffic but for each individual interface then that would help the firewall bypass issue.

pass carp from lan_subnet to lan_subnet
pass carp from lan_subnet to 224.0.0.x

pass carp from wan_subnet to wan_subnet
pass carp from wan_subnet to 224.0.0.x

These rules would at least protect from being able to push pfsync and carp traffic into your protected networks from the WAN (or any other interface). They could of course be further restricted to their sync partners src IP.

It would be great to be able to setup a queue on this rules too to make sure they get top priority or some other solution which is what this discussion was really supposed to be talking about.

eri--

I thought in 2.1 this is already the case!

I am looking at fixing this queue issue.

namezero111111

Thanks ermal!

It would be great if you could post the changes here so we could backport the "official" solution to 2.0.1 for our systems that are already deployed and won't be upgraded soon.

Jon

This issue still exists with the latest snapshots. There needs to be default bypass for carp/pfxync traffic when the traffic shaper is configured, or a way to edit the queue for the built-in rules. Enabling the traffic shaper in a CARP/pfsync environment without making sure the default queue isnt rate-limited will break things.

Thanks,

Jon

armandocroce

Hi all,
running pfSense 2.1.5-RELEASE (amd64) with CARP and Traffic shaper configured is still causing false failovers from master to backup under heavy traffic.

Any update about this subject? Does the latest version (2.2.6) solve this problem?

Thanks,
armando

reghum

Hello all,

I'd also like to know if this issue is still present in pfSense 2.2.6. Anyone using such configuration ?

Regards,

Régis