Exclude CARP traffic from Traffic Shaping

adam65535

I just ran across this too. It would be nice to have an option in the queue section to set a carp/pfsync queue. This setting would then need to modify the built-in rules below to use the queue you specify.

pass quick proto carp
pass quick proto pfsync

I guess it really isn't a big deal. As long as you know that those carp rules are hidden and go to the default queue. It would be nice if that were mentioned in the shaper docs.

mdima

Hehe! Thanks for the reply, it looked like this problem was involving almost none and this post looked like a monologue… but I think that the combination CARP + Traffic Shaper isn't that rare...

Btw your suggestions would prevent any other user to run into this problem.

Ciao,
Michele

SeventhSon

I'm running something like this, just not at 100% at any time really, but the options adam here put in, should work just fine.

eri--

I added http://redmine.pfsense.org/issues/2997 probably it want be looked upon before 2.1 gets released.

SeventhSon

Thanks ermal

namezero111111

We have struggled with random CARP failovers for almost two years, and have come to the same conclusion now that the traffic shaper is the culprit.
We will verify this shortly and if that turns out to be the case, we'll add a shaper rule to allow the carp protocol floating with a 7 priority (higher than ACK).

Manually enabling/disabling CARP on either member fixed it for us in the past, but due to the loss of state on failback this was annoying…

mdima

Hi Namezero,
probably the solution by adam is the easiest to implement:

pass quick proto carp
pass quick proto pfsync

Just add this two floating rules on top of the list (pay attention to the "quick" option).

Ciao,
Michele

namezero111111

thanks for the reply.
In the GUI, would a pass quick translate to an unqualified rule wih only the protocol selected?

Also, we dont run pfsync on a shaped interface. Should the rule still be added just to be sure?

mdima

Hi namezero,
if in the rule is specified only the selected protocol, yes, it would apply only to that protocol. Consider that the CARP protocol has sense only in the devices attached to the same physical network, and should be already protected by the password specified for each virtual IP.

About pfSync, I would add it just to be sure, even if it does not have such bad effects if shaped.

namezero111111

Thank you for your quick answer. I have created the rule as can be seen in the screenshots.
However, it doesn't seem to catch CARP traffic.
I have also tried any protocol with destination 224.0.0.18 with the same results.
Then I tried both with state set to none instead of keep state with the same results.

But somehow the rule doesn't catch. The /tmp/rules.debug looks like it is correct:


# User-defined rules follow

anchor "userrules/*"
pass log  quick  proto carp  from any to any keep state  label "USER_RULE: CARP must never be shaped"
...
...

I have also noticed that before the user rules section there is something from a snort package:


# Snort package
block quick from <snort2c>to any label "Block snort2c hosts"
block quick from any to <snort2c>label "Block snort2c hosts"
block in log quick proto carp from (self) to any
[b]pass quick proto carp
pass quick proto pfsync[/b]</snort2c></snort2c>

However, snort isn't installed on this machine…

Any ideas?

EDIT: Could this be because the state for cars is NO_TRAFFIC:SINGLE?


carp 	224.0.0.18 <- xxx.yyy.134.23 	NO_TRAFFIC:SINGLE

Untitled.png_thumb

Untitled2.png_thumb

namezero111111

Greetings,

I have dug a little deeper, and it seems that in 2.0.1 release this has been taken into consideration…
In functions.inc, on line 2752 there is a function "function filter_process_carp_rules()" that is called during the rule generation:


function filter_process_carp_rules() {
	global $g, $config;
	if(isset($config['system']['developerspew'])) {
		$mt = microtime();
		echo "filter_process_carp_rules() being called $mt\n";
	}
	$lines = "";
	/* return if there are no carp configured items */
	if(isset($config['installedpackages']['carp']['config']) &&
			 $config['installedpackages']['carpsettings']['config'] <> "" or
			 $config['virtualip']['vip'] <> "") {
		$lines .= "block in log quick proto carp from (self) to any\n";
		$lines .= "pass quick proto carp\n";
		$lines .= "pass quick proto pfsync\n";
	}
	return $lines;
}

It also has nothing do do with snort (functions.inc, line 2111):


	$ipfrules .= << <eod<br># Snort package
block quick from <snort2c>to any label "Block snort2c hosts"
block quick from any to <snort2c>label "Block snort2c hosts"

EOD;

	[b]$ipfrules .= filter_process_carp_rules();[/b]

	$ipfrules .= "\n# SSH lockout\n";
	if(is_array($config['system']['ssh']) && !empty($config['system']['ssh']['port'])) {
		$ipfrules .= "block in log quick proto tcp from <sshlockout>to any port ";
		$ipfrules .= $config['system']['ssh']['port'];
		$ipfrules .= " label \"sshlockout\"\n";
	} else {
		if($config['system']['ssh']['port'] <> "")
			$sshport = $config['system']['ssh']['port'];
		else
			$sshport = 22;
		if($sshport)
			$ipfrules .= "block in log quick proto tcp from <sshlockout>to any port {$sshport} label \"sshlockout\"\n";
	}</sshlockout></sshlockout></snort2c></snort2c></eod<br>

This should take care of the solution, no?
If this rule is in there, why are we missing CARP packets under heavy load?

mdima

Hi Namezero,
good question, but unfortunately I don't have an answer.

If you want to be sure that the CARP traffic is not limited by the traffic shaper, I can suggest you the same solution I implemented in my production environment: Create a "gigabit speed queue" and make it the default queue (do this for each interface). Then assign the traffic that you want explicitly to be limited/shaped using the floating rules.
This solution has a higher impact on the configuration, but I can grant you that it totally works (since I implemented it I had no more issues with CARP under heavy load).

You can find the detailed description of the solution in this thread, read my message posted on: "August 24, 2012, 06:03:57 pm".

Ciao,
Michele

adam65535

There is no way to override the pfsync and carp rules going to the default queue. The rules I listed are already listed and hidden before any rules you create. They will always go to the default queue. You must make sure your default queue is high priority or even it's own dedicated queue. Then make sure to classify all other traffic. I have essentially 4 queues. Low, medium, high, very high. I set the default queue to very high for the built in hidden rules that you can not override and I put everything else in different queues on the floating tab with match rules. I first have a catch all to put everything in low priority. I then classify other traffic into the medium and high queues(http, https, VPN, etc). This leaves BitTorrent on the low queue.

namezero111111

Hmm that seems like a bug then.

I was able to provoke the issue by jamming the connection and it started missing heartbeats right away and failed over after approximately 10-15 seconds.
Then I tested by changing the link speed to 950Gpbs and the problem disappeared.
I then set it back to the normal value and I cannot produce the problem since. Maybe a reboot would make the problem reoccur though when the states are cleared.

In that case, I will comment out the code in functions.inc so that I can assign CARP & PFSYNC to the highest available queue in the GUI, since it seems too difficult changing everything going through the default queue to something else.

mdima

Hi Namezero,
well… I did not know about that code:

@namezero111111:

I have dug a little deeper, and it seems that in 2.0.1 release this has been taken into consideration…
In functions.inc, on line 2752 there is a function "function filter_process_carp_rules()" that is called during the rule generation:

but even if in my rules.debug I find them, I had to implement a "gigabit default queue" to solve all the CARP/shaping problems… :S

I update the ticket in redmine...

adam65535

The hidden rules that your are talking about that specify carp do not have a queue specified so they go to the default queue.

mdima

@adam65535:

The hidden rules that your are talking about that specify carp do not have a queue specified so they go to the default queue.

OK, now I understand why I can't "shape" the CARP traffic (assign it to another queue), because there's a quick rule before that assigns the CARP protocol to the default queue… now everything is clearer, thanks! ;)

namezero111111

Thanks for all your replies.
You are correct, Adam. Since those rules catch first they go to the default queue.

Hence my workaround to comment the code out so I can define the rules in the GUI.
I haven't decided whether to do it that way yet or whether I'll path a queue (qAck, qVeryHigh) into functions.inc, but the first way seems cleaner somehow.
Either way should work though.

I suppose those two "hidden" rules were well intentioned, but they do seem to wreak havoc when congested and shaped interfaces come into play.

adam65535

I really wish rules like this were not hidden. Put them in floating and give them a comment that they are built-in system rules for pfsync and carp and maybe allow users to disable them. Hidden rules can become a problem for audits if the user does not know they exist. If there is a rule that allows access to something I honestly feel it should be shown to the user somehow. I feel that even DHCP rules should be shown on the floating tab as a system rule.

For example… the hidden pfsync and carp rules when using carp seem to allow all traffic through the firewall that match because they don't specify an in, out, src, or dst (which users might not know how open those rules are). Immediately I would worry that would allow any traffic to pass not only to your firewall but through your firewall with that protocol. A quick test using hping seems to indicate the traffic is passed through any of the interfaces from any to any ( I modified the builtin rules to log and the logs show pass). If you don't allow any incoming traffic from your WAN or you don't allow outgoing traffic from your DMZ network your security policy would have a hole in it for those protocols if the firewall forwards pfsync protocol or carp protocol packets because of the default rules. I doubt many ISPs will pass that traffic but even with that restriction I doubt many people would really like the idea of someone on their WAN subnet being able to craft packets through their firewall if they use carp features causing those hidden rules to be put in place.

The hidden rules I think really need to be restricted to certain IPs to restrict forwarding.

That is just an example of why I think hidden rules are a bad idea. Out of sight... out of mind.

namezero111111

Hmm yes you're right about that.
This could become problematic, because theoretically someone on a DMZ or WAN on the same subnet would be able to pass malicious CARP / PFSYNC traffic through the firewall to interfere with these protocols on other subnets, no?
Maybe my understanding of PFSYNC is a little too limited, but couldn't someone try to gratuitously inject invalid states into the firewall?