UDP Advanced Rules - Under UDP DDoS Attack
-
I see a TON of the VRRP announcments being sent out of pfsense to some other devices on the network. About 50 a second, is this normal?
-
Solved… Except you called it when you said your not sure that it is an attack.
You're never going to believe what happened. I can't even explain it.
Something with CARP caused it to send out so many VRRP announcements that it would take the LAN interface down almost every 15 minutes, but first it slowed everything so much that all the states built up making it look like I was under an attack. I have no idea why UDP states would build like that, and not others.
This firewall has been running without issues for over a year, why all of a sudden would there be a VRRP loop. I suspect that our upstream carrier might have used VRRP on the same vhid causing the loop. In any case, I didn't even have to restart pfsense, I just disabled carp then reenabled carp and the announcements stopped.
Maybe this is worth me opening a seperate topic on just to help others who might have had an issue like this. I appreciate all of your help, you guys sure know your stuff. I really do apologize for waisting your time.
-Fred
-
"About 50 a second, is this normal?"
No I doubt that could ever be considered normal?
So now that you have cleared up your crash issue - are you stilling seeing bursts of dns requests every 10 minutes? I would assume that if your seeing bursts of dns at such a small interval that someone has a ttl too small for the amount of traffic the zone(s) see.
So are these all internal zones that are managed by you or someone else in your company.. Or are they just zones you host for other people? If hosted for other people, so many users do not understand dns in the slightest – very possible to have funked up configurations that cause way too much traffic.
-
Hi Johnpoz: 50/sec was the VRRP announcements being sent out by CARP. When I reset the carp they stopped and all of the problems cleared up, until several hours later when various VIP's started going offline and the announcements started again. I have a feeling the upstream carrier is running VRRP on the same ID's as nothing else was added to that network. This firewall is part of an older network of ours and is segmented from our others so it didnt change at all, no new switches or anything. As jimp mentioned, probably a layer 2 loop.
the odd part of it all, this caused so many strange issues with the states that the UDP states would not close and they kept piling up so it looked like a DDoS when infact it was CARP.
I had to take down CARP all together and use IPAlias instead. Working fine for a year until the other day.
**If anyone else see's Out errors on the lan interface for pfsense yet it's passing traffic but having issues with states building/resetting (causing repeated LAN packet loss/drops every so often), don't assume it's a bad cable or nic. Check your system log and also use tcpdump. You would be surprised at what could happen. :-)**Maybe this will help others too.
-
A VHID conflict wouldn't cause that many CARP advertisements in a short period, only a real layer 2 loop could have done that. CARP advertisements are constant: 1 per second (+skew) per VIP on a segment. You'd either see a constant 50 per second every second with or without a problem, or in the case of a single vhid conflict, you'd only see an extra 2 or so.
But if something is looping/bridging the traffic and it's just turning circles on the network, it would cause exactly the behavior you're seeing. If you are mixing CARP+bridging, that's a very bad thing do to.
-
Hi jimp: This firewall is only for NAT and had CARP runnning for 2 WANs, other then that no special configurations. The constant advertisments only started recently and the CARP IPs became unstable. Some would work, some wouldn't. I agree with your thought about the loop but I'm not sure where or when that would have happened. The only reason I used CARP is when I started out I was on pfsense 1.2.3 and if I can remmeber IPAlias was not available at that point so when I did the upgrade, it was still working and I left it as is. a year later the problems start, something probably gave it a kick. Maybe it was a bad config, but something pushed it. If there was a loop somewhere all along and a VHID conflict occured would that do it?
-
A VHID conflict would not have any impact on the amount of traffic. The number of advertisements would be constant no matter what the case is.
The only side effect of a VHID conflict would be loss of connectivity (or intermittent connectivity) to only the IP involved in the conflict.
Check Interfaces > (assign) on the bridge tab to make sure you don't have any defined, and "ifconfig -a" would show it for sure.
If you have any other servers or equipment that plugs into two separate parts of the network, any of them could cause such a loop.
You can also get some loop-like behavior in certain other cases: http://redmine.pfsense.org/issues/2073
-
I did a ifconfig -a and only showed the physical interfaces and now the _MAGIC/IPAlias interfaces, no bridge or anything.
What you said about the VHIDs, that happened too. Some of the Virtual IPs started going offline, some worked after I changed the VHID ID but later went offline too. IPAlias/disable carp solved the IP problems and all of the other problems with states not closing out, LAN errors and packet loss. As far as I can tell, since shutting down CARP everything is normal. Though, maybe there is still a loop somewhere.
I will have to look through everything to see about that loop. As always, appreciate the help and expertise here.
-
It was much higher yesterday the traffic but I blocked so many networks in China.
This is why on a lot of my rules I only allow IPs registered to North America to pass. I was getting to much junk traffic from China and Russia.
