1000x WAN Traffic increase
-
It seems that a large chunk of the traffic is attributed to ping the Gateway
And then there is an Akamai IP address with equal amount of bytes
-
18mbps of traffic should stand out dramatically if it is from only a few hosts. Since there is nothing obvious in the pfTop output, that leads me to suspect that the traffic is coming from many hosts, perhaps even a botnet. If so, snort ought to be going bananas.
Here's that facts as I interpret them:
- The traffic doesn't appear to be passing out the LAN (or any) port so it's probably being sent directly at the router.
- The traffic is not showing on the graph as "WAN-in-block" so it's likely that it's destination is either an open port or protocol (ICMP?).
At this point I would try using iftop and trafshow to identify the traffic. Failing that, I would use Packet Capture to download a sample into wireshark. If you really want avoid the command line, the ntop package might yield useful results.
It might not hurt to put a call in to your ISP as well to enlist their help. I'm sure they'd prefer not to have unwanted traffic on their network either.
-
Massive increase in traffic that appears to be coming from the router itself; immediately suspect you are being used in a DDOS amplification attack. Most likely to be DNS or NTP which should appear as traffic on ports 53 or 123.
What version of pfSense are you running? Do you have DNS or NTP exposed WAN side?You haven't actually shown us the WAN graphand some of that traffic appears to be between two local services perhaps?Edit: Now I look closely you have shown us that and the traffic is WAN in-pass so it's unlikely a DDOS issue. Snort or HAVP stuck in a loop downloading updates?
Steve
-
Try looking at Diagnostics: Sockets: (click 'show all') to see what is connecting using those outgoing ports shown in a current pftop table.
Steve
-
A quick search shows that HAVP uses port 3125 as its default proxy port.
All the large connection sizes you see in the pftop table (WAN to some remote IP) are followed by identically sized transfers between local services one of which is running on port 3125.
I'm going to suggest that this is HAVP downloading updates but obviously it's running far too often for some reason. Check the logs.178MB over 2 weeks for apinger traffic looks about right by the way.
Steve
-
Edit: Now I look closely you have shown us that and the traffic is WAN in-pass so it's unlikely a DDOS issue. Snort or HAVP stuck in a loop downloading updates?
SteveThat's what I thought at first also, but I with that sort of bandwidth I would think it would show up clearly in pfTop, wouldn't you think?
Newburns, have you considered just rebooting to see if the issue goes away? That might eliminate a stuck process as the cause. (You could always do Packet Capture before the reboot to continue researching the issue in the event the reboot does clear it up.)
-
I would think that. I'm suggesting it's the 212, 74 and 73MB connections shown. Earlier connections have timed out and arw no longer in the table. Those downloads appear to ge linked to HAVP in some way. I think they're most likely definition updates but since it's a proxy it could be proxying something else I guess.
Steve
-
I performed an update to pfSense (Latest Stable now), and, of course, invoked the upgrade with a reboot.
It still appears that the traffic is happening.
I am not afraid of command line, but I am used to Redhat (CentOS) and do not know which commands to run or even what to look for. I can see that it is not happening from my LAN or any other local networks, so it is a matter of the firewall causing the issue.
Should the HAVP Definitions URL be listed under the "Do Not Cache" listing?
I can only see that the 23.67.253.161:80 belongs to the Akamai Netwokr from the whois property. I don't really know what else to look for.
Is it possible that the HAVP definitions are over 1gb?Attached are the current pfTop, the HAVP settings, and the Sockets view.
I wasn't sure what to do with the packet capture. I have it running on the firewall for that IP host, but it just says "Packet Capture is running." Has been like that for a while.SIDE NOTE: I don't get any email notifications that someone has posted to this thread. Am I supposed to enable something special?
-
Thought this may be relevant as well since I see that Windows update has been hosted on the 23.67.253.161 URL
https://www.virustotal.com/en/ip-address/23.67.253.161/information/
-
I performed an update to pfSense (Latest Stable now), and, of course, invoked the upgrade with a reboot.
It still appears that the traffic is happening.
I am not afraid of command line, but I am used to Redhat (CentOS) and do not know which commands to run or even what to look for. I can see that it is not happening from my LAN or any other local networks, so it is a matter of the firewall causing the issue.
Should the HAVP Definitions URL be listed under the "Do Not Cache" listing?
I can only see that the 23.67.253.161:80 belongs to the Akamai Netwokr from the whois property. I don't really know what else to look for.
Is it possible that the HAVP definitions are over 1gb?Attached are the current pfTop, the HAVP settings, and the Sockets view.
I wasn't sure what to do with the packet capture. I have it running on the firewall for that IP host, but it just says "Packet Capture is running." Has been like that for a while.SIDE NOTE: I don't get any email notifications that someone has posted to this thread. Am I supposed to enable something special?
Easiest first:
To get email notifications, click on "Notify" just to the right of the "Reply" button.
The Packet Capture will run until you click stop at which point you'll be able to view a summary or download the raw pcap data.
I think I'm going to try cheating here. How do you feel about adding a WAN rule to block those top IPs in the list to see if the bandwidth drops back to normal? You could just try one IP at a time until the culprit is found. Specifically: 23.67.253.161 and 4.27.11.126 (as long as they aren't known to you…)
Another shortcut is to perhaps try disabling some of those services sequentially to see if they are the cause? Specifically squid and HAVP. The more I think about this, the more inclined I am to agree with stephenw10 - that those two IPs are the culprits and they're just sending <300MB per stream.
(Maybe when you increased the squid cache, it "decided" to try to "fill" it up... or got stuck in some kind of retrieval loop...)
-
I'm assuming you're running a 'full' install?
Usually when you run an upgrade packages get reinstalled. It's hard to believe that this is some 'stuck in a loop' problem because it contiunes across an upgrade. :-\You can follow the trail between the pftop table and the sockets list quite clearly though. The problem is I'm not familiar enough with havp to know if this is it's normal behaviour. ::)
Take 184MB transfer, we can see in pftop that the local WAN address downloaded it from 4.27.11.126:80. We also see it was tranfered in both directions between loacalhost services on port 3125 (havp) and port 6053 (?). Then looking down the sockets table we see that at least two of those processes are listed as havp.
I also see that you have at least 4 interfaces. Have you checked the RRD graphs for those other interfaces? Given it's still happening after an update it still points to some internal machine being the culprit here.
Steve
-
Specifically: 23.67.253.161 and 4.27.11.126
From the first IP, looks like it could be an update process?
I don't have those addresses in my Blocklists and I have alot of them.
https://www.virustotal.com/en/ip-address/23.67.253.161/information
2013-06-28 a4.mzstatic.com
2013-09-14 acs.pandasoftware.com
2013-07-10 aru-akam.oracle.com
2013-09-12 au.v4.download.windowsupdate.com
2013-07-02 ax.itunes.apple.com
2013-07-02 cbsbigbrother-lh.akamaihd.net
2013-06-27 cdn.mysql.com
2013-07-09 csd.aeriagames.com
2013-07-09 d.computerbild.de
2013-09-14 de.download.nvidia.comhttps://www.virustotal.com/en/ip-address/4.27.11.126/information/
2013-06-23 a.ligatus.com
2013-06-24 cdn.dli.trymedia.com
2013-06-19 cdn.kaisergames.de
2014-04-30 cdn.ricaud.com
2013-06-18 cdn.royale.spongecell.com
2014-02-20 cdn.static.cyclingnews.com
2014-04-30 cdn.thomascook.com
2013-06-18 cdn2.worldoftanks.com
2012-12-01 conflash.ribob01.net
2013-07-10 dl.wargaming.netEDIT : Guess I should have read the whole thread before posting a repeat!!
-
Worth pointing out again though. ;)
Do Snort or HAVP use a CDN to distribute their updates? I didn't think they did. In which case that's further evidence pointing to it being something behind pfSense.Steve
-
It was HAVP
I turned it off, and IMMEDIATELY the traffic stopped.
This is my whitelist, I wonder if that has anything to do with it.logitech.com/
navisite.net/
.lenovo.com/
.omniti.com/
clamav.net/
sourceforge.net/
70.38.0.134
188.121.46.128
alternate.mtrosemedia.org/*Also, is it possible that I'm trying to cache all of the virus DB? Not really sure about what I'm talking about, but I don't know why it is still downloading definitions from that URL.
I removed the following from the whitelist, and the problem is gone. Any explanation?
logitech.com/
navisite.net/
.lenovo.com/
.omniti.com/
clamav.net/
sourceforge.net/ -
Worth pointing out again though. ;)
Do Snort or HAVP use a CDN to distribute their updates? I didn't think they did. In which case that's further evidence pointing to it being something behind pfSense.Steve
Yes, that's my current hypothesis too. I suspect that a LAN client is making a request that squid is trying to cache - repeatedly. So squid (or possibly HAVP) just keeps looping the request over and over but failing to complete the process. If the bandwidth disappears after temporarily deactivating the service that would at least give the OP a place to start looking.
-
Sorry, that did not fix the issue. Turning off HAVP fixes the issue, but removing those lines from whitelist did not solve anything, and I didn't think that it would.
-
but removing those lines from whitelist did not solve anything, and I didn't think that it would.
I'm not sure what you mean by removing lines from a whitelist, but it's probably not relevant at this point.Oops.Sorry, that did not fix the issue. Turning off HAVP fixes the issue,
Ok, excellent. Now you know the source of the trouble, it's HAVP. Have you had a look at the HAVP logs to see if it's reporting errors? If not, we could probably increase it's debug level.
-
How do I view the logs for HAVP?
Going to status >> Package Logs show nothing.
System logs does not have anything specific to HAVP -
Also, I added a rule in my firewall, and it doesn't seem to work.
I added the ip
23.67.253.163, .167, .168 to the "Spammer_Hacker" alias
23.57.253.0/24 to the "Spammer_Network" alias
But it doesn't seem to block the traffic.
Attached are my rulesets and my current traffic graph
-
How do I view the logs for HAVP?
Going to status >> Package Logs show nothing.
System logs does not have anything specific to HAVPFrom the command prompt/console:
clog /var/log/havp/havp.log
