1000x WAN Traffic increase

stephenw10

Massive increase in traffic that appears to be coming from the router itself; immediately suspect you are being used in a DDOS amplification attack. Most likely to be DNS or NTP which should appear as traffic on ports 53 or 123.
What version of pfSense are you running? Do you have DNS or NTP exposed WAN side?

~~You haven't actually shown us the WAN graph~~ and some of that traffic appears to be between two local services perhaps?

Edit: Now I look closely you have shown us that and the traffic is WAN in-pass so it's unlikely a DDOS issue. Snort or HAVP stuck in a loop downloading updates?

Steve

stephenw10

Try looking at Diagnostics: Sockets: (click 'show all') to see what is connecting using those outgoing ports shown in a current pftop table.

Steve

stephenw10

A quick search shows that HAVP uses port 3125 as its default proxy port.
All the large connection sizes you see in the pftop table (WAN to some remote IP) are followed by identically sized transfers between local services one of which is running on port 3125.
I'm going to suggest that this is HAVP downloading updates but obviously it's running far too often for some reason. Check the logs.

178MB over 2 weeks for apinger traffic looks about right by the way.

Steve

MindfulCoyote

@stephenw10:

Edit: Now I look closely you have shown us that and the traffic is WAN in-pass so it's unlikely a DDOS issue. Snort or HAVP stuck in a loop downloading updates?
Steve

That's what I thought at first also, but I with that sort of bandwidth I would think it would show up clearly in pfTop, wouldn't you think?

Newburns, have you considered just rebooting to see if the issue goes away? That might eliminate a stuck process as the cause. (You could always do Packet Capture before the reboot to continue researching the issue in the event the reboot does clear it up.)

stephenw10

I would think that. I'm suggesting it's the 212, 74 and 73MB connections shown. Earlier connections have timed out and arw no longer in the table. Those downloads appear to ge linked to HAVP in some way. I think they're most likely definition updates but since it's a proxy it could be proxying something else I guess.

Steve

newburns

I performed an update to pfSense (Latest Stable now), and, of course, invoked the upgrade with a reboot.
It still appears that the traffic is happening.
I am not afraid of command line, but I am used to Redhat (CentOS) and do not know which commands to run or even what to look for. I can see that it is not happening from my LAN or any other local networks, so it is a matter of the firewall causing the issue.
Should the HAVP Definitions URL be listed under the "Do Not Cache" listing?
I can only see that the 23.67.253.161:80 belongs to the Akamai Netwokr from the whois property. I don't really know what else to look for.
Is it possible that the HAVP definitions are over 1gb?

Attached are the current pfTop, the HAVP settings, and the Sockets view.
I wasn't sure what to do with the packet capture. I have it running on the firewall for that IP host, but it just says "Packet Capture is running." Has been like that for a while.

SIDE NOTE: I don't get any email notifications that someone has posted to this thread. Am I supposed to enable something special?

Diagnostics_pfTop.png_thumb

havp-settings.png_thumb

Diagnostics_Sockets_2.png_thumb

newburns

Thought this may be relevant as well since I see that Windows update has been hosted on the 23.67.253.161 URL
https://www.virustotal.com/en/ip-address/23.67.253.161/information/

Proxy_server_General_settings.png_thumb

MindfulCoyote

@newburns:

I performed an update to pfSense (Latest Stable now), and, of course, invoked the upgrade with a reboot.
It still appears that the traffic is happening.
I am not afraid of command line, but I am used to Redhat (CentOS) and do not know which commands to run or even what to look for. I can see that it is not happening from my LAN or any other local networks, so it is a matter of the firewall causing the issue.
Should the HAVP Definitions URL be listed under the "Do Not Cache" listing?
I can only see that the 23.67.253.161:80 belongs to the Akamai Netwokr from the whois property. I don't really know what else to look for.
Is it possible that the HAVP definitions are over 1gb?

Attached are the current pfTop, the HAVP settings, and the Sockets view.
I wasn't sure what to do with the packet capture. I have it running on the firewall for that IP host, but it just says "Packet Capture is running." Has been like that for a while.

SIDE NOTE: I don't get any email notifications that someone has posted to this thread. Am I supposed to enable something special?

Easiest first:

To get email notifications, click on "Notify" just to the right of the "Reply" button.

The Packet Capture will run until you click stop at which point you'll be able to view a summary or download the raw pcap data.

I think I'm going to try cheating here. How do you feel about adding a WAN rule to block those top IPs in the list to see if the bandwidth drops back to normal? You could just try one IP at a time until the culprit is found. Specifically: 23.67.253.161 and 4.27.11.126 (as long as they aren't known to you…)

Another shortcut is to perhaps try disabling some of those services sequentially to see if they are the cause? Specifically squid and HAVP. The more I think about this, the more inclined I am to agree with stephenw10 - that those two IPs are the culprits and they're just sending <300MB per stream.

(Maybe when you increased the squid cache, it "decided" to try to "fill" it up... or got stuck in some kind of retrieval loop...)

stephenw10

I'm assuming you're running a 'full' install?
Usually when you run an upgrade packages get reinstalled. It's hard to believe that this is some 'stuck in a loop' problem because it contiunes across an upgrade. :-\

You can follow the trail between the pftop table and the sockets list quite clearly though. The problem is I'm not familiar enough with havp to know if this is it's normal behaviour. ::)

Take 184MB transfer, we can see in pftop that the local WAN address downloaded it from 4.27.11.126:80. We also see it was tranfered in both directions between loacalhost services on port 3125 (havp) and port 6053 (?). Then looking down the sockets table we see that at least two of those processes are listed as havp.

I also see that you have at least 4 interfaces. Have you checked the RRD graphs for those other interfaces? Given it's still happening after an update it still points to some internal machine being the culprit here.

Steve

BBcan177

Specifically: 23.67.253.161 and 4.27.11.126

From the first IP, looks like it could be an update process?

I don't have those addresses in my Blocklists and I have alot of them.

https://www.virustotal.com/en/ip-address/23.67.253.161/information

2013-06-28 a4.mzstatic.com
2013-09-14 acs.pandasoftware.com
2013-07-10 aru-akam.oracle.com
2013-09-12 au.v4.download.windowsupdate.com
2013-07-02 ax.itunes.apple.com
2013-07-02 cbsbigbrother-lh.akamaihd.net
2013-06-27 cdn.mysql.com
2013-07-09 csd.aeriagames.com
2013-07-09 d.computerbild.de
2013-09-14 de.download.nvidia.com

https://www.virustotal.com/en/ip-address/4.27.11.126/information/

2013-06-23 a.ligatus.com
2013-06-24 cdn.dli.trymedia.com
2013-06-19 cdn.kaisergames.de
2014-04-30 cdn.ricaud.com
2013-06-18 cdn.royale.spongecell.com
2014-02-20 cdn.static.cyclingnews.com
2014-04-30 cdn.thomascook.com
2013-06-18 cdn2.worldoftanks.com
2012-12-01 conflash.ribob01.net
2013-07-10 dl.wargaming.net

EDIT : Guess I should have read the whole thread before posting a repeat!!

stephenw10

Worth pointing out again though. ;)
Do Snort or HAVP use a CDN to distribute their updates? I didn't think they did. In which case that's further evidence pointing to it being something behind pfSense.

Steve

newburns

It was HAVP
I turned it off, and IMMEDIATELY the traffic stopped.
This is my whitelist, I wonder if that has anything to do with it.

logitech.com/
navisite.net/
.lenovo.com/
.omniti.com/
clamav.net/
sourceforge.net/
70.38.0.134
188.121.46.128
alternate.mtrosemedia.org/*

Also, is it possible that I'm trying to cache all of the virus DB? Not really sure about what I'm talking about, but I don't know why it is still downloading definitions from that URL.

I removed the following from the whitelist, and the problem is gone. Any explanation?
logitech.com/
navisite.net/
.lenovo.com/
.omniti.com/
clamav.net/
sourceforge.net/

MindfulCoyote

@stephenw10:

Worth pointing out again though. ;)
Do Snort or HAVP use a CDN to distribute their updates? I didn't think they did. In which case that's further evidence pointing to it being something behind pfSense.

Steve

Yes, that's my current hypothesis too. I suspect that a LAN client is making a request that squid is trying to cache - repeatedly. So squid (or possibly HAVP) just keeps looping the request over and over but failing to complete the process. If the bandwidth disappears after temporarily deactivating the service that would at least give the OP a place to start looking.

newburns

Sorry, that did not fix the issue. Turning off HAVP fixes the issue, but removing those lines from whitelist did not solve anything, and I didn't think that it would.

MindfulCoyote

@newburns:

but removing those lines from whitelist did not solve anything, and I didn't think that it would.

~~I'm not sure what you mean by removing lines from a whitelist, but it's probably not relevant at this point.~~ Oops.

@newburns:

Sorry, that did not fix the issue. Turning off HAVP fixes the issue,

Ok, excellent. Now you know the source of the trouble, it's HAVP. Have you had a look at the HAVP logs to see if it's reporting errors? If not, we could probably increase it's debug level.

newburns

How do I view the logs for HAVP?
Going to status >> Package Logs show nothing.
System logs does not have anything specific to HAVP

newburns

Also, I added a rule in my firewall, and it doesn't seem to work.
I added the ip
23.67.253.163, .167, .168 to the "Spammer_Hacker" alias
23.57.253.0/24 to the "Spammer_Network" alias
But it doesn't seem to block the traffic.
Attached are my rulesets and my current traffic graph

Rules.PNG_thumb

current_traffic.PNG_thumb

MindfulCoyote

@newburns:

How do I view the logs for HAVP?
Going to status >> Package Logs show nothing.
System logs does not have anything specific to HAVP

From the command prompt/console:

clog /var/log/havp/havp.log

newburns

I'm assuming the "(Bad address)" is when I disabled it.

05/06/2014 11:06:05 === Starting HAVP Version: 0.91
05/06/2014 11:06:05 === Mandatory locking disabled! KEEPBACK settings not used!
05/06/2014 11:06:05 Running as user: havp, group: havp
05/06/2014 11:06:05 --- Initializing Clamd Socket Scanner
05/06/2014 11:06:05 Clamd Socket Scanner passed EICAR virus test (Eicar-Test-Signature)
05/06/2014 11:06:05 --- All scanners initialized
05/06/2014 11:06:05 Process ID: 52553
05/06/2014 11:12:55 === Starting HAVP Version: 0.91
05/06/2014 11:12:55 === Mandatory locking disabled! KEEPBACK settings not used!
05/06/2014 11:12:55 Running as user: havp, group: havp
05/06/2014 11:12:55 --- Initializing Clamd Socket Scanner
05/06/2014 11:12:55 Clamd Socket Scanner passed EICAR virus test (Eicar-Test-Signature)
05/06/2014 11:12:55 --- All scanners initialized
05/06/2014 11:12:55 Process ID: 35010
05/06/2014 11:59:20 === Starting HAVP Version: 0.91
05/06/2014 11:59:20 === Mandatory locking disabled! KEEPBACK settings not used!
05/06/2014 11:59:20 Running as user: havp, group: havp
05/06/2014 11:59:20 --- Initializing Clamd Socket Scanner
05/06/2014 11:59:20 Clamd Socket Scanner passed EICAR virus test (Eicar-Test-Signature)
05/06/2014 11:59:20 --- All scanners initialized
05/06/2014 11:59:20 Process ID: 3913
clog: ERROR: could not write output (Bad address)

MindfulCoyote

@newburns:

Also, I added a rule in my firewall, and it doesn't seem to work.
I added the ip
23.67.253.163, .167, .168 to the "Spammer_Hacker" alias
23.57.253.0/24 to the "Spammer_Network" alias
But it doesn't seem to block the traffic.
Attached are my rulesets and my current traffic graph

It's probably still in the state table. Try: Menu; Diagnostics; Show States; Reset States; "Reset"