1000x WAN Traffic increase

stephenw10

I'm assuming you're running a 'full' install?
Usually when you run an upgrade packages get reinstalled. It's hard to believe that this is some 'stuck in a loop' problem because it contiunes across an upgrade. :-\

You can follow the trail between the pftop table and the sockets list quite clearly though. The problem is I'm not familiar enough with havp to know if this is it's normal behaviour. ::)

Take 184MB transfer, we can see in pftop that the local WAN address downloaded it from 4.27.11.126:80. We also see it was tranfered in both directions between loacalhost services on port 3125 (havp) and port 6053 (?). Then looking down the sockets table we see that at least two of those processes are listed as havp.

I also see that you have at least 4 interfaces. Have you checked the RRD graphs for those other interfaces? Given it's still happening after an update it still points to some internal machine being the culprit here.

Steve

BBcan177

Specifically: 23.67.253.161 and 4.27.11.126

From the first IP, looks like it could be an update process?

I don't have those addresses in my Blocklists and I have alot of them.

https://www.virustotal.com/en/ip-address/23.67.253.161/information

2013-06-28 a4.mzstatic.com
2013-09-14 acs.pandasoftware.com
2013-07-10 aru-akam.oracle.com
2013-09-12 au.v4.download.windowsupdate.com
2013-07-02 ax.itunes.apple.com
2013-07-02 cbsbigbrother-lh.akamaihd.net
2013-06-27 cdn.mysql.com
2013-07-09 csd.aeriagames.com
2013-07-09 d.computerbild.de
2013-09-14 de.download.nvidia.com

https://www.virustotal.com/en/ip-address/4.27.11.126/information/

2013-06-23 a.ligatus.com
2013-06-24 cdn.dli.trymedia.com
2013-06-19 cdn.kaisergames.de
2014-04-30 cdn.ricaud.com
2013-06-18 cdn.royale.spongecell.com
2014-02-20 cdn.static.cyclingnews.com
2014-04-30 cdn.thomascook.com
2013-06-18 cdn2.worldoftanks.com
2012-12-01 conflash.ribob01.net
2013-07-10 dl.wargaming.net

EDIT : Guess I should have read the whole thread before posting a repeat!!

stephenw10

Worth pointing out again though. ;)
Do Snort or HAVP use a CDN to distribute their updates? I didn't think they did. In which case that's further evidence pointing to it being something behind pfSense.

Steve

newburns

It was HAVP
I turned it off, and IMMEDIATELY the traffic stopped.
This is my whitelist, I wonder if that has anything to do with it.

logitech.com/
navisite.net/
.lenovo.com/
.omniti.com/
clamav.net/
sourceforge.net/
70.38.0.134
188.121.46.128
alternate.mtrosemedia.org/*

Also, is it possible that I'm trying to cache all of the virus DB? Not really sure about what I'm talking about, but I don't know why it is still downloading definitions from that URL.

I removed the following from the whitelist, and the problem is gone. Any explanation?
logitech.com/
navisite.net/
.lenovo.com/
.omniti.com/
clamav.net/
sourceforge.net/

MindfulCoyote

@stephenw10:

Worth pointing out again though. ;)
Do Snort or HAVP use a CDN to distribute their updates? I didn't think they did. In which case that's further evidence pointing to it being something behind pfSense.

Steve

Yes, that's my current hypothesis too. I suspect that a LAN client is making a request that squid is trying to cache - repeatedly. So squid (or possibly HAVP) just keeps looping the request over and over but failing to complete the process. If the bandwidth disappears after temporarily deactivating the service that would at least give the OP a place to start looking.

newburns

Sorry, that did not fix the issue. Turning off HAVP fixes the issue, but removing those lines from whitelist did not solve anything, and I didn't think that it would.

MindfulCoyote

@newburns:

but removing those lines from whitelist did not solve anything, and I didn't think that it would.

~~I'm not sure what you mean by removing lines from a whitelist, but it's probably not relevant at this point.~~ Oops.

@newburns:

Sorry, that did not fix the issue. Turning off HAVP fixes the issue,

Ok, excellent. Now you know the source of the trouble, it's HAVP. Have you had a look at the HAVP logs to see if it's reporting errors? If not, we could probably increase it's debug level.

newburns

How do I view the logs for HAVP?
Going to status >> Package Logs show nothing.
System logs does not have anything specific to HAVP

newburns

Also, I added a rule in my firewall, and it doesn't seem to work.
I added the ip
23.67.253.163, .167, .168 to the "Spammer_Hacker" alias
23.57.253.0/24 to the "Spammer_Network" alias
But it doesn't seem to block the traffic.
Attached are my rulesets and my current traffic graph

Rules.PNG_thumb

current_traffic.PNG_thumb

MindfulCoyote

@newburns:

How do I view the logs for HAVP?
Going to status >> Package Logs show nothing.
System logs does not have anything specific to HAVP

From the command prompt/console:

clog /var/log/havp/havp.log

newburns

I'm assuming the "(Bad address)" is when I disabled it.

05/06/2014 11:06:05 === Starting HAVP Version: 0.91
05/06/2014 11:06:05 === Mandatory locking disabled! KEEPBACK settings not used!
05/06/2014 11:06:05 Running as user: havp, group: havp
05/06/2014 11:06:05 --- Initializing Clamd Socket Scanner
05/06/2014 11:06:05 Clamd Socket Scanner passed EICAR virus test (Eicar-Test-Signature)
05/06/2014 11:06:05 --- All scanners initialized
05/06/2014 11:06:05 Process ID: 52553
05/06/2014 11:12:55 === Starting HAVP Version: 0.91
05/06/2014 11:12:55 === Mandatory locking disabled! KEEPBACK settings not used!
05/06/2014 11:12:55 Running as user: havp, group: havp
05/06/2014 11:12:55 --- Initializing Clamd Socket Scanner
05/06/2014 11:12:55 Clamd Socket Scanner passed EICAR virus test (Eicar-Test-Signature)
05/06/2014 11:12:55 --- All scanners initialized
05/06/2014 11:12:55 Process ID: 35010
05/06/2014 11:59:20 === Starting HAVP Version: 0.91
05/06/2014 11:59:20 === Mandatory locking disabled! KEEPBACK settings not used!
05/06/2014 11:59:20 Running as user: havp, group: havp
05/06/2014 11:59:20 --- Initializing Clamd Socket Scanner
05/06/2014 11:59:20 Clamd Socket Scanner passed EICAR virus test (Eicar-Test-Signature)
05/06/2014 11:59:20 --- All scanners initialized
05/06/2014 11:59:20 Process ID: 3913
clog: ERROR: could not write output (Bad address)

MindfulCoyote

@newburns:

Also, I added a rule in my firewall, and it doesn't seem to work.
I added the ip
23.67.253.163, .167, .168 to the "Spammer_Hacker" alias
23.57.253.0/24 to the "Spammer_Network" alias
But it doesn't seem to block the traffic.
Attached are my rulesets and my current traffic graph

It's probably still in the state table. Try: Menu; Diagnostics; Show States; Reset States; "Reset"

MindfulCoyote

[quote]
I'm assuming the "(Bad address)" is when I disabled it.
clog: ERROR: could not write output (Bad address)

[/quote]

My bad, I should have said "cat /var/log/havp/havp.log"

Ok, that log seems reasonable enough. Maybe it's clamav, try:

cat /var/log/clamav/clamav.log

newburns

Resetting the states seems to have done it.
However, it appears that HAVP really isn't doing too much of anything
With my workflow being:
Internet >> Snort >> pfBlocker >> Squidguard >> Squid >> Client
I'm thinking it may be best to uninstall HAVP. There seems to be a lot of issues with it from people on the forums.
I don't believe there are any alternatives

newburns

clamav.log is empty.
Does it need to be running in order for it to generate logs, or are the old logs saved ?

MindfulCoyote

@newburns:

Resetting the states seems to have done it.

Bear in mind that blocking that IP address was only as a temporary diagnostic and will likely prevent HAVP from functioning correctly once, er, it is, er, functioning correctly…

@newburns:

clamav.log is empty.
Does it need to be running in order for it to generate logs, or are the old logs saved ?

Yes, it needs to be running. And to properly diagnose it's error the temporary block(s) should be removed. There may also be additional logs in each directory:

ls /var/log/havp

and

ls /var/log/clamav

stephenw10

Hmm, curiouser and curiouser. ;)

Putting a firewall rules on the WAN interface will not block any traffic that is initiated by HAVP. Firewall rules on WAN only block new incoming connections. If you want to block new outgoing connection, like this, you need to use a floating rule.

Just to be perfectly clear you didn't respond to my question about other interfaces you have. How many interfaces do you have? Have you check the RRD graphs for those interfaces to make sure it's traffic from there?

Steve

newburns

Sorry. I checked the RRD Graph for those interfaces, and none of them were causing the Traffic

You can disregard the WAN2DHCP interface. I was trying to create a Gateway Group for all of my traffic. I have Comcast with a static IP, but apparently the DHCP IP still works as well. Which gives me (2) outbound connections. I was trying to set both as a shared outbound connection for all traffic, giving priority for WANGW, but the deployment did not work out very well. I don't have a good grasp on the workflow.

firewall_summary_1.jpg_thumb

firewall_summary_2.jpg_thumb

firewall_summary_3.jpg_thumb

MindfulCoyote

@stephenw10:

Putting a firewall rules on the WAN interface will not block any traffic that is initiated by HAVP. Firewall rules on WAN only block new incoming connections.

You're not saying that pfSense would allow a two-way connection to be established despite the WAN entry blocking traffic from that IP? That seems counter-intuitive to me. I would have expected the firewall to permit the outbound packets to be sent to the blocked IP but then to block any response coming from the blocked IP. i.e. one-way traffic only. I'd better hit the man pages again…

kpa

@MindfulCoyote:

@stephenw10:

Putting a firewall rules on the WAN interface will not block any traffic that is initiated by HAVP. Firewall rules on WAN only block new incoming connections.

You're not saying that pfSense would allow a two-way connection to be established despite the WAN entry blocking traffic from that IP? That seems counter-intuitive to me. I would have expected the firewall to permit the outbound packets to be sent to the blocked IP but then to block any response coming from the blocked IP. i.e. one-way traffic only. I'd better hit the man pages again…

That's not how stateful tracking works. Pass decisions are made when the first "new" packet is seen. In TCP connections is the initial SYN packet and in UDP or other IP protocols it's the first packet that does not match any existing state. Block rules apply to any packets that are seen but they won't match packets that match an existing state. PfSense allows all outbound connections (as seen from the point of the interface) by default unless you restrict them with floating rules.