1000x WAN Traffic increase

stephenw10

Worth pointing out again though. ;)
Do Snort or HAVP use a CDN to distribute their updates? I didn't think they did. In which case that's further evidence pointing to it being something behind pfSense.

Steve

newburns

It was HAVP
I turned it off, and IMMEDIATELY the traffic stopped.
This is my whitelist, I wonder if that has anything to do with it.

logitech.com/
navisite.net/
.lenovo.com/
.omniti.com/
clamav.net/
sourceforge.net/
70.38.0.134
188.121.46.128
alternate.mtrosemedia.org/*

Also, is it possible that I'm trying to cache all of the virus DB? Not really sure about what I'm talking about, but I don't know why it is still downloading definitions from that URL.

I removed the following from the whitelist, and the problem is gone. Any explanation?
logitech.com/
navisite.net/
.lenovo.com/
.omniti.com/
clamav.net/
sourceforge.net/

MindfulCoyote

@stephenw10:

Worth pointing out again though. ;)
Do Snort or HAVP use a CDN to distribute their updates? I didn't think they did. In which case that's further evidence pointing to it being something behind pfSense.

Steve

Yes, that's my current hypothesis too. I suspect that a LAN client is making a request that squid is trying to cache - repeatedly. So squid (or possibly HAVP) just keeps looping the request over and over but failing to complete the process. If the bandwidth disappears after temporarily deactivating the service that would at least give the OP a place to start looking.

newburns

Sorry, that did not fix the issue. Turning off HAVP fixes the issue, but removing those lines from whitelist did not solve anything, and I didn't think that it would.

MindfulCoyote

@newburns:

but removing those lines from whitelist did not solve anything, and I didn't think that it would.

~~I'm not sure what you mean by removing lines from a whitelist, but it's probably not relevant at this point.~~ Oops.

@newburns:

Sorry, that did not fix the issue. Turning off HAVP fixes the issue,

Ok, excellent. Now you know the source of the trouble, it's HAVP. Have you had a look at the HAVP logs to see if it's reporting errors? If not, we could probably increase it's debug level.

newburns

How do I view the logs for HAVP?
Going to status >> Package Logs show nothing.
System logs does not have anything specific to HAVP

newburns

Also, I added a rule in my firewall, and it doesn't seem to work.
I added the ip
23.67.253.163, .167, .168 to the "Spammer_Hacker" alias
23.57.253.0/24 to the "Spammer_Network" alias
But it doesn't seem to block the traffic.
Attached are my rulesets and my current traffic graph

Rules.PNG_thumb

current_traffic.PNG_thumb

MindfulCoyote

@newburns:

How do I view the logs for HAVP?
Going to status >> Package Logs show nothing.
System logs does not have anything specific to HAVP

From the command prompt/console:

clog /var/log/havp/havp.log

newburns

I'm assuming the "(Bad address)" is when I disabled it.

05/06/2014 11:06:05 === Starting HAVP Version: 0.91
05/06/2014 11:06:05 === Mandatory locking disabled! KEEPBACK settings not used!
05/06/2014 11:06:05 Running as user: havp, group: havp
05/06/2014 11:06:05 --- Initializing Clamd Socket Scanner
05/06/2014 11:06:05 Clamd Socket Scanner passed EICAR virus test (Eicar-Test-Signature)
05/06/2014 11:06:05 --- All scanners initialized
05/06/2014 11:06:05 Process ID: 52553
05/06/2014 11:12:55 === Starting HAVP Version: 0.91
05/06/2014 11:12:55 === Mandatory locking disabled! KEEPBACK settings not used!
05/06/2014 11:12:55 Running as user: havp, group: havp
05/06/2014 11:12:55 --- Initializing Clamd Socket Scanner
05/06/2014 11:12:55 Clamd Socket Scanner passed EICAR virus test (Eicar-Test-Signature)
05/06/2014 11:12:55 --- All scanners initialized
05/06/2014 11:12:55 Process ID: 35010
05/06/2014 11:59:20 === Starting HAVP Version: 0.91
05/06/2014 11:59:20 === Mandatory locking disabled! KEEPBACK settings not used!
05/06/2014 11:59:20 Running as user: havp, group: havp
05/06/2014 11:59:20 --- Initializing Clamd Socket Scanner
05/06/2014 11:59:20 Clamd Socket Scanner passed EICAR virus test (Eicar-Test-Signature)
05/06/2014 11:59:20 --- All scanners initialized
05/06/2014 11:59:20 Process ID: 3913
clog: ERROR: could not write output (Bad address)

MindfulCoyote

@newburns:

Also, I added a rule in my firewall, and it doesn't seem to work.
I added the ip
23.67.253.163, .167, .168 to the "Spammer_Hacker" alias
23.57.253.0/24 to the "Spammer_Network" alias
But it doesn't seem to block the traffic.
Attached are my rulesets and my current traffic graph

It's probably still in the state table. Try: Menu; Diagnostics; Show States; Reset States; "Reset"

MindfulCoyote

[quote]
I'm assuming the "(Bad address)" is when I disabled it.
clog: ERROR: could not write output (Bad address)

[/quote]

My bad, I should have said "cat /var/log/havp/havp.log"

Ok, that log seems reasonable enough. Maybe it's clamav, try:

cat /var/log/clamav/clamav.log

newburns

Resetting the states seems to have done it.
However, it appears that HAVP really isn't doing too much of anything
With my workflow being:
Internet >> Snort >> pfBlocker >> Squidguard >> Squid >> Client
I'm thinking it may be best to uninstall HAVP. There seems to be a lot of issues with it from people on the forums.
I don't believe there are any alternatives

newburns

clamav.log is empty.
Does it need to be running in order for it to generate logs, or are the old logs saved ?

MindfulCoyote

@newburns:

Resetting the states seems to have done it.

Bear in mind that blocking that IP address was only as a temporary diagnostic and will likely prevent HAVP from functioning correctly once, er, it is, er, functioning correctly…

@newburns:

clamav.log is empty.
Does it need to be running in order for it to generate logs, or are the old logs saved ?

Yes, it needs to be running. And to properly diagnose it's error the temporary block(s) should be removed. There may also be additional logs in each directory:

ls /var/log/havp

and

ls /var/log/clamav

stephenw10

Hmm, curiouser and curiouser. ;)

Putting a firewall rules on the WAN interface will not block any traffic that is initiated by HAVP. Firewall rules on WAN only block new incoming connections. If you want to block new outgoing connection, like this, you need to use a floating rule.

Just to be perfectly clear you didn't respond to my question about other interfaces you have. How many interfaces do you have? Have you check the RRD graphs for those interfaces to make sure it's traffic from there?

Steve

newburns

Sorry. I checked the RRD Graph for those interfaces, and none of them were causing the Traffic

You can disregard the WAN2DHCP interface. I was trying to create a Gateway Group for all of my traffic. I have Comcast with a static IP, but apparently the DHCP IP still works as well. Which gives me (2) outbound connections. I was trying to set both as a shared outbound connection for all traffic, giving priority for WANGW, but the deployment did not work out very well. I don't have a good grasp on the workflow.

firewall_summary_1.jpg_thumb

firewall_summary_2.jpg_thumb

firewall_summary_3.jpg_thumb

MindfulCoyote

@stephenw10:

Putting a firewall rules on the WAN interface will not block any traffic that is initiated by HAVP. Firewall rules on WAN only block new incoming connections.

You're not saying that pfSense would allow a two-way connection to be established despite the WAN entry blocking traffic from that IP? That seems counter-intuitive to me. I would have expected the firewall to permit the outbound packets to be sent to the blocked IP but then to block any response coming from the blocked IP. i.e. one-way traffic only. I'd better hit the man pages again…

kpa

@MindfulCoyote:

@stephenw10:

Putting a firewall rules on the WAN interface will not block any traffic that is initiated by HAVP. Firewall rules on WAN only block new incoming connections.

You're not saying that pfSense would allow a two-way connection to be established despite the WAN entry blocking traffic from that IP? That seems counter-intuitive to me. I would have expected the firewall to permit the outbound packets to be sent to the blocked IP but then to block any response coming from the blocked IP. i.e. one-way traffic only. I'd better hit the man pages again…

That's not how stateful tracking works. Pass decisions are made when the first "new" packet is seen. In TCP connections is the initial SYN packet and in UDP or other IP protocols it's the first packet that does not match any existing state. Block rules apply to any packets that are seen but they won't match packets that match an existing state. PfSense allows all outbound connections (as seen from the point of the interface) by default unless you restrict them with floating rules.

MindfulCoyote

@kpa:

@MindfulCoyote:

@stephenw10:

Putting a firewall rules on the WAN interface will not block any traffic that is initiated by HAVP. Firewall rules on WAN only block new incoming connections.

You're not saying that pfSense would allow a two-way connection to be established despite the WAN entry blocking traffic from that IP? That seems counter-intuitive to me. I would have expected the firewall to permit the outbound packets to be sent to the blocked IP but then to block any response coming from the blocked IP. i.e. one-way traffic only. I'd better hit the man pages again…

That's not how stateful tracking works. Pass decisions are made when the first "new" packet is seen. In TCP connections is the initial SYN packet and in UDP or other IP protocols it's the first packet that does not match any existing state. Block rules apply to any packets that are seen but they won't match packets that match an existing state. PfSense allows all outbound connections (as seen from the point of the interface) by default unless you restrict them with floating rules.

I knew firewalls were hard! So it's true that "It ain't ignorance causes so much trouble; it's folks knowing so much that ain't so." I feel quite humbled for misunderstanding such a fundamental attribute of pfSense.

So my instructions to the OP to enter a temporary diagnostic rule should have been:
"Add two floating rules, one to block traffic destined for 23.67.253.161 and one to block traffic originating from 23.67.253.161. Place them at the top of the list, and reset the state table." They would have been processed before any of his whitelists and ensured that internal processes (as well as LAN clients) didn't set up [rule bypassing] states to that destination.

Of course now that I look for the rule vs. state processing order, it appears everywhere:

"When a rule creates state, the first packet matching the rule creates a "state" between the sender and receiver. Now, not only do packets going from the sender to receiver match the state entry and bypass ruleset evaluation, but so do the reply packets from receiver to sender.
http://www.openbsd.org/faq/pf/filter.html

"Keeping state information allows return traffic for all connections we have initiated to pass back to us."
http://home.nuug.no/~peter/pf/en/long-firewall.html

"This state information allows return traffic for those connections to pass back […]"
http://www.freebsd.org/doc/handbook/firewalls-pf.html

"The reply traffic to connections initiated inside your network is automatically allowed back into your
network by the state table."
The Book https://www.pfsense.org/get-support/index.html#gold-membership

"Once traffic is passed on the interface it enters, an entry in the state table is created, which allows through subsequent packets that are part of that connection. "
https://doc.pfsense.org/index.php/Firewall_Rule_Basics

Those statements are absolutes. I had always been mentally completing them with the phrase "… as long as no rules explicitly block the traffic." when in fact the states are processed ahead of all the rules by default. So... now I will proceed to review all my pfSense rules to see if they are actually doing what I thought they were doing.

stephenw10

Yep, stateful firewall. ;)
That's the reason you have to put block rules on ther LAN side to prevent access from clients. Using floating rules you can do a whole lot more but you can also very easily get it wrong! ::) I don't use floating rules unless it's the only way to achieve something.

Steve