1000x WAN Traffic increase



  • My WAN traffic has went from a 18mb spike every other week, to a constant 20mb/s usage.
    I increased my squid cache to 200gb, but I don't think that will do it.
    How can I see what is generating this amount of traffic?

    Attached are images of current increased traffic, previous stable traffic, current squid settings.
    I am also using Snort ID/S, HAVP Antivirus, and pfBlocker.
    If I can find out what is using the traffic, that would be great.








  • You could start with the LAN traffic graph to see what IP is generating the load.



  • I guess I should have added that.
    Here is my Complete weekly Graph Report with the throughput and LAN Traffic.

    I'm at a loss to find out what is using so much traffic… Virus?






  • This is a very interesting issue. The graphs appear to indicate that the router itself is the destination for all of the traffic. If you look at Menu; Diagnostics; pfTop, you should be able to see the IP address(es) where the traffic is coming from. (They'll likely be at the top of the list since it's sorted by total bytes.)

    I should add that onceyou know the source IP address(es) you can look in Menu; Diagnostics; Show States for the actual connections and also in Menu; Status; System Logs to if the address(es) appear there.

    If you're comfortable with the command line, two very useful utilities in this sort of analysis are iftop and trafshow: https://doc.pfsense.org/index.php/How_can_I_monitor_bandwidth_usage



  • It seems that a large chunk of the traffic is attributed to ping the Gateway
    And then there is an Akamai IP address with equal amount of bytes




  • 18mbps of traffic should stand out dramatically if it is from only a few hosts. Since there is nothing obvious in the pfTop output, that leads me to suspect that the traffic is coming from many hosts, perhaps even a botnet. If so, snort ought to be going bananas.

    Here's that facts as I interpret them:

    • The traffic doesn't appear to be passing out the LAN (or any) port so it's probably being sent directly at the router.
    • The traffic is not showing on the graph as "WAN-in-block" so it's likely that it's destination is either an open port or  protocol (ICMP?).

    At this point I would try using iftop and trafshow to identify the traffic. Failing that, I would use Packet Capture to download a sample into wireshark. If you really want avoid the command line, the ntop package might yield useful results.

    It might not hurt to put a call in to your ISP as well to enlist their help. I'm sure they'd prefer not to have unwanted traffic on their network either.


  • Netgate Administrator

    Massive increase in traffic that appears to be coming from the router itself; immediately suspect you are being used in a DDOS amplification attack. Most likely to be DNS or NTP which should appear as traffic on ports 53 or 123.
    What version of pfSense are you running? Do you have DNS or NTP exposed WAN side?

    You haven't actually shown us the WAN graph and some of that traffic appears to be between two local services perhaps?

    Edit: Now I look closely you have shown us that and the traffic is WAN in-pass so it's unlikely a DDOS issue. Snort or HAVP stuck in a loop downloading updates?

    Steve


  • Netgate Administrator

    Try looking at Diagnostics: Sockets: (click 'show all') to see what is connecting using those outgoing ports shown in a current pftop table.

    Steve


  • Netgate Administrator

    A quick search shows that HAVP uses port 3125 as its default proxy port.
    All the large connection sizes you see in the pftop table (WAN to some remote IP) are followed by identically sized transfers between local services one of which is running on port 3125.
    I'm going to suggest that this is HAVP downloading updates but obviously it's running far too often for some reason. Check the logs.

    178MB over 2 weeks for apinger traffic looks about right by the way.

    Steve



  • @stephenw10:

    Edit: Now I look closely you have shown us that and the traffic is WAN in-pass so it's unlikely a DDOS issue. Snort or HAVP stuck in a loop downloading updates?
    Steve

    That's what I thought  at first also, but I with that sort of bandwidth I would think it would show up clearly in pfTop, wouldn't you think?

    Newburns, have you considered just rebooting to see if the issue goes away? That might eliminate a stuck process as the cause. (You could always do Packet Capture before the reboot to continue researching the issue in the event the reboot does clear it up.)


  • Netgate Administrator

    I would think that. I'm suggesting it's the 212, 74 and 73MB connections shown. Earlier connections have timed out and arw no longer in the table. Those downloads appear to ge linked to HAVP in some way. I think they're most likely definition updates but since it's a proxy it could be proxying something else I guess.

    Steve



  • I performed an update to pfSense (Latest Stable now), and, of course, invoked the upgrade with a reboot.
    It still appears that the traffic is happening.
    I am not afraid of command line, but I am used to Redhat (CentOS) and do not know which commands to run or even what to look for. I can see that it is not happening from my LAN or any other local networks, so it is a matter of the firewall causing the issue.
    Should the HAVP Definitions URL be listed under the "Do Not Cache" listing?
    I can only see that the 23.67.253.161:80 belongs to the Akamai Netwokr from the whois property. I don't really know what else to look for.
    Is it possible that the HAVP definitions are over 1gb?

    Attached are the current pfTop, the HAVP settings, and the Sockets view.
    I wasn't sure what to do with the packet capture. I have it running on the firewall for that IP host, but it just says "Packet Capture is running." Has been like that for a while.

    SIDE NOTE: I don't get any email notifications that someone has posted to this thread. Am I supposed to enable something special?








  • Thought this may be relevant as well since I see that Windows update has been hosted on the 23.67.253.161 URL
    https://www.virustotal.com/en/ip-address/23.67.253.161/information/




  • @newburns:

    I performed an update to pfSense (Latest Stable now), and, of course, invoked the upgrade with a reboot.
    It still appears that the traffic is happening.
    I am not afraid of command line, but I am used to Redhat (CentOS) and do not know which commands to run or even what to look for. I can see that it is not happening from my LAN or any other local networks, so it is a matter of the firewall causing the issue.
    Should the HAVP Definitions URL be listed under the "Do Not Cache" listing?
    I can only see that the 23.67.253.161:80 belongs to the Akamai Netwokr from the whois property. I don't really know what else to look for.
    Is it possible that the HAVP definitions are over 1gb?

    Attached are the current pfTop, the HAVP settings, and the Sockets view.
    I wasn't sure what to do with the packet capture. I have it running on the firewall for that IP host, but it just says "Packet Capture is running." Has been like that for a while.

    SIDE NOTE: I don't get any email notifications that someone has posted to this thread. Am I supposed to enable something special?

    Easiest first:

    To get email notifications, click on "Notify" just to the right of the "Reply" button.

    The Packet Capture will run until you click stop at which point you'll be able to view a summary or download the raw pcap data.

    I think I'm going to try cheating here. How do you feel about adding a WAN rule to block those top IPs in the list to see if the bandwidth drops back to normal? You could just try one IP at a time until the culprit is found. Specifically: 23.67.253.161 and 4.27.11.126 (as long as they aren't known to you…)

    Another shortcut is to perhaps try disabling some of those services sequentially to see if they are the cause? Specifically squid and HAVP. The more I think about this, the more inclined I am to agree with stephenw10 - that those two IPs are the culprits and they're just sending <300MB per stream.

    (Maybe when you increased the squid cache, it "decided" to try to "fill" it up... or got stuck in some kind of retrieval loop...)


  • Netgate Administrator

    I'm assuming you're running a 'full' install?
    Usually when you run an upgrade packages get reinstalled. It's hard to believe that this is some 'stuck in a loop' problem because it contiunes across an upgrade.  :-\

    You can follow the trail between the pftop table and the sockets list quite clearly though. The problem is I'm not familiar enough with havp to know if this is it's normal behaviour.  ::)

    Take 184MB transfer, we can see in pftop that the local WAN address downloaded it from 4.27.11.126:80. We also see it was tranfered in both directions between loacalhost services on port 3125 (havp) and port 6053 (?). Then looking down the sockets table we see that at least two of those processes are listed as havp.

    I also see that you have at least 4 interfaces. Have you checked the RRD graphs for those other interfaces? Given it's still happening after an update it still points to some internal machine being  the culprit here.

    Steve


  • Moderator

    Specifically: 23.67.253.161 and 4.27.11.126

    From the first IP, looks like it could be an update process?

    I don't have those addresses in my Blocklists and I have alot of them.

    https://www.virustotal.com/en/ip-address/23.67.253.161/information

    2013-06-28 a4.mzstatic.com
    2013-09-14 acs.pandasoftware.com
    2013-07-10 aru-akam.oracle.com
    2013-09-12 au.v4.download.windowsupdate.com
    2013-07-02 ax.itunes.apple.com
    2013-07-02 cbsbigbrother-lh.akamaihd.net
    2013-06-27 cdn.mysql.com
    2013-07-09 csd.aeriagames.com
    2013-07-09 d.computerbild.de
    2013-09-14 de.download.nvidia.com

    https://www.virustotal.com/en/ip-address/4.27.11.126/information/

    2013-06-23 a.ligatus.com
    2013-06-24 cdn.dli.trymedia.com
    2013-06-19 cdn.kaisergames.de
    2014-04-30 cdn.ricaud.com
    2013-06-18 cdn.royale.spongecell.com
    2014-02-20 cdn.static.cyclingnews.com
    2014-04-30 cdn.thomascook.com
    2013-06-18 cdn2.worldoftanks.com
    2012-12-01 conflash.ribob01.net
    2013-07-10 dl.wargaming.net

    EDIT : Guess I should have read the whole thread before posting a repeat!!


  • Netgate Administrator

    Worth pointing out again though.  ;)
    Do Snort or HAVP use a CDN to distribute their updates? I didn't think they did. In which case that's further evidence pointing to it being something behind pfSense.

    Steve



  • It was HAVP
    I turned it off, and IMMEDIATELY the traffic stopped.
    This is my whitelist, I wonder if that has anything to do with it.

    logitech.com/
    navisite.net/
    .lenovo.com/
    .omniti.com/
    clamav.net/
    sourceforge.net/
    70.38.0.134
    188.121.46.128
    alternate.mtrosemedia.org/*

    Also, is it possible that I'm trying to cache all of the virus DB? Not really sure about what I'm talking about, but I don't know why it is still downloading definitions from that URL.

    I removed the following from the whitelist, and the problem is gone. Any explanation?
    logitech.com/
    navisite.net/
    .lenovo.com/
    .omniti.com/
    clamav.net/
    sourceforge.net/



  • @stephenw10:

    Worth pointing out again though.  ;)
    Do Snort or HAVP use a CDN to distribute their updates? I didn't think they did. In which case that's further evidence pointing to it being something behind pfSense.

    Steve

    Yes, that's my current hypothesis too. I suspect that a LAN client is making a request that squid is trying to cache - repeatedly. So squid (or possibly HAVP) just keeps looping the request over and over but failing to complete the process. If the bandwidth disappears after temporarily deactivating the service that would at least give the OP a place to start looking.



  • Sorry, that did not fix the issue. Turning off HAVP fixes the issue, but removing those lines from whitelist did not solve anything, and I didn't think that it would.



  • @newburns:

    but removing those lines from whitelist did not solve anything, and I didn't think that it would.

    I'm not sure what you mean by removing lines from a whitelist, but it's probably not relevant at this point. Oops.

    @newburns:

    Sorry, that did not fix the issue. Turning off HAVP fixes the issue,

    Ok, excellent. Now you know the source of the trouble, it's HAVP. Have you had a look at the HAVP logs to see if it's reporting errors? If not, we could probably increase it's debug level.



  • How do I view the logs for HAVP?
    Going to status >> Package Logs show nothing.
    System logs does not have anything specific to HAVP



  • Also, I added a rule in my firewall, and it doesn't seem to work.
    I added the ip
    23.67.253.163, .167, .168 to the "Spammer_Hacker" alias
    23.57.253.0/24 to the "Spammer_Network" alias
    But it doesn't seem to block the traffic.
    Attached are my rulesets and my current traffic graph






  • @newburns:

    How do I view the logs for HAVP?
    Going to status >> Package Logs show nothing.
    System logs does not have anything specific to HAVP

    From the command prompt/console:

    clog /var/log/havp/havp.log



  • I'm assuming the "(Bad address)" is when I disabled it.

    05/06/2014 11:06:05 === Starting HAVP Version: 0.91
    05/06/2014 11:06:05 === Mandatory locking disabled! KEEPBACK settings not used!
    05/06/2014 11:06:05 Running as user: havp, group: havp
    05/06/2014 11:06:05 --- Initializing Clamd Socket Scanner
    05/06/2014 11:06:05 Clamd Socket Scanner passed EICAR virus test (Eicar-Test-Signature)
    05/06/2014 11:06:05 --- All scanners initialized
    05/06/2014 11:06:05 Process ID: 52553
    05/06/2014 11:12:55 === Starting HAVP Version: 0.91
    05/06/2014 11:12:55 === Mandatory locking disabled! KEEPBACK settings not used!
    05/06/2014 11:12:55 Running as user: havp, group: havp
    05/06/2014 11:12:55 --- Initializing Clamd Socket Scanner
    05/06/2014 11:12:55 Clamd Socket Scanner passed EICAR virus test (Eicar-Test-Signature)
    05/06/2014 11:12:55 --- All scanners initialized
    05/06/2014 11:12:55 Process ID: 35010
    05/06/2014 11:59:20 === Starting HAVP Version: 0.91
    05/06/2014 11:59:20 === Mandatory locking disabled! KEEPBACK settings not used!
    05/06/2014 11:59:20 Running as user: havp, group: havp
    05/06/2014 11:59:20 --- Initializing Clamd Socket Scanner
    05/06/2014 11:59:20 Clamd Socket Scanner passed EICAR virus test (Eicar-Test-Signature)
    05/06/2014 11:59:20 --- All scanners initialized
    05/06/2014 11:59:20 Process ID: 3913
    clog: ERROR: could not write output (Bad address)
    
    


  • @newburns:

    Also, I added a rule in my firewall, and it doesn't seem to work.
    I added the ip
    23.67.253.163, .167, .168 to the "Spammer_Hacker" alias
    23.57.253.0/24 to the "Spammer_Network" alias
    But it doesn't seem to block the traffic.
    Attached are my rulesets and my current traffic graph

    It's probably still in the state table. Try: Menu; Diagnostics; Show States; Reset States; "Reset"



  • [quote]
    I'm assuming the "(Bad address)" is when I disabled it.
    clog: ERROR: could not write output (Bad address)
    
    [/quote]
    
    My bad, I should have said "cat /var/log/havp/havp.log"
    
    Ok, that log seems reasonable enough. Maybe it's clamav, try:
    
    cat /var/log/clamav/clamav.log
    


  • Resetting the states seems to have done it.
    However, it appears that HAVP really isn't doing too much of anything
    With my workflow being:
    Internet >> Snort >> pfBlocker >> Squidguard >> Squid >> Client
    I'm thinking it may be best to uninstall HAVP. There seems to be a lot of issues with it from people on the forums.
    I don't believe there are any alternatives



  • clamav.log is empty.
    Does it need to be running in order for it to generate logs, or are the old logs saved ?



  • @newburns:

    Resetting the states seems to have done it.

    Bear in mind that blocking that IP address was only as a temporary diagnostic and will likely prevent HAVP from functioning correctly once, er, it is, er, functioning correctly…

    @newburns:

    clamav.log is empty.
    Does it need to be running in order for it to generate logs, or are the old logs saved ?

    Yes, it needs to be running. And to properly diagnose it's error the temporary block(s) should be removed.  There may also be additional logs in each directory:

    ls /var/log/havp

    and

    ls /var/log/clamav


  • Netgate Administrator

    Hmm, curiouser and curiouser.  ;)

    Putting a firewall rules on the WAN interface will not block any traffic that is initiated by HAVP. Firewall rules on WAN only block new incoming connections. If you want to block new outgoing connection, like this, you need to use a floating rule.

    Just to be perfectly clear you didn't respond to my question about other interfaces you have. How many interfaces do you have? Have you check the RRD graphs for those interfaces to make sure it's traffic from there?

    Steve



  • Sorry. I checked the RRD Graph for those interfaces, and none of them were causing the Traffic

    You can disregard the WAN2DHCP interface. I was trying to create a Gateway Group for all of my traffic. I have Comcast with a static IP, but apparently the DHCP IP still works as well. Which gives me (2) outbound connections. I was trying to set both as a shared outbound connection for all traffic, giving priority for WANGW, but the deployment did not work out very well. I don't have a good grasp on the workflow.








  • @stephenw10:

    Putting a firewall rules on the WAN interface will not block any traffic that is initiated by HAVP. Firewall rules on WAN only block new incoming connections.

    You're not saying that pfSense would allow a two-way connection to be established despite the WAN entry blocking traffic from that IP? That seems counter-intuitive to me. I would have expected the firewall to permit the outbound packets to be sent to the blocked IP but then to block any response coming from the blocked IP.  i.e. one-way traffic only. I'd better hit the man pages again…



  • @MindfulCoyote:

    @stephenw10:

    Putting a firewall rules on the WAN interface will not block any traffic that is initiated by HAVP. Firewall rules on WAN only block new incoming connections.

    You're not saying that pfSense would allow a two-way connection to be established despite the WAN entry blocking traffic from that IP? That seems counter-intuitive to me. I would have expected the firewall to permit the outbound packets to be sent to the blocked IP but then to block any response coming from the blocked IP.  i.e. one-way traffic only. I'd better hit the man pages again…

    That's not how stateful tracking works. Pass decisions are made when the first "new" packet is seen. In TCP connections is the initial SYN packet and in UDP or other IP protocols it's the first packet that does not match any existing state. Block rules apply to any packets that are seen but they won't match packets that match an existing state. PfSense allows all outbound connections (as seen from the point of the interface) by default unless you restrict them with floating rules.



  • @kpa:

    @MindfulCoyote:

    @stephenw10:

    Putting a firewall rules on the WAN interface will not block any traffic that is initiated by HAVP. Firewall rules on WAN only block new incoming connections.

    You're not saying that pfSense would allow a two-way connection to be established despite the WAN entry blocking traffic from that IP? That seems counter-intuitive to me. I would have expected the firewall to permit the outbound packets to be sent to the blocked IP but then to block any response coming from the blocked IP.  i.e. one-way traffic only. I'd better hit the man pages again…

    That's not how stateful tracking works. Pass decisions are made when the first "new" packet is seen. In TCP connections is the initial SYN packet and in UDP or other IP protocols it's the first packet that does not match any existing state. Block rules apply to any packets that are seen but they won't match packets that match an existing state. PfSense allows all outbound connections (as seen from the point of the interface) by default unless you restrict them with floating rules.

    I knew firewalls were hard! So it's true that "It ain't ignorance causes so much trouble; it's folks knowing so much that ain't so." I feel quite humbled for misunderstanding such a fundamental attribute of pfSense.

    So my instructions to the OP to enter a temporary diagnostic rule should have been:
    "Add two floating rules, one to block traffic destined for 23.67.253.161 and one to block traffic originating from 23.67.253.161. Place them at the top of the list, and reset the state table." They would have been processed before any of his whitelists and ensured that internal processes (as well as LAN clients) didn't set up [rule bypassing] states to that destination.

    Of course now that I look for the rule vs. state processing order, it appears everywhere:

    "When a rule creates state, the first packet matching the rule creates a "state" between the sender and receiver. Now, not only do packets going from the sender to receiver match the state entry and bypass ruleset evaluation, but so do the reply packets from receiver to sender.
    http://www.openbsd.org/faq/pf/filter.html

    "Keeping state information allows return traffic for all connections we have initiated to pass back to us."
    http://home.nuug.no/~peter/pf/en/long-firewall.html

    "This state information allows return traffic for those connections to pass back […]"
    http://www.freebsd.org/doc/handbook/firewalls-pf.html

    "The reply traffic to connections initiated inside your network is automatically allowed back into your
    network by the state table."
    The Book https://www.pfsense.org/get-support/index.html#gold-membership

    "Once traffic is passed on the interface it enters, an entry in the state table is created, which allows through subsequent packets that are part of that connection. "
    https://doc.pfsense.org/index.php/Firewall_Rule_Basics

    Those statements are absolutes. I had always been mentally completing them with the phrase "… as long as no rules explicitly block the traffic." when in fact the states are processed ahead of all the rules by default. So... now I will proceed to review all my pfSense rules to see if they are actually doing what I thought they were doing.


  • Netgate Administrator

    Yep, stateful firewall.  ;)
    That's the reason you have to put block rules on ther LAN side to prevent access from clients. Using floating rules you can do a whole lot more but you can also very easily get it wrong!  ::) I don't use floating rules unless it's the only way to achieve something.

    Steve



  • That's the reason you have to put block rules on ther LAN side to prevent access from clients

    So if I have a separate vlan network already setup and functioning, say DEVELOPER. But I didn't want it to have access to the LAN, I would have to add the firewall rule to the LAN or to the DEVELOPER?
    I have been adding rules to the originating network, so to create a sort of separate DMZ type of VLAN, I added rules to that network blocking all access to other networks, then adding a pass all filter at the bottom to allow internet access.
    Your statement reads to me that I would need to add a block filter on each of the networks for that DMZ network, is that true?


  • Netgate Administrator

    You would add it to the DEVELOPER interface.
    All firewall rules (except floating) act on traffic going into the pfSense box. All traffic coming out of the pfSense box is allowed by default.
    Looks to me like you're doing it right. Easy enough to lest the rules though, which I assume you're already doing.  ;)

    Steve



  • Is it beneficial to continue troubleshooting the HVAP issue?
    I have turned it off and removed the custom rules from Squid Proxy.
    It seems that there has been many issues with HAVP, and any new development and/or bug reports will not be fixed in the package.
    I can replace the Squid Proxy custom rules and turn it back on to try and generate the logs, but if it will not help, I don't think that it benefits anyone. Also, how do I even know that it is working and scanning all downloaded images and exe like it states. There isn't any verbosity except when a virus is downloaded. And I'm not sure how to even test that.
    Anytime I download something that is blocked, it is usually SNORT that blocks it, although I am presented with an HVAP error page. HVAP has not blocked anything before, or blocked anything and logged it to me.



  • @newburns:

    Is it beneficial to continue troubleshooting the HVAP issue?

    Well, I'm thinking that your initial concern is now resolved - the WAN traffic has been identified and stopped. If you want to keep working on the HVAP issue(s) perhaps start a new topic in the Packages forum? https://forum.pfsense.org/index.php?board=15.0


Log in to reply