How to find spambot? Got network abuse report from my ISP



  • I got a mail from my ISP with a network abuse report.
    There are 3282 spam mails reported from my IP between Dec 3, 2019 3:04am and Dec 4, 2019 10:22am.

    They look like:

        ip 176.199.xxx.xxx
        send_date 2019-12-04T09:22:15Z
        received_date 2019-12-04T09:22:15Z
        format marf
    
    Incident part
    
        feedback-type: abuse
        user-agent: abusix-nifi/1.0
        version: 1
        source_port: 33431
        source-port: 33431
        original-rcpt-to: www.tiaanxxxxxxxx.09@gmail.com
        source-ip: 176.199.xxx.xxx
        original-envelope-id: A413E514-A0CF-4908-93E8-388A2D262B4E.5
        arrival-date: Wed, 4 Dec 2019 09:22:11 +0000
        original-mail-from: pqyc55@abusix.invalid
    
    Evidence part
    
        authentication-results: example.me; auth=pass (plain)
        received: from ip-176-199-xxx-xxx.hsi06.unitymediagroup.de (ip-176-199-xxx-xxx.hsi06.unitymediagroup.de [176.199.xxx.xxx]) by example.me (Haraka/2.8.25) with ESMTPSA id A413E514-A0CF-4908-93E8-388A2D262B4E.5 envelope-from <pqyc55@abusix.invalid> (authenticated bits=0) (cipher=ECDHE-RSA-AES128-GCM-SHA256); Wed, 04 Dec 2019 09:22:10 +0000
        date: Wed, 04 Dec 2019 09:22:10 +0000
        from: Tonia <pqyc55@abusix.invalid>
        to: www.tiaanxxxxxxxx.09@gmail.com
        content-disposition: attachment; filename="dcq25tte.jpg"
        x-attachment-id: ubsf7esc94gf
        content-id: <ubsf7esc94gf>
    

    In the Snort alerts i have many "141:1 (IMAP) Unknown IMAP4 command"
    from 2019-12-03 04:54:11 to 2019-12-04 12:17:55 with my IP as source and different destination IPs.
    Has it to do with the spam?

    I don't use IMAP.

    The thing is that i did a lot of stuff the last days.
    Since monday i have bridge mode from my ISP (before it was double NAT)...
    I moved pfSense from Proxmox to Hyper-V...
    I did play with different OSs on RPi and Odroid...
    Bought a Fire TV stick...

    Is there a way to find out what was spamming?



  • @MrGlasspoole said in How to find spambot? Got network abuse report from my ISP:

    There are 3282 spam mails reported from my IP ...

    So you have a quiet busy spammer running on a device on your network. Installed some really 'free' software lately ?

    Is there a way to find out what was spamming?

    Simple, a device on your network !!
    Well, you know it's a mail going out. Only 3 ports are used to send mail : 25, 587, 465 - all TCP.
    Block them all on you LAN, and make the block rule to log.
    Check the log, and you will know what device (= IP) is hammering.
    Disconnect that device and your ISP will be happy, also the other 3282 guys that received pure rubbish from you ...
    ( and/or clean the device and have a talk with the owner ).

    Remove snort. It should be there to warn you way before your ISP. It isn't working for you so pretty useless ^^



  • I'm not sure if i'm doing the blocking right.
    I have a lot of this in this log:
    1.jpg

    But i don't see a LAN IP?
    Is it the right place to look?



  • Euuuh, I said :

    @Gertjan said in How to find spambot? Got network abuse report from my ISP:

    Block them all on you LAN,

    Something like this :

    4b32212e-1a92-43b3-9e44-2c8a284bf2e5-image.png

    edit : the alias "mail_ports" is an 'ports' alias with mail ports (25 587 465)



  • Those blocks are on your WAN, so they won't tell you much. Like @Gertjan mentioned, you'll want to either create 3 separate rules on your LAN interface which blocks and logs each port. Or create an alias that contains all 3 ports and create one block rule.

    Either way, if you want to capture the internal IP's, you have to put the rule on your LAN interface. Also, make sure the block rule is above your LAN net/any rule.



  • Yes just realized i can still send mails and moved the rules to LAN.

    @Gertjan said in How to find spambot? Got network abuse report from my ISP:

    Remove snort. It should be there to warn you way before your ISP. It isn't working for you so pretty useless

    How should it warn me? I mean i don't sit there and watch it the whole day.
    I only check Snort if something is not working to see if something got blocked.

    How do others handle that kind of scenario?
    We don't send much mails. Would be cool to build something that blinks an LED if a mail was send
    and makes a peep every time. That way you realize there a hundreds of mails going out.
    Sound like a job for an Arduino.



  • The first step is creating the firewall rule(s) on the correct interface so you can log the traffic. Next, since PFsense has a really short log buffer, I would start exporting your logs to a syslog server so you have historical data that can be filtered, which will help you identify the offending device(s).


  • Netgate Administrator

    If you had Snort running on the LAN it would show the internal source IP on those alerts.

    If you have that block rule in place then filter the firewall logs to show only blocked traffic on LAN, check what's there.

    Steve



  • Back then (2014) when i did set up Snort the tutorial said WAN.
    But i made a search now and it seems like the recommendation is to put Snort on the LAN.

    I guess unless that spam thing happens again i will never know what it was.
    As i said i did a lot of testing/playing the last days and maybe it was something on a Raspberry Pi



  • @MrGlasspoole There may be differing options on this subject, but I have Suricata running on both interfaces. The WAN to identify incoming nefarious activity and the LAN to identify the same thing outgoing (e.g. infected machines or the same thing you're experiencing right now, etc)


  • LAYER 8 Global Moderator

    @marvosa said in How to find spambot? Got network abuse report from my ISP:

    The WAN to identify incoming nefarious activity

    Why would it matter unless forwarded, if forwarded through to lan - you would see it there, and stop it, etc.



  • @marvosa said in How to find spambot? Got network abuse report from my ISP:

    I have Suricata running on both interfaces. .... and the LAN to identify the same thing outgoing (e.g. infected machines or the same thing you're experiencing right now, etc)

    You are aware of the fact that most traffic is "invisible" these day : SSL encrypted.
    The unwanted traffic also encrypted these days.

    Site that use 'http' web access start to fade away, as do 'clear' smtp / pop and imap communication. It's smtps these days (or pops or imaps).
    Suricata can not inspect that SSL traffic flow.

    The information that is known at the firewall level is : the destination IP, port, DNS and reverse DNS.
    What's in the subsequent traffic stays hidden.


  • Netgate Administrator

    If you have the spare CPU cycles then inspect traffic on both but that's going to be double inspecting on the vast majority of traffic. If you run on only one interface I would recommend LAN. You don't see hits on the firewall itself but having visibility on the internal IPs of more valuable IMO. I depends what you're trying to catch.

    Steve



  • @johnpoz said in How to find spambot? Got network abuse report from my ISP:

    Why would it matter unless forwarded, if forwarded through to lan - you would see it there, and stop it, etc.

    I just like to see what alerts are generated on the WAN.. just because I'm curious... also, IMO, if you're going to run it at all... you might as well run it on both fronts.

    @Gertjan said in How to find spambot? Got network abuse report from my ISP:

    You are aware of the fact that most traffic is "invisible" these day : SSL encrypted.
    The unwanted traffic also encrypted these days.
    Site that use 'http' web access start to fade away, as do 'clear' smtp / pop and imap communication. It's smtps these days (or pops or imaps).
    Suricata can not inspect that SSL traffic flow.
    The information that is known at the firewall level is : the destination IP, port, DNS and reverse DNS.
    What's in the subsequent traffic stays hidden.

    Yes, I am aware of encrypted traffic and neither said Suricata could inspect SSL traffic nor do I expect it to. It's all about rules, signatures, and reputation.


  • LAYER 8 Global Moderator

    @marvosa said in How to find spambot? Got network abuse report from my ISP:

    you might as well run it on both fronts.

    Nope - that logic makes no sense..



  • @MrGlasspoole FWIW, I'd block the mail ports as suggested overnight. Probably be able to come up with the offender by morning. Also, how many clients are you serving and what kind of antivirus protection do you have in place? Could also possibly a laptop someone brought in.



  • I have the ports blocked since 3 days and Snort runs now on LAN.

    Nothing shows up. All phones, PCs, tablets, the print server, satellite receiver, DECT station are connected as ever.
    Nobody was here with a another device.
    As i said: could be something temporary from playing around and it's no longer present.
    But it's hard to believe there was something in Raspbian, Armbian or DietPi.
    Maybe something i did but on the new Fire TV stick that i already did uninstall...

    The funny thing is that the day before someone from my ISP was here because bridge mode did not work and there box sometimes did crash/reboot.
    He was the only one with other devices.
    He told me the router did not receive the last firmware automatically and made a reset to factory defaults. After that the router did pull the newest firmware.



  • @MrGlasspoole said in How to find spambot? Got network abuse report from my ISP:

    Nothing shows up

    You mean : even you didn't send a mail ?
    Show us the LAN firewall ?

    @MrGlasspoole said in How to find spambot? Got network abuse report from my ISP:

    He was the only one with other devices.

    These devices are not passing through pfSense ...



  • Does anybody know what this means, in his screenshot above?

    source_port: 33431

    Jeff



  • @MrGlasspoole said in How to find spambot? Got network abuse report from my ISP:

    source_port: 33431
    source-port: 33431
    original-rcpt-to: www.tiaanxxxxxxxx.09@gmail.com
    source-ip: 176.199.xxx.xxx
    

    When the mail was send, it cam from the IP 176.199.xxx.xxx using port 33431.
    The device that send the mail the mail was behind pfSense (but was it - see my question just above) and so it was NATted.
    The original source LAN IP and source port can't be known to the ISP.
    The only thing they and we know is the destination, a gmail mail server and one of the mail destination ports : 465 or 25 if gmail still accepts mail on port '25'.



  • @Gertjan said in How to find spambot? Got network abuse report from my ISP:

    You mean : even you didn't send a mail

    What i mean is that nothing shows up that is trying to send mails.
    Sure, if I try to send one (what of course is not working) it shows up.



  • It happened again.

    Nothing in Snort and mail ports are blocked:
    5.jpg

    I have a lot of blocked UDP but i don't know what device it is with that IPv6:
    2Status_-System-Logs_-Firewall_-Normal-V_---10.1.0.gif

    And this and 10.1.254.10 is my Nvidia Shield:
    4Status_-System-Logs_-Firewall_-Normal-V_---10.1.0.1.gif



  • @akuma1x said in How to find spambot? Got network abuse report from my ISP:

    Does anybody know what this means, in his screenshot above?

    source_port: 33431

    Jeff

    A quick search revealed this...could it be his router in front of pfSense doing the spamming?

    Screen Shot 2019-12-13 at 10.01.25 AM.png



  • Humm.

    No need to list firewall logs - or the rules. As the link (from your ISP ?) shows, the last 2 days there were no spam mails. It stopped 11 December, that was two days ago.

    Your firewall rules look ok to me.

    I played with an identical rule :

    5aa45e4b-1618-4f5f-bb1a-b50291235318-image.png

    My second rules, with the port-alias including 25,465,587 works just fine : Outlook, the mail client is now hurling it can't mail out any mails ....
    That the rules is capturing traffic can be seen by :

    522af0fc-06e4-4575-be1f-964a2dbd545c-image.png

    What this all means : if mail reaches the Internet by your WAN IP, it isn't injected via your LAN (?) interface.
    By OpenVPN (is this a VPN server interface ?) ?
    If not, that it's pretty save to say : this spam mail does flow through pfSense. What this means : you can disconnect the LAN interface and spam will still be send. Because it's not originating from a device 'behind' pfSense - on it's LAN.
    pfSense is connected to what upstream device ? A router with wifi or other devices connected to it ?



  • @Gertjan Wondered whether his ISP serviceman loaded a corrupted firmware on the router!



  • There is only the ISP router/modem in front of pfSense.
    Until last week i had double NAT but it is now switched to bridged mode.
    So the routers WLAN and everything is off.

    OpenVPN is the OpenVPN Server in pfSense.

    What also changed is that i moved from Proxmox to Hyper-V (in Windows 10) that is giving me trouble and not working reliable. Had pfSense running in Proxmox for a long time and also in Hyper-V 2016 Core without problems.
    https://forum.netgate.com/topic/148638/can-t-get-puplic-ip-in-pfsense-with-hyper-v-in-windows-10-pro



  • That only happens in fake Youtube movies and other SF films.
    It can be done, though .... but who wins what where and when ?
    If only the tech service can update such a router, the scandal would be huge : ISP out of business .... that's mega $$$$$$.
    If one can't trust his ISP router any more ..... ☹

    Said that : my ISP router is just a router : no TV - phone - Wifi facilities are enabled. It's a pure VDSL to LAN-port device, having pfSense as it's only client-device. No one connects to the other 3 available LAN ports - never.



  • @MrGlasspoole said in How to find spambot? Got network abuse report from my ISP:

    OpenVPN is the OpenVPN Server in pfSense.

    So, this could be another vector ?!

    @MrGlasspoole said in How to find spambot? Got network abuse report from my ISP:

    Hyper-V (in Windows 10)

    Yet another vector : if the hosting W10 machine has access to the pfSense WAN interface (ir shouldn't) any software in the host (W10) could output mail, completely bypassing pfSense.

    Again : these spam mails are not passing through pfSense. Up to you to test the rest.

    Btw :
    https://docs.netgate.com/pfsense/en/latest/virtualization/virtualizing-pfsense-with-hyper-v.html

    When they talk about the WAN :

    9d02ca9a-9be9-4bfe-ac90-f1a1e9d8c008-image.png

    the option : "Allow management operating system to share this network adapter" should be unchecked !!
    If not, the host could use it, bypassing pfSense.



  • Sure "Allow management operating system to share this network adapter" is disabled.
    That is how i had it for 3 years in Hyper-V 2016.

    I did disabled OpenVPN for now.

    I also made a AdwCleaner and Malwarebytes Anti-Malware scan on the host (Win 10).

    For now i switched to my pfSense standalone box.
    If it's the Hyper-V box then it should show up in pfSense!?

    Is there something (Virus, Trojan or whatever) that could bypass the blocked mail ports?
    Sending mails over something else?



  • @MrGlasspoole You're never going to catch anything on PFsense's log viewer unless you switch to dynamic view and watch it live. And even then, because the log buffer is so short, you still may miss it. In order for the firewall logs to be useful, you need to export them to a syslog server. However, up until now, it's been moot cause your block rules show no hits anyway.

    After looking around, it looks like there may be some additional ports that need to be blocked. Apparently, 2525 and 2526 are non-standard ports that are used as alternates on some mail servers. Which makes the list now:

    25, 465, 587, 2525, 2526

    I would start exporting your firewall logs to a syslog server at a minimum. Some other things to look at:

    • Start sending netflow data to a collector. The data would be captured either at the switch, which would be ideal if you have capable gear or on PFsense via the softflowd package. The idea being you would have historical top talker/top connection data that can be cross-referenced with the timestamps in your ISP's abuse reports which will aid in tracking down the offending device.
    • Do you have any full tunnel road warrior setups or site-to-site tunnels configured where the remote end is routing its traffic through your connection? If so, it's possible that the offending device may not even be on your network.
    • You could install Untangle in bridge mode behind PFsense and leverage the SPAM filter app


  • @MrGlasspoole said in How to find spambot? Got network abuse report from my ISP:

    Is there something (Virus, Trojan or whatever) that could bypass the blocked mail ports?
    Sending mails over something else?

    Sure !
    That's like running a web server on port ....81 (for http) and, example, port 512 (https).
    No one will be able to connect to it, except if the user knows about it, and uses on his side - the client - the special port override.

    Now for mails : Take a look at this setting :
    System > Advanced > Notifications

    650a61d4-ece8-40ac-b865-db1a94640747-image.png

    I could set up any port here. Why not.
    But .... when I want to send a notification mail to gmail, it has to be port 587 or 465 because gmail hasn't set up any other ports for receiving mail.

    You could be running your own mail server for your own domains( I actually do that) and thus mails leaving my local network could use any port, if I set up my mail server to use these ports to receive mails, so it can relay them afterwards. My mail server would have to use the standard ports 25 (clear text mail) or 465 (smtps mail) to hand it over to the destination MX.
    No exception possible.



  • @marvosa said in How to find spambot? Got network abuse report from my ISP:
    Some other things to look at:

    • Start sending netflow data to a collector. The data would be captured either at the switch, which would be ideal if you have capable gear or on PFsense via the softflowd package. The idea being you would have historical top talker/top connection data that can be cross-referenced with the timestamps in your ISP's abuse reports which will aid in tracking down the offending device.
    • Do you have any full tunnel road warrior setups or site-to-site tunnels configured where the remote end is routing its traffic through your connection? If so, it's possible that the offending device may not even be on your network.
    • You could install Untangle in bridge mode behind PFsense and leverage the SPAM filter app

    That all sounds really complicated. This is a home network and not the FBI headquarters :-)
    I have a Netgear WNDR4300 with DD-WRT as AP and a ZyXEL GS1900-24E switch.

    I wonder what the Nvidia Shield is doing there all the time what gets blocked.
    And how can i found out that other device that only shows a IPv6 (the one with 5353 UDP)?

    I did google if other people get mails from my ISP but there mails are about Open-DNS Resolver and open mDNS Service.
    Most people get mails because of Amazon Fire TV/Fire HD or PlayStation 4 and port 5353.


  • LAYER 8 Global Moderator

    @MrGlasspoole said in How to find spambot? Got network abuse report from my ISP:

    port 5353.

    has zero to do with email - its mdns... No you shouldn't have those ports open to the internet.



  • @johnpoz said in How to find spambot? Got network abuse report from my ISP:

    port 5353.

    has zero to do with email - its mdns... No you shouldn't have those ports open to the internet.

    I know it has nothing to do with email. I just wonder that my ISP is sending abuse mails to a lot of people about it and most of the time the PS4 was the problem.

    I did install PRTG Network Monitor. Now i need to find out how that works.
    Looks like flying a Boeing 747.

    I need to monitor the WAN port? I do that with softflowd?



  • Ok there are 18 new events on that abuse site this morning 2:52am (Dec 14).
    pfSense is running on a standalone machine (no HyperV) and all the mail ports are blocked.

    I have now PRTG running but I'm not sure with some settings.

    Should i leave the "Source Address" default or should i set it to LAN since I'm only looking for mails going out?
    For "Syslog Contents" i only selected "Firewall Events".
    remote log1.jpg

    I did set the "Include Filter" in PRTG to "message[igb1]" because I'm only looking for what is blocked on the LAN interface.
    This is what i have then:
    remote log3.jpg
    I cant figure out how to filter that by port or how to make it more readable.
    This two messages is PRTG trying to send some mails.

    Now that NetFlow thing. I have no idea what values to use on Max Flows, Hop Limit and on all this timeout settings.
    Is it enough to select "IP" in "Flow Tracking Level"?
    IPFIX settings2.jpg

    This this from PRTG but is over my head:
    Make sure that you set an Active Flow Timeout in the sensor settings that is one minute greater than the flow timeout value that you set in the target device.
    For more information, see https://kb.paessler.com/en/topic/66485.

    I did set the filter to Port[25] or Port[465] or Port[587] or Port[2525] or Port[2526]
    IPFIX settings1.jpg

    Here i did retrieve and send a mail with Thunderbird. Is it normal that i don't see the IP from my workstation?
    testmail.jpg

    This is some other stuff i see. Stuff i don't know what it is:
    testmail3.jpg



  • After my last posting i see this:
    spam.gif

    And ~50 new entries on the abuse website.
    ☹



  • Have you considered that you might be relaying spam from an external source, rather than originating the spam on your own network - running an open relay?

    Show us a screenshot of your WAN rules.



  • wan.jpg


  • LAYER 8 Global Moderator

    There is a whole bunch of traffic to 25 in that listing... But its showing your wan IP..



  • @MrGlasspoole :
    What type of network is this: a home network or a business?

    If home, then NAT is going to mask the local host that is spamming when looking at traffic on the WAN interface. To see that IP, you need to put your tools on the LAN interface and then examine all devices you see initiating connections to an Internet host on ports 25, 465 and 587. A spambot will need to make outbound connections to other mail servers using one of those three ports. The source port is immaterial as it will be an ephemeral port. What matters is the destination port (one of the big 3 mail server ports). Once you find that traffic entering the LAN interface, examine the source IP addresses to find the infection.

    If you have a business network with multiple subnets hanging off the firewall, you will follow the same process but just do it for each configured interface.

    You could have an infected PC or even an infected IoT device (Smart TV, Smart Home Hub, etc.).



  • @johnpoz said in How to find spambot? Got network abuse report from my ISP:

    There is a whole bunch of traffic to 25 in that listing... But its showing your wan IP..

    I did set softflowd to the WAN because if i understand it right: If it would be something in my LAN i would see it in the System Log?

    @bmeeks said in How to find spambot? Got network abuse report from my ISP:

    @MrGlasspoole :
    What type of network is this: a home network or a business?

    If home, then NAT is going to mask the local host that is spamming when looking at traffic on the WAN interface. To see that IP, you need to put your tools on the LAN interface and then examine all devices you see initiating connections to an Internet host on ports 25, 465 and 587.
    You could have an infected PC or even an infected IoT device (Smart TV, Smart Home Hub, etc.).

    Its a home network and as you can see in this threat the mail ports are blocked on the LAN interface and nothing raises an alert in the log.


Log in to reply