How to find spambot? Got network abuse report from my ISP



  • @johnpoz said in How to find spambot? Got network abuse report from my ISP:

    Why would it matter unless forwarded, if forwarded through to lan - you would see it there, and stop it, etc.

    I just like to see what alerts are generated on the WAN.. just because I'm curious... also, IMO, if you're going to run it at all... you might as well run it on both fronts.

    @Gertjan said in How to find spambot? Got network abuse report from my ISP:

    You are aware of the fact that most traffic is "invisible" these day : SSL encrypted.
    The unwanted traffic also encrypted these days.
    Site that use 'http' web access start to fade away, as do 'clear' smtp / pop and imap communication. It's smtps these days (or pops or imaps).
    Suricata can not inspect that SSL traffic flow.
    The information that is known at the firewall level is : the destination IP, port, DNS and reverse DNS.
    What's in the subsequent traffic stays hidden.

    Yes, I am aware of encrypted traffic and neither said Suricata could inspect SSL traffic nor do I expect it to. It's all about rules, signatures, and reputation.


  • LAYER 8 Global Moderator

    @marvosa said in How to find spambot? Got network abuse report from my ISP:

    you might as well run it on both fronts.

    Nope - that logic makes no sense..



  • @MrGlasspoole FWIW, I'd block the mail ports as suggested overnight. Probably be able to come up with the offender by morning. Also, how many clients are you serving and what kind of antivirus protection do you have in place? Could also possibly a laptop someone brought in.



  • I have the ports blocked since 3 days and Snort runs now on LAN.

    Nothing shows up. All phones, PCs, tablets, the print server, satellite receiver, DECT station are connected as ever.
    Nobody was here with a another device.
    As i said: could be something temporary from playing around and it's no longer present.
    But it's hard to believe there was something in Raspbian, Armbian or DietPi.
    Maybe something i did but on the new Fire TV stick that i already did uninstall...

    The funny thing is that the day before someone from my ISP was here because bridge mode did not work and there box sometimes did crash/reboot.
    He was the only one with other devices.
    He told me the router did not receive the last firmware automatically and made a reset to factory defaults. After that the router did pull the newest firmware.



  • @MrGlasspoole said in How to find spambot? Got network abuse report from my ISP:

    Nothing shows up

    You mean : even you didn't send a mail ?
    Show us the LAN firewall ?

    @MrGlasspoole said in How to find spambot? Got network abuse report from my ISP:

    He was the only one with other devices.

    These devices are not passing through pfSense ...



  • Does anybody know what this means, in his screenshot above?

    source_port: 33431

    Jeff



  • @MrGlasspoole said in How to find spambot? Got network abuse report from my ISP:

    source_port: 33431
    source-port: 33431
    original-rcpt-to: www.tiaanxxxxxxxx.09@gmail.com
    source-ip: 176.199.xxx.xxx
    

    When the mail was send, it cam from the IP 176.199.xxx.xxx using port 33431.
    The device that send the mail the mail was behind pfSense (but was it - see my question just above) and so it was NATted.
    The original source LAN IP and source port can't be known to the ISP.
    The only thing they and we know is the destination, a gmail mail server and one of the mail destination ports : 465 or 25 if gmail still accepts mail on port '25'.



  • @Gertjan said in How to find spambot? Got network abuse report from my ISP:

    You mean : even you didn't send a mail

    What i mean is that nothing shows up that is trying to send mails.
    Sure, if I try to send one (what of course is not working) it shows up.



  • It happened again.

    Nothing in Snort and mail ports are blocked:
    5.jpg

    I have a lot of blocked UDP but i don't know what device it is with that IPv6:
    2Status_-System-Logs_-Firewall_-Normal-V_---10.1.0.gif

    And this and 10.1.254.10 is my Nvidia Shield:
    4Status_-System-Logs_-Firewall_-Normal-V_---10.1.0.1.gif



  • @akuma1x said in How to find spambot? Got network abuse report from my ISP:

    Does anybody know what this means, in his screenshot above?

    source_port: 33431

    Jeff

    A quick search revealed this...could it be his router in front of pfSense doing the spamming?

    Screen Shot 2019-12-13 at 10.01.25 AM.png



  • Humm.

    No need to list firewall logs - or the rules. As the link (from your ISP ?) shows, the last 2 days there were no spam mails. It stopped 11 December, that was two days ago.

    Your firewall rules look ok to me.

    I played with an identical rule :

    5aa45e4b-1618-4f5f-bb1a-b50291235318-image.png

    My second rules, with the port-alias including 25,465,587 works just fine : Outlook, the mail client is now hurling it can't mail out any mails ....
    That the rules is capturing traffic can be seen by :

    522af0fc-06e4-4575-be1f-964a2dbd545c-image.png

    What this all means : if mail reaches the Internet by your WAN IP, it isn't injected via your LAN (?) interface.
    By OpenVPN (is this a VPN server interface ?) ?
    If not, that it's pretty save to say : this spam mail does flow through pfSense. What this means : you can disconnect the LAN interface and spam will still be send. Because it's not originating from a device 'behind' pfSense - on it's LAN.
    pfSense is connected to what upstream device ? A router with wifi or other devices connected to it ?



  • @Gertjan Wondered whether his ISP serviceman loaded a corrupted firmware on the router!



  • There is only the ISP router/modem in front of pfSense.
    Until last week i had double NAT but it is now switched to bridged mode.
    So the routers WLAN and everything is off.

    OpenVPN is the OpenVPN Server in pfSense.

    What also changed is that i moved from Proxmox to Hyper-V (in Windows 10) that is giving me trouble and not working reliable. Had pfSense running in Proxmox for a long time and also in Hyper-V 2016 Core without problems.
    https://forum.netgate.com/topic/148638/can-t-get-puplic-ip-in-pfsense-with-hyper-v-in-windows-10-pro



  • That only happens in fake Youtube movies and other SF films.
    It can be done, though .... but who wins what where and when ?
    If only the tech service can update such a router, the scandal would be huge : ISP out of business .... that's mega $$$$$$.
    If one can't trust his ISP router any more ..... ☹

    Said that : my ISP router is just a router : no TV - phone - Wifi facilities are enabled. It's a pure VDSL to LAN-port device, having pfSense as it's only client-device. No one connects to the other 3 available LAN ports - never.



  • @MrGlasspoole said in How to find spambot? Got network abuse report from my ISP:

    OpenVPN is the OpenVPN Server in pfSense.

    So, this could be another vector ?!

    @MrGlasspoole said in How to find spambot? Got network abuse report from my ISP:

    Hyper-V (in Windows 10)

    Yet another vector : if the hosting W10 machine has access to the pfSense WAN interface (ir shouldn't) any software in the host (W10) could output mail, completely bypassing pfSense.

    Again : these spam mails are not passing through pfSense. Up to you to test the rest.

    Btw :
    https://docs.netgate.com/pfsense/en/latest/virtualization/virtualizing-pfsense-with-hyper-v.html

    When they talk about the WAN :

    9d02ca9a-9be9-4bfe-ac90-f1a1e9d8c008-image.png

    the option : "Allow management operating system to share this network adapter" should be unchecked !!
    If not, the host could use it, bypassing pfSense.



  • Sure "Allow management operating system to share this network adapter" is disabled.
    That is how i had it for 3 years in Hyper-V 2016.

    I did disabled OpenVPN for now.

    I also made a AdwCleaner and Malwarebytes Anti-Malware scan on the host (Win 10).

    For now i switched to my pfSense standalone box.
    If it's the Hyper-V box then it should show up in pfSense!?

    Is there something (Virus, Trojan or whatever) that could bypass the blocked mail ports?
    Sending mails over something else?



  • @MrGlasspoole You're never going to catch anything on PFsense's log viewer unless you switch to dynamic view and watch it live. And even then, because the log buffer is so short, you still may miss it. In order for the firewall logs to be useful, you need to export them to a syslog server. However, up until now, it's been moot cause your block rules show no hits anyway.

    After looking around, it looks like there may be some additional ports that need to be blocked. Apparently, 2525 and 2526 are non-standard ports that are used as alternates on some mail servers. Which makes the list now:

    25, 465, 587, 2525, 2526

    I would start exporting your firewall logs to a syslog server at a minimum. Some other things to look at:

    • Start sending netflow data to a collector. The data would be captured either at the switch, which would be ideal if you have capable gear or on PFsense via the softflowd package. The idea being you would have historical top talker/top connection data that can be cross-referenced with the timestamps in your ISP's abuse reports which will aid in tracking down the offending device.
    • Do you have any full tunnel road warrior setups or site-to-site tunnels configured where the remote end is routing its traffic through your connection? If so, it's possible that the offending device may not even be on your network.
    • You could install Untangle in bridge mode behind PFsense and leverage the SPAM filter app


  • @MrGlasspoole said in How to find spambot? Got network abuse report from my ISP:

    Is there something (Virus, Trojan or whatever) that could bypass the blocked mail ports?
    Sending mails over something else?

    Sure !
    That's like running a web server on port ....81 (for http) and, example, port 512 (https).
    No one will be able to connect to it, except if the user knows about it, and uses on his side - the client - the special port override.

    Now for mails : Take a look at this setting :
    System > Advanced > Notifications

    650a61d4-ece8-40ac-b865-db1a94640747-image.png

    I could set up any port here. Why not.
    But .... when I want to send a notification mail to gmail, it has to be port 587 or 465 because gmail hasn't set up any other ports for receiving mail.

    You could be running your own mail server for your own domains( I actually do that) and thus mails leaving my local network could use any port, if I set up my mail server to use these ports to receive mails, so it can relay them afterwards. My mail server would have to use the standard ports 25 (clear text mail) or 465 (smtps mail) to hand it over to the destination MX.
    No exception possible.



  • @marvosa said in How to find spambot? Got network abuse report from my ISP:
    Some other things to look at:

    • Start sending netflow data to a collector. The data would be captured either at the switch, which would be ideal if you have capable gear or on PFsense via the softflowd package. The idea being you would have historical top talker/top connection data that can be cross-referenced with the timestamps in your ISP's abuse reports which will aid in tracking down the offending device.
    • Do you have any full tunnel road warrior setups or site-to-site tunnels configured where the remote end is routing its traffic through your connection? If so, it's possible that the offending device may not even be on your network.
    • You could install Untangle in bridge mode behind PFsense and leverage the SPAM filter app

    That all sounds really complicated. This is a home network and not the FBI headquarters :-)
    I have a Netgear WNDR4300 with DD-WRT as AP and a ZyXEL GS1900-24E switch.

    I wonder what the Nvidia Shield is doing there all the time what gets blocked.
    And how can i found out that other device that only shows a IPv6 (the one with 5353 UDP)?

    I did google if other people get mails from my ISP but there mails are about Open-DNS Resolver and open mDNS Service.
    Most people get mails because of Amazon Fire TV/Fire HD or PlayStation 4 and port 5353.


  • LAYER 8 Global Moderator

    @MrGlasspoole said in How to find spambot? Got network abuse report from my ISP:

    port 5353.

    has zero to do with email - its mdns... No you shouldn't have those ports open to the internet.



  • @johnpoz said in How to find spambot? Got network abuse report from my ISP:

    port 5353.

    has zero to do with email - its mdns... No you shouldn't have those ports open to the internet.

    I know it has nothing to do with email. I just wonder that my ISP is sending abuse mails to a lot of people about it and most of the time the PS4 was the problem.

    I did install PRTG Network Monitor. Now i need to find out how that works.
    Looks like flying a Boeing 747.

    I need to monitor the WAN port? I do that with softflowd?



  • Ok there are 18 new events on that abuse site this morning 2:52am (Dec 14).
    pfSense is running on a standalone machine (no HyperV) and all the mail ports are blocked.

    I have now PRTG running but I'm not sure with some settings.

    Should i leave the "Source Address" default or should i set it to LAN since I'm only looking for mails going out?
    For "Syslog Contents" i only selected "Firewall Events".
    remote log1.jpg

    I did set the "Include Filter" in PRTG to "message[igb1]" because I'm only looking for what is blocked on the LAN interface.
    This is what i have then:
    remote log3.jpg
    I cant figure out how to filter that by port or how to make it more readable.
    This two messages is PRTG trying to send some mails.

    Now that NetFlow thing. I have no idea what values to use on Max Flows, Hop Limit and on all this timeout settings.
    Is it enough to select "IP" in "Flow Tracking Level"?
    IPFIX settings2.jpg

    This this from PRTG but is over my head:
    Make sure that you set an Active Flow Timeout in the sensor settings that is one minute greater than the flow timeout value that you set in the target device.
    For more information, see https://kb.paessler.com/en/topic/66485.

    I did set the filter to Port[25] or Port[465] or Port[587] or Port[2525] or Port[2526]
    IPFIX settings1.jpg

    Here i did retrieve and send a mail with Thunderbird. Is it normal that i don't see the IP from my workstation?
    testmail.jpg

    This is some other stuff i see. Stuff i don't know what it is:
    testmail3.jpg



  • After my last posting i see this:
    spam.gif

    And ~50 new entries on the abuse website.
    ☹



  • Have you considered that you might be relaying spam from an external source, rather than originating the spam on your own network - running an open relay?

    Show us a screenshot of your WAN rules.



  • wan.jpg


  • LAYER 8 Global Moderator

    There is a whole bunch of traffic to 25 in that listing... But its showing your wan IP..



  • @MrGlasspoole :
    What type of network is this: a home network or a business?

    If home, then NAT is going to mask the local host that is spamming when looking at traffic on the WAN interface. To see that IP, you need to put your tools on the LAN interface and then examine all devices you see initiating connections to an Internet host on ports 25, 465 and 587. A spambot will need to make outbound connections to other mail servers using one of those three ports. The source port is immaterial as it will be an ephemeral port. What matters is the destination port (one of the big 3 mail server ports). Once you find that traffic entering the LAN interface, examine the source IP addresses to find the infection.

    If you have a business network with multiple subnets hanging off the firewall, you will follow the same process but just do it for each configured interface.

    You could have an infected PC or even an infected IoT device (Smart TV, Smart Home Hub, etc.).



  • @johnpoz said in How to find spambot? Got network abuse report from my ISP:

    There is a whole bunch of traffic to 25 in that listing... But its showing your wan IP..

    I did set softflowd to the WAN because if i understand it right: If it would be something in my LAN i would see it in the System Log?

    @bmeeks said in How to find spambot? Got network abuse report from my ISP:

    @MrGlasspoole :
    What type of network is this: a home network or a business?

    If home, then NAT is going to mask the local host that is spamming when looking at traffic on the WAN interface. To see that IP, you need to put your tools on the LAN interface and then examine all devices you see initiating connections to an Internet host on ports 25, 465 and 587.
    You could have an infected PC or even an infected IoT device (Smart TV, Smart Home Hub, etc.).

    Its a home network and as you can see in this threat the mail ports are blocked on the LAN interface and nothing raises an alert in the log.


  • LAYER 8 Global Moderator

    But how you suppose to figure out what is sending it? You need to see both!

    Also what are you rules on your floating?

    Do you have more than 1 IP? Looks like your public IP is different or you have more than 1, see traffic to x.x.36.95, and then your also showing traffic to 25 from x.x.35.103..



  • There is nothing in floating.

    pfSense is showing me a 176.xxx.xxx.xxx address on WAN.
    On that abuse site the last reports are 88.xxx.xxx.xxx.

    I enabled softflowd now also on the LAN.
    Is it normal that there is no time on every single entry in that IPFIX thing?

    For testing i did open the port on LAN and did send a mail and it shows up and has the 176.... address:
    test.gif


  • Netgate Administrator

    So what is using that 88.x.x.x IP? Seems like it must be on the firewall...

    Check the pfSense routing table in Diag > Routes, do both IPs appear there?

    Is the WAN configured as DHCP? Could it just be pulling a new IP from the ISP there?

    Steve



  • Sure WAN is DHCP since the ISP router is in bridge mode.

    This is what routes is showing:
    routes.jpg

    The last abuse entries. Always new entries:
    spam.jpg


  • LAYER 8 Global Moderator

    Well that aint your IP...

    Neither of them are.. atleast not with pfsense - do you have other devices connected to this isp?



  • Only the pfSense WAN is connected to the ISP router/modem.

    And that network abuse report has my customer ID.
    That 176.199.143.125 in that report is the old IP.
    Now its 176.xxx.xxx.174 that i see on the WAN in pfSense.
    Most of the time when you restart the modem you get a new one.

    Its driving me crazy ☹


  • LAYER 8 Global Moderator

    And how are you sure its not reports from some other guy that had that IP before you got it? ;)

    That is part of the problem of changing IPs all the time - you can be held responsible for the sins of the guy that had it before you..

    But you did show a bunch of traffic to 25...



  • But i can see that 88.xxx.xxx.103 that is on top on that report also in softflowd?

    @johnpoz said in How to find spambot? Got network abuse report from my ISP:

    But you did show a bunch of traffic to 25...

    And that is what I (we) trying to figure out? Where does it come from?
    It does not come from the LAN because the ports are blocked...
    Only pfSense is connected to the modem...

    I don't get it 😢

    That IPFIX log is full of mail port entries.
    100 every 15 minutes...
    Are this entries blocked stuff from outside?


  • LAYER 8 Global Moderator

    Look in your state table... What doe you see for those connections to 25.



  • Diagnostics > States?

    There is nothing with port 25, 587 or the 88.xxx.xxx.xxx IP.


  • LAYER 8 Global Moderator

    Well are you seeing the connections - you stated that your seeing 100 connections every 15 minutes... Watch it for a while.. that 88.x address not going to be in use any more since your on a different IP now.



  • @MrGlasspoole said in How to find spambot? Got network abuse report from my ISP:

    The funny thing is that the day before someone from my ISP was here because bridge mode did not work and there box sometimes did crash/reboot.
    He was the only one with other devices.
    He told me the router did not receive the last firmware automatically and made a reset to factory defaults. After that the router did pull the newest firmware.

    I am really curious about his situation, and somehow I suspect his router especially under the above statement...the firmware could be corrupted at no fault of anyone. If nothing is coming from LAN, it must be the router...does the router has any known vulnerabilities?


Log in to reply