Went from no internet access to slow access



  • So I came into work today, and everyone's Internet was dropping in and out. After awhile, it completely cut out. I did some initial troubleshooting, and found the "router" to be the problem. Since I am new to this job, I found out our router is a computer running pfSense. After restarting it, the internet was back. But after awhile it seems to be either extremely slow, or dropping momentarily again. I don't know anything about pfSense, or reading logs, so please advise me what to post.

    To me, the system log looks a little suspicious:

    
    Last 50 system log entries
    Aug 23 13:00:27 	pftpx[529]: #16998 client reset connection
    Aug 23 13:00:27 	pftpx[529]: #16998 client reset connection
    Aug 23 13:00:27 	pftpx[529]: #16999 client reset connection
    Aug 23 13:00:27 	pftpx[529]: #16999 client reset connection
    Aug 23 13:00:27 	pftpx[529]: #17000 client reset connection
    Aug 23 13:00:27 	pftpx[529]: #17000 client reset connection
    Aug 23 13:00:27 	pftpx[529]: #17001 client reset connection
    Aug 23 13:00:27 	pftpx[529]: #17001 client reset connection
    Aug 23 13:00:27 	pftpx[529]: accept failed: Software caused connection abort
    Aug 23 13:00:27 	pftpx[529]: accept failed: Software caused connection abort
    Aug 23 13:00:27 	pftpx[529]: #17002 client reset connection
    Aug 23 13:00:27 	pftpx[529]: #17002 client reset connection
    Aug 23 13:00:27 	pftpx[529]: accept failed: Software caused connection abort
    Aug 23 13:00:27 	pftpx[529]: accept failed: Software caused connection abort
    Aug 23 13:00:27 	pftpx[529]: #17003 client reset connection
    Aug 23 13:00:27 	pftpx[529]: #17003 client reset connection
    Aug 23 13:00:27 	pftpx[529]: #17004 client reset connection
    Aug 23 13:00:27 	pftpx[529]: #17004 client reset connection
    Aug 23 13:00:28 	pftpx[529]: #17005 client reset connection
    Aug 23 13:00:28 	pftpx[529]: #17005 client reset connection
    Aug 23 13:00:29 	pftpx[529]: #17006 client reset connection
    Aug 23 13:00:29 	pftpx[529]: #17006 client reset connection
    Aug 23 13:00:29 	pftpx[529]: #17007 client reset connection
    Aug 23 13:00:29 	pftpx[529]: #17007 client reset connection
    Aug 23 13:00:36 	pftpx[529]: #17008 client reset connection
    Aug 23 13:00:36 	pftpx[529]: #17008 client reset connection
    Aug 23 13:00:37 	pftpx[529]: #17009 client reset connection
    Aug 23 13:00:37 	pftpx[529]: #17009 client reset connection
    Aug 23 13:00:37 	pftpx[529]: accept failed: Software caused connection abort
    Aug 23 13:00:37 	pftpx[529]: accept failed: Software caused connection abort
    Aug 23 13:00:37 	pftpx[529]: #17010 client reset connection
    Aug 23 13:00:37 	pftpx[529]: #17010 client reset connection
    Aug 23 13:00:37 	pftpx[529]: accept failed: Software caused connection abort
    Aug 23 13:00:37 	pftpx[529]: accept failed: Software caused connection abort
    Aug 23 13:00:37 	last message repeated 2 times
    Aug 23 13:00:37 	pftpx[529]: #17011 client reset connection
    Aug 23 13:00:37 	last message repeated 2 times
    Aug 23 13:00:37 	pftpx[529]: #17011 client reset connection
    Aug 23 13:00:37 	pftpx[529]: #17012 client reset connection
    Aug 23 13:00:37 	pftpx[529]: #17012 client reset connection
    Aug 23 13:00:40 	pftpx[529]: #17013 client reset connection
    Aug 23 13:00:40 	pftpx[529]: #17013 client reset connection
    Aug 23 13:00:40 	pftpx[529]: #17014 client reset connection
    Aug 23 13:00:40 	pftpx[529]: #17014 client reset connection
    Aug 23 13:00:41 	pftpx[529]: #17015 client reset connection
    Aug 23 13:00:41 	pftpx[529]: #17015 client reset connection
    Aug 23 13:00:44 	pftpx[529]: #17016 client reset connection
    Aug 23 13:00:44 	pftpx[529]: #17016 client reset connection
    Aug 23 13:00:44 	pftpx[529]: #17017 client reset connection
    Aug 23 13:00:44 	pftpx[529]: #17017 client reset connection
    
    

    If it helps diagnose any, sometimes Ill click on something in the webGUI of pfSense, and it will not load unless I click on it a few more times. Also, I contacted my ISP and they see no trouble on their end.



  • I know this may sound paranoid, but could I be getting attacked? I left here last night with an intense scan going for 192.168.. using zenmap. Maybe someone was hit by the scan and is retaliating? I did this so I can try and figure out the network topology here. 192.168… is my local network, so it wouldn't make sense for the scan to reach outside the building. Are there logs I can check to rid me of this hypothesis?


  • Netgate Administrator

    The log entries are from pftpx, the FTP helper. This can be enabled or disabled on each interface.
    Sounds like some thing is trying to use FTP through your box that wasn't happening until recently.

    What version of pfSense are you running? What sort of network is it firewalling? Are you running any packages?

    Steve



  • FYI, the problem seems to be gone - but I would like to figure out what happened.

    I am running pfSense 1.2.3

    Our main FTP server is accessed quite a bit from outside our network. The interface LAN has FTP helper enabled, the other two - WAN and OPT1 has the option "Disable the userland FTP-Proxy application" checked.

    There are no packages installed. Sorry I sound dumb, but I am unsure of what you mean by what kind of network it is firewalling. We have a cable internet line and a DSL line going into the pfSense box, with another interface (LAN) connecting it to the local users' switch (about 40 users).

    Is there a way to figure out which "client" reset the connection so many times per minute? Because the problem seemed to fix itself once I turned off all the users' machines… They are all back on now, and its not happening anymore. I thought it would happen again once a certain machine was turned back on and logged into, but it did not. Here is my system log now, I don't understand the top 3:

    
    Aug 23 22:20:00 	check_reload_status: check_reload_status is starting
    Aug 23 14:11:24 	syslogd: kernel boot file is /boot/kernel/kernel
    Aug 23 14:11:24 	syslogd: exiting on signal 15
    Aug 23 14:11:24 	pftpx[529]: #31261 client reset connection
    Aug 23 14:11:24 	pftpx[529]: #31261 client reset connection
    Aug 23 14:11:19 	pftpx[529]: #31260 client reset connection
    Aug 23 14:11:19 	pftpx[529]: #31260 client reset connection
    Aug 23 14:11:18 	pftpx[529]: #31259 client reset connection
    Aug 23 14:11:18 	pftpx[529]: #31259 client reset connection
    Aug 23 14:11:18 	pftpx[529]: #31258 client reset connection
    Aug 23 14:11:18 	pftpx[529]: #31258 client reset connection
    Aug 23 14:11:17 	pftpx[529]: #31257 client reset connection
    Aug 23 14:11:17 	pftpx[529]: #31257 client reset connection
    Aug 23 14:11:17 	pftpx[529]: #31256 client reset connection
    Aug 23 14:11:17 	pftpx[529]: #31256 client reset connection
    Aug 23 14:11:17 	pftpx[529]: accept failed: Software caused connection abort
    Aug 23 14:11:17 	pftpx[529]: accept failed: Software caused connection abort
    Aug 23 14:11:17 	pftpx[529]: #31255 client reset connection
    Aug 23 14:11:17 	pftpx[529]: #31255 client reset connection
    Aug 23 14:11:17 	pftpx[529]: #31254 client reset connection
    Aug 23 14:11:17 	pftpx[529]: #31254 client reset connection
    Aug 23 14:11:17 	pftpx[529]: #31253 client reset connection
    Aug 23 14:11:17 	pftpx[529]: #31253 client reset connection
    Aug 23 14:11:17 	pftpx[529]: #31252 client reset connection
    Aug 23 14:11:17 	pftpx[529]: #31252 client reset connection
    Aug 23 14:11:17 	pftpx[529]: accept failed: Software caused connection abort
    Aug 23 14:11:17 	pftpx[529]: accept failed: Software caused connection abort
    Aug 23 14:11:17 	pftpx[529]: #31251 client reset connection
    Aug 23 14:11:17 	pftpx[529]: #31251 client reset connection
    Aug 23 14:11:17 	pftpx[529]: #31250 client reset connection
    Aug 23 14:11:17 	pftpx[529]: #31250 client reset connection
    Aug 23 14:11:17 	pftpx[529]: accept failed: Software caused connection abort
    Aug 23 14:11:17 	pftpx[529]: accept failed: Software caused connection abort
    Aug 23 14:11:17 	pftpx[529]: #31249 client reset connection
    Aug 23 14:11:17 	pftpx[529]: #31249 client reset connection
    Aug 23 14:11:16 	pftpx[529]: #31248 client reset connection
    Aug 23 14:11:16 	pftpx[529]: #31248 client reset connection
    Aug 23 14:11:16 	pftpx[529]: #31247 client reset connection
    Aug 23 14:11:16 	pftpx[529]: #31247 client reset connection
    Aug 23 14:11:16 	pftpx[529]: accept failed: Software caused connection abort
    Aug 23 14:11:16 	pftpx[529]: accept failed: Software caused connection abort
    Aug 23 14:11:16 	pftpx[529]: #31246 client reset connection
    Aug 23 14:11:16 	pftpx[529]: #31246 client reset connection
    Aug 23 14:11:16 	pftpx[529]: accept failed: Software caused connection abort
    Aug 23 14:11:16 	pftpx[529]: accept failed: Software caused connection abort
    Aug 23 14:11:16 	pftpx[529]: #31245 client reset connection
    Aug 23 14:11:16 	pftpx[529]: #31245 client reset connection
    Aug 23 14:11:16 	pftpx[529]: accept failed: Software caused connection abort
    Aug 23 14:11:16 	pftpx[529]: accept failed: Software caused connection abort
    Aug 23 14:11:16 	pftpx[529]: #31244 client reset connection
    
    

    Edit: Thanks for your reply stephen!


  • Netgate Administrator

    @mallen324:

    Sorry I sound dumb, but I am unsure of what you mean by what kind of network it is firewalling. We have a cable internet line and a DSL line going into the pfSense box, with another interface (LAN) connecting it to the local users' switch (about 40 users).

    That's exactly what I meant, I should have phrased it better.
    So you have two internet connections, are they setup for loadbalancing either in or out?
    Do you have just one subnet for your LAN containing all your clients and your FTP server?
    You have a total of 3 interfaces?

    @mallen324:

    Is there a way to figure out which "client" reset the connection so many times per minute? Because the problem seemed to fix itself once I turned off all the users' machines… They are all back on now, and its not happening anymore. I thought it would happen again once a certain machine was turned back on and logged into, but it did not. Here is my system log now, I don't understand the top 3:

    I wouldn't worry about the top three log entries. I see those syslogd messges in my logs from time to time (when the logs rotate?).

    I'm not sure how you would check the logs for pftpx clients. Does it keep a separate log file in /var/log? I don't have one there but I don't use it.

    Steve

    Edit: Actually it looks like I have it enabled on every interface but not a single log entry! (and no separate log)
    Edit Again: Doh! No it's not enabled on any interface (no interfaces highlighted).



  • Yeah it is set up for load balancing. I don't understand how it is, so I'll attach a screenshot with the IPs blocked out.

    The FTP server has two NICs. One is for internal (LAN) traffic to access it, so that would be on the same subnet. The other NIC is configured for with an external IP for people abroad to connect to it - so that has a different subnet. Should I be checking the logs of the FTP server? If so, where would those logs be? It's a FreeBSD server that I, nor anyone who's still working here set up.

    I can't find any logs in /var/log that identifies pftpx clients. Yes, I have a total of 3 interfaces.

    Sorry, but I am confused by:

    Edit: Actually it looks like I have it enabled on every interface but not a single log entry! (and no separate log)
    Edit Again: Doh! No it's not enabled on any interface (no interfaces highlighted).

    Just not sure what you meant by 'it'.

    Thanks for your help so far.

    ![pfsense screen1.gif](/public/imported_attachments/1/pfsense screen1.gif)
    ![pfsense screen1.gif_thumb](/public/imported_attachments/1/pfsense screen1.gif_thumb)


  • Netgate Administrator

    By 'it' I was referring to the FTP proxy helper. When I looked on my box, which is pfSense 2.0, the GUI lists all my interfaces under 'Choose the interfaces where you want TFTP proxy helper to be enabled.' However the, slightly confusing IMHO, way in which pfSense works is that you highlight the interfaces you want in order to select them.

    Load balancing works better and is easier to setup in 2.0, if you decide to upgrade.

    Searching through the forum most references to pftpx seem to lead to malfunctioning software on an internal machine. But it could be something else.

    This may be of some help. http://devwiki.pfsense.org/PftpxManPage

    Steve



  • Thanks steve. Yeah I believe it was some malfunctioning software as well.

    I don't think I'll upgrade anytime soon, unless I continue to have problems. I just don't want to deal with the hassle of fixing a working box.


Log in to reply