Really Strange Behaviour - Have I been Hacked?
-
@chpalmer said in Really Strange Behaviour - Have I been Hacked?:
You are on a cable system and your modem must have gone offline for some reason. The modem itself will hand out dhcp addresses in that range when the ISP is not reachable so that you can still reach the modem at 192.168.100.1 in order to troubleshoot.
Thanks @chpalmer I believe you are right.... Power cycling the Cable modem and then the ATA resolved the problem. (I don't know if the ATA was a "must", but I did it just to be sure - didn't reset pfSense).
I'm wondering why this only effected the ATA? Other computers were working fine and resetting the ATA made no difference.
Any idea why this 192.168.100.10 address would be persistent? (Would it be in the DNS resolver?)
Would there be a way of clearing this without resetting the cable modem? My wife was on a video training, so I had to go several hours without my phone because I had to wait for her to finish before I reset the cable modem.
-
Im not sure why it would be persistent.. Is your ATA somehow connected directly to the cable modem or do you have ports forwarded to it?
-
@guardian In "Interfaces/WAN/DHCP Client Configuration/Reject leases from" you can add the administrative IP address of your modem, 100.1 in my case.
-
@provels The entire piece of text from that pfsense setting is helpful in this case, since I also believe this is exactly what's happening.
"To have the DHCP client reject offers from specific DHCP servers, enter their IP addresses here (separate multiple entries with a comma). This is useful for rejecting leases from cable modems that offer private IP addresses when they lose upstream sync."
Jeff
-
Yeah, without doing that, I'd be in trouble when VPN'd in. If the line dropped (which happens), pfSense would happily take its WAN address from the modem, and the two would sit there happily looking at each other with nowhere to go and no one getting in or out without a reboot.
-
@chpalmer said in Really Strange Behaviour - Have I been Hacked?:
Im not sure why it would be persistent.. Is your ATA somehow connected directly to the cable modem or do you have ports forwarded to it?
@chpalmer, there are no port forwards... the ATA calls out to the provider which opens the firewall as required. I checked at the time and the ATA had the correct IP address 172.x.x.x as a result of DHCP assignment from pfSense.
@provels said in Really Strange Behaviour - Have I been Hacked?:
without doing that, I'd be in trouble when VPN'd in. If the line dropped (which happens), pfSense would happily take its WAN address from the modem, and the two would sit there happily looking at each other with nowhere to go and no one getting in or out without a
@provels, I have had the problem before, but it was system wide on all machines. If it was a probem, my Linux desktop and my wife's Win8.1 PC managed to figure out what to do, because they still worked, or pfSense was caching the old data.
@akuma1x said in Really Strange Behaviour - Have I been Hacked?:
@provels The entire piece of text from that pfsense setting is helpful in this case, since I also believe this is exactly what's happening.
"To have the DHCP client reject offers from specific DHCP servers, enter their IP addresses here (separate multiple entries with a comma). This is useful for rejecting leases from cable modems that offer private IP addresses when they lose upstream sync."
@provels said in Really Strange Behaviour - Have I been Hacked?:
@guardian In "Interfaces/WAN/DHCP Client Configuration/Reject leases from" you can add the administrative IP address of your modem, 100.1 in my case.
@provels/@akuma1x thanks very much for this!
As a practical matter this issue is solved, a reboot fixed the connectivity issues and your fix this should keep the problem from reoccurring. Thanks very much for all who responded.
===========================================================
Quesion for discussion/to improve group knowledge/understanding of pfSense operation:
Why the issue was partial and did not cause problems with the desktop PCs etc.?
===========================================================The Following info are what makes this problem interesting:
-
I didn't notice the ATA was offline until late afternoon because my PC and my wife's PC was working fine.
-
The ATA was working correctly/had the correct ip address and had been rebooted. pfSense changed the IP address on the outgoing WAN interface which would seem to indicate an internal routing table that was only partially updated when the WAN changed.
-
I have an OpenVPN client plugin runnning of pfSense that I use for my WiFi network, and it figured things out and kept working.
-
I have a cron job that runs on the pfSense box and pings my web server--the log entries stopped for about 30/40 minutes in the early morning, and then resumed.
This means that part of pfSense had recovered on it's own and part of it didn't.
-
-
@guardian said in Really Strange Behaviour - Have I been Hacked?:
Quesion for discussion/to improve group knowledge/understanding of pfSense operation:
Why the issue was partial and did not cause problems with the desktop PCs etc.?Guessing difference in DHCP lease time config embedded in the clients. The one requesting renewal first might get issued a bum address from the modem while slower units would retain the old IP while the circuit bounces. Perhaps if you had rebooted the ATA, it would have recovered?
-
Lease time from cable modems is really short. Just minutes. Most likely the ATA would have held open the firewall state in pfsense pretty much indefinitely and thus the modems "fall back" address range would have stayed there.
Sounds like a perfect storm. The ATA may have tried to re-register as your WAN went down and maybe was just coming back up..
I assume you saw the 192.168.100.0/ subnet in your firewall states..??
-
@provels said in Really Strange Behaviour - Have I been Hacked?:
@guardian said in Really Strange Behaviour - Have I been Hacked?:
Quesion for discussion/to improve group knowledge/understanding of pfSense operation:
Why the issue was partial and did not cause problems with the desktop PCs etc.?Guessing difference in DHCP lease time config embedded in the clients. The one requesting renewal firs
t might get issued a bum address from the modem while slower units would retain the old IP while the circuit bounces. Perhaps if you had rebooted the ATA, it would have recovered?@chpalmer said in Really Strange Behaviour - Have I been Hacked?:
Lease time from cable modems is really short. Just minutes. Most likely the ATA would have held open the firewall state in pfsense pretty much indefinitely and thus the modems "fall back" address range would have stayed there.
Sounds like a perfect storm. The ATA may have tried to re-register as your WAN went down and maybe was just coming back up..
I assume you saw the 192.168.100.0/ subnet in your firewall states..??
Hi @provels/@chpalmer, thanks for the reply
The ATA was rebooted, and since it was in it's own VLAN, it got a 172.x.x.x IP addresss from the pfSense DHCP server. The issue is definitely with pfSense routing/NAT. From the packet capture taken by pfSense on the WAN interface, the source address for the outgoing register events was 192.168.100.10 which has been correctly identified as coming from the cable modem as a result of a loss of connnectivity with the internet.I should have checked the outgoing firewall states, but I think @chpalmer is likely correct about the firewall state persisting. (Any idea what the defaut timeout is? I searched the docs, and I could see how to change it, but not how to find out what the current values are.)
My theory is based on our discussion here:
Since the ATA reboot takes only about 30s or less, and would have immediately sent register packets from 172.x.x.x -> VoipProvider, pfSense would have assumed it was part of the same session, and the NAT routing table would have kept the 192.168.100.10 address and therefore sent all registration packets with that address as a source address--hence ATA never reconnects. Does that sound correct?When I rebooted the cable modem (tales several minutes), that was enough time for all the states to drop, and the ATA packets were then routed correctly. IIUC, I could have got to the same place by simply clearing the state table.
As recommended by @provels I have also refused DHCP offers from 192.168.100.0/24, so that will hopefully prevent the problem from occurring in the first place.
Follow up question:
If the cable modem successfully reconnected, (which it had already done many hours before I noticed the problem), it seems to me that the quickest way to get back to normal is simply flushing the state table.. That is a lot quicker and less disruptive than rebooting the cable modem and/or pfSense. Are there any other states/tables that would persist and cause problems? -
SIP clients are designed to keep the connection live. 24/7. Some devices are better designed than others.
SIP was not originally designed to be behind NAT. NAT was hacked in (emphasis on hack) later when the idea of marketing to the residential and small business markets. Vonage was sued early on for patent infringement. Since then other carriers are being very careful to keep out of that particular court room and thus everyone does things just a little different.
The problem becomes when you as a customer of one company with their specific devices has an issue trying to find someone that knows that exact system and their requirements/method of operation can be difficult. Generally things are close enough and the knowledge that is bestowed is usually enough. But little things can crop up and stimie everyone..
You don't want your ATA states to expire normally. The whole idea is that a constant connection is kept active between the ATA/phone device and the carrier SIP server. Otherwise you would not be happy with the quality of your VOIP carrier.
-
@chpalmer said in Really Strange Behaviour - Have I been Hacked?:
SIP clients are designed to keep the connection live. 24/7. Some devices are better designed than others.
SIP was not originally designed to be behind NAT. NAT was hacked in (emphasis on hack) later when the idea of marketing to the residential and small business markets. Vonage was sued early on for patent infringement. Since then other carriers are being very careful to keep out of that particular court room and thus everyone does things just a little different.
The problem becomes when you as a customer of one company with their specific devices has an issue trying to find someone that knows that exact system and their requirements/method of operation can be difficult. Generally things are close enough and the knowledge that is bestowed is usually enough. But little things can crop up and stimie everyone..
You don't want your ATA states to expire normally. The whole idea is that a constant connection is kept active between the ATA/phone device and the carrier SIP server. Otherwise you would not be happy with the quality of your VOIP carrier.
Thanks for the reply @chpalmer - As a result of your email, I did a quick pcap to see what what going on (now that my system is functioning normally), and from what I can see the ATA sends a UDP packet about very 20-25s to keep the firewall open.
And I agree with you that documentation of SIP is somewhat "spotty"... you may have uncovered the reason why. I don't know when that was or when the suit occurred, but IIUC a patent is good for 17 years, so it should hopefully be expiring soon as this is a very mature protocol.