[SOLVED] Need help troubleshooting a web site access problem
-
My network appears to be working normally, except that a single web site (https://client.schwab.com) is no longer reachable through my firewall, which is running pfsense 2.1.
I can reach the site without problems, if I plug a laptop directly into my cable modem, bypassing my pfsense firewall.
I'm running a plain-vanilla pfsense install, with no custom firewall rules, and I do not see any evidence in the firewall logs of the site being filtered out. Other sites, with and without SSL, work fine.
The specific behavior I'm getting: enter URL into web browser; browser never gets any response from the site. The behavior is exactly the same with all of these combinations: firefox/Win7, IE11/Win7, firefox/linux, safari/ios7.
Under failing conditions: (1) a DNS lookup returns the same IP address as that returned under working conditions, and (2) traceroute succeeds quickly.
I've done a bit of searching here and elsewhere on the Internet, but I'm out of ideas. Any suggestions about how to diagnose this problem would be greatly appreciated.
-
Restart your cable modem. A friend of mine was telling me he was having this exact problem and that is what fixed his issue. This is a shot in the dark but who knows.
-
Restart your cable modem.
Ah, something I'd overlooked. Unfortunately, it didn't help, but thanks for the suggestion.
I've also tried connecting a PC directly to the pfsense router, with no switches in the path, and that also made no difference.
This one has me stumped. Everything seems to work, except that having the router in the path makes one web site fail. It used to work, and I'm not aware of anything having changed.
Is there a way to make pfsense give me more visibility into what's happening?
-
Often this can be caused by something in the chain having an abnormally small MTU setting causing normal sized packets to be rejected. Try pinging with increasing packet size or tracerouting similarly. That doesn't really explain why it doesn't effect a machine connected to the modem directly, unless perhaps the usual protocols that reduce the packet size are not working for some reason.
https://doc.pfsense.org/index.php/Unable_to_Access_Some_Websites
Steve
-
Often this can be caused by something in the chain having an abnormally small MTU setting causing normal sized packets to be rejected. Try pinging with increasing packet size or tracerouting similarly. …
Thanks!
The destination won't respond to pings, so I'm trying to use traceroute.
Traceroute, issued from the router's GUI or from a Win7 PC connected to the router, succeeds, but there is no option to vary the packet size.
Traceroute on my linux box has a –mtu option, but it can't reach the final destination, with or without the --mtu option. It always gets all the way to the second-last node, and then times out trying to reach the last node. It does indicate that everything in the path up to the penultimate node can handle an MTU of 1500.
By the way, a reverse IP lookup of the destination IP yields unknown.prolexic.com. Perhaps prolexic somehow thinks I'm a bad guy, although I can't see how bypassing my router and connecting directly to my cable modem would change its judgment.
-
Hmm. I have no difficulty accessing that site and I am using default MTU settings.
Was the site ever reachable through pfSense?
Steve
-
the prolexic reverse lookup is a big clue: they are in the business of mitigating DDOS attacks for their clients, and i'd guess client.schwab.com is a client of theirs.
I suspect you are right in thinking you are tagged as a bad guy and are being blocked. DDOS bad guys are known to use IP spoofing to try to get around IP based blocks. I don't know much about mitigation methods, but could they be looking at your MAC address? I don't think prolexic clients are routed full-time through prolexic servers, but only when the client is under attack, so you may not have the problem when traffic goes back to the client.
Maybe try to swap in a different nic into your pfsense machine? Or change your MAC?
-
Hmm. I have no difficulty accessing that site and I am using default MTU settings.
Was the site ever reachable through pfSense?Yes. The problem started happening some time in the last week or so. Up until then, I could reach the site, including times both before and after updating to pfsense 2.1.
-
… I suspect you are right in thinking you are tagged as a bad guy and are being blocked. DDOS bad guys are known to use IP spoofing to try to get around IP based blocks. I don't know much about mitigation methods, but could they be looking at your MAC address? ...
I'm pretty sure they can't see my MAC address, however you gave me an idea: I tried plugging my laptop directly into the cable modem again, and my laptop got a different IP address from the one my router gets!!! (When I plugged the router back in, it got its usual IP address.)
Since I want to go through the router, and I want to keep its current IP address, I think I need to take this up with the destination web site's tech support.
-
You guys beat me to the punch. Usually your ISP is going to assign you a IP based on your MAC address. If you change your mac address that should force your ISP to give you a new IP. Once you get a new IP try to access the site. If you can access the site then you know that they are blocking you. Is there a reason that you need the IP address that you have. If you are allowing incoming connections into your network, think about using dynamic DNS. I find it a lot more convenient and there are services out there that are free of charge.
Also I would ask the question why you? There has to be a reason why they are blocking you. It could be IP spoofing but could it be that one of the machines in your network is compromised? Just asking, it is probably worth taking a look at your systems to make sure there is nothing unusual going on.
-
I've got the problem fixed now.
Thanks to everyone for your help.
For those who are interested, here's the last chapter of the story.I learned from Schwab tech support: (1) a number of other customers in my area can't log in as of a couple of days ago, (2) prolexic does occasionally block their customers, and (3) they have no way to make prolexic unblock individual IP addresses. They readily accepted my statement that I get through when I come in via one IP address and get blocked when I come in via a different one. They suggested I try to change my IP address and said someone else would call back in a day or two with more ideas for working around the problem.
My ISP's tech support said they can't change my IP address. They say their IP addresses change, although I've observed that they don't change often, if at all. I've also now observed that different devices get different IP addresses, and if you swap back and forth, each device continues to get the IP address originally assigned to it.
Putting this all together, I decided to swap the LAN and WAN ports on my pfsense router, so that the upstream device would see a different MAC address and thus assign me a new IP address. I tried this, and it did, indeed, solve my problem!
So … the moral of this story is ... if you can't log in to schwab.com, swap the LAN and WAN ports on your pfsense router! ;)
W.r.t. the possibility that we may have a compromised computer behind the firewall: I'm as careful as I know how to be with the computers on this network. I do have a walled-off wireless network for house guests, whose traffic also flows through this router, but we haven't had any guests recently. If this new IP address gets black-listed, then I'll have to re-visit my security policy.
-
Nice. :)
Thanks for completing the story.Steve
-
so that the upstream device would see a different MAC address and thus assign me a new IP address.
I know with DD-WRT every time I needed a new IP address from my ISP I could simply just change the mac address of the WAN connection. Not sure if pfSense can do this or not.
-Jamie M.
-
Yes you can just spoof the MAC of the existing NIC which is probably easier for a test, no swapping cables and reassigning interfaces. However moving away from the real MAC can cause problems if only in remembering what it was!
Steve
-
and
In some cases, spoofing the MAC may require running the NIC in promiscuous mode.
from: https://doc.pfsense.org/index.php/Interface_Settings
that caught me for a while on one system a few months ago. -
My ISP's tech support guy said that, if a given MAC address isn't seen for a day or two, it's likely that it will get a different IP address the next time it's connected. Thus, if I ever need to change IP addresses again, I can probably just swap LAN and WAN again. It was very easy and quick with the pfsense GUI.
