Error The requested URL could not be retrieved cant seem to resolve



  • hi
    so I been running the squid proxy for a month or 2 now.. now past few days I been getting

    ERROR

    The requested URL could not be retrieved

    The following error was encountered while trying to retrieve the URL: http://www.msn.com/?

    Connection to 204.79.197.203 failed.

    The system returned: (51) Network is unreachable

    The remote host or network may be down. Please try the request again.

    Your cache administrator is

    now I found if I turn off my vpn and restart it then websites work again for a day.. then I get this error.. I did find if I disable squid proxy... websites work fine... I restart squidproxy back to the error...
    I even set the local cache hard drive cache to 20000
    and memory cache 512mb and maximum to 1024mb
    but still get the error.. is something else I need to adjust
    I take it the error is coming up because the cache is full ??



  • i still cant seem to resolve this issue... i disable it i can goto websites... i enable... and she gives me the above error... i even tried under local cache to clear it right away button.... and that did nothing to solve this issue... any ideas whats causing this error? or how to fix it



  • so I have uninstalled squid rebooted... several times reinstalled
    I only have 5mbs download so I wanted a proxy to help boost my internet by caching
    but no luck
    i even set the local cache to 200megbytes.. had it at 2000 and 20000 but doesn't help
    has no one come up with this
    0_1540216096467_squid.JPG



  • When you say help boost your internet, what is your expectation? Is 5 Mbps the download speed you are paying for from your ISP? You may already know this, but a web proxy server can only do so much. It will not increase (boost?) throughput to and from your ISP. However, it will in theory allow using less bandwidth for accessing sites that are commonly visited (cached) since it doesn't have to go out on the web to do so.

    I don't think your problem is with the local cache settings since you cleared the cache and that didn't help. How do you have squid setup? Is it transparent or explicit? Are you doing ssl filtering? It sounds like you are and it could be a certificate issue. If ssl filtering is enabled, try disabling it. Does that make a difference? That might help narrow it down.

    What about your DNS settings? Are you using the default Unbound setup or forwarding to other servers?

    Raffi



  • To give you an idea of what the traffic generally looks like in my office network. We have about 10 users and looking at the monitoring graphs, on average we are only using a few hundred kbps (not even 1 Mbps of our bandwidth) since most of what we do is accessing websites. In other words accessing websites is not a major bandwidth hog in any case so addressing that with a proxy will not be saving you a lot. With that said, we are using a proxy, but again that is not the reason for the low bandwidth usage. The main thing the proxy will do is help with page loading times, save you a few milliseconds maybe. Or if you have a network of hundreds or thousands of systems accessing websites, then it will make a bigger difference since collectively the bandwidth usage would accumulate if not using cached pages.

    Raffi



  • what i meant for boost my internet.. so like certain images like your google search image doesn't have to be reloaded each time.. so i could have less downloading each time.. same for what i read if i download a file from a website and i wanna do it on a couple computers.. its supposed to download it from the proxy server so instead of me downloading a 200 meg file twice.. and i know i could use a usb and copy but i just used it as an example... or the windows updates.. if 1 computer downloads it the proxy server supposed to hold it and then the other computers getting same file would get it much faster.. that's what i watched videos or read about the proxy server
    and ya my internet is 5megbits down and 300-500k upload
    so not the best but better then dial up lol

    as for the way i setup the squid...all i did was installed it.. click enable and that was it because when i watched older videos of pfsense of the older version none of the setttings changed.. so all that was changed was the megabits... because orginally when i had this running on a USB i found out the proxy server bricked the usb too many writes to the usb… and what i wanted was a USB to boot and a hard drive one of my old 1TB7200rpm drives to be the cache but i couldn't find a way to choose the hard drive to cache it too.. so now i running off a 250gb ssd and set it to like 20000megabits.. but i just left everything to default.. since i seen from old youtube

    as for certicate and ssl filtering have no idea.. explain it for dummies like the books.. i just a regular joe blow not a IT guy well went to school for IT never got job in it.. so youd have to explain where to disable etc.. i using pfsense because it was more powerful then my asus router.. but i find the asus router dummie proof... as i have made many a mistake with this pfsense as i learn as i go

    sooo if a proxy really good for loading websites but you mentioned downst really use that much bandwith.. but wouldn't it help my pages load a lot..
    im just learning as i go... and im mostly using the internet .. house of 2 people but i have my webserver running behind the pfsense.. and i wanna add some Vlans learning about that too..

    i wish i could get 2 5mbps ISP and then do this thing to make it 10mpbs but the ISPs don't offer that but some other company offers it but im sure costs too much to be worth while for 10mbps



  • as for the dns settings.. its just the pfsense as a dns i don't do any forwarding.. i don't think.. i use nordvpn so all computers are behind the vpn not sure the unbound setup either sorry i still new at this program.. it sure has more options then my asus router



  • Considering how squid is practically useless for caching these days, I would wonder if it's even worth the hassle of running it at all.



  • oh doesn't it work and why is it useless these days.. is that because faster internet?
    so what would help me or nothing really like i mentioned i was watching youtube videos how proxy server help with windows updates on multiple computers and files you download if the same.. i just trying to maximise my internet connection which is just the slowest option in dsl



  • @kom very true. I would agree with kom on this. The reason the youtube videos you are watching on squid are old is because squid was likely more relevant then for caching purposes. I don't think it is going to help you very much in your situation comet.

    Also, I'm not familiar with it, but it looks like NordVPN is 3rd party service that creates a VPN connection for individual devices. I assume you must use a client app on each PC or device to connect to the service? Are all your computers individually connecting to NordVPN? If so, I don't see how pfSense, or more specifically squid will be very helpful. If your network has limited speed, a VPN is only going to make it more slow. In fact, if each PC is connecting directly through NordVPN, wouldn't it be impossible for squid to work as a local caching server since all traffic (and web page requests) from each client will go through the encrypted tunnel to NordVPN servers first?



  • i was watching the youtube videos of the old pfsense version
    so why isnt squid proxy revleant anymore still not sure??? as for the vpn.. no i have nordvpn setup client on Pfsense so any computer connected on the network is covered... i use OpenVPN client under pfsense OpenVPN so thats how i do the vpn.. so proxy wouldnt help me..so it only helps people with the faster internet connection?



  • @comet424 said in Error The requested URL could not be retrieved cant seem to resolve:

    so why isnt squid proxy revleant anymore still not sure???

    Because the dynamic web is extremely hard to cache these days without you being a master guru at squid refresh patterns and store_ids. When I was running squid with cache at my company, the hit rate was something like 4-7%. That's terrible. Others I spoke to had similar results. It caused problems with Windows Updates, Netflix YouTube and other services. These days I still use it for URL filtering with squidguard but have the cache turned off.



  • KOM is right, for caching purposes I'm getting extremely low hit rates (see below). It's normally not even 1%. The total average is 0.76%. The hit % or hit rate is a measure of how often the cache is being used.
    The main reason for the proxy these days is url filtering with squidguard and/or for security with clamav. I personally use it for security, although even that is arguably not very effective. Below is a link of other discussions on this exact topic, so you can get other people's opinion on it.
    https://community.spiceworks.com/topic/85181-are-proxy-servers-still-necessary

    0_1540301652271_hit rate.JPG



  • ok ill read article.. so what is dynamic web compared to whatever it was before? ok so i wont use squid proxy.. and what is the url filtering is that to say add that to porn hub so no one can goto that website so i don't want my kids accessing that website...
    and what does the squidguard do i seen that as an option.. and what hs the clamav.

    and another question well 2
    1.. with windows you get virus's does pfsense get virus's?
    2.. when i was in college back in late 90s we used BlackICE to test your network for security leaks.. whats good now a days.. that tells me pfsense has closed off anything and hackers cant get in etc.. i googled some some didn't work some where out of date.. but how you guys test your network.. and remember i just home use not network company


  • Rebel Alliance Global Moderator

    So you have 5mbps internet, and you want to run that through a vpn to make it even slower.. Have fun with that.

    Dynamic internet vs static means that almost every page you hit these days is dynamically produced on the fly.. Not just static text or images.. So a proxy has a hard time storing information in its cache and then knowing that the the next browser actually wants the same exact stuff vs what the site is dynamically creating which is just slightly different.. So you storing info in the cache is not going to save you much of anything - and could actually slow you done.

    Out of the box pfsense blocks ALL inbound unsolicited traffic - so what exactly are you going to test? If your talking about pentesting/security tools or vulnerability scanners.. Say security onion distro or kali these sorts of toolkits are going to come with a HUGE learning curve for your home user.. HUGE!!! Something as simple to use as say nessus home can be overly complex for your typical home user.. But sure you might want to look into that for scanning your local machines for security concerns..

    No pfsense is not going to get viruses ;)
    Keep in mind that if you want to scan pfsense to validate nothing is open - it has to be done from OUTSIDE your network.. You can not do a valid scan of your wan/internet rules from inside your network. You could only validate your own lan rules.

    Why do you need to block p0rnhub as example? Do you have teenage boys or something in your home that your wanting to stop from surfing porn? Do they not have phones with data plans? In a home setup I find the use of a proxy of really zero value..



  • Yes, squidguard is the package which allows doing url filtering and it works along with squid. Url filtering is exactly what you said, you can block porn or any other sites you don't want people to access. There are many publicly available lists of sites which are maintained and constantly being updated with known bad url's and IP's. If you use those lists and keep them up to date, then it reduces your chances of getting malware for instance. It is another layer of security. Pfblockerng is another package that does url filtering. That is what I use for url filtering since it's easy to set up and keep the lists updated. Search the forums for those packages if you are interested in learning more on them. I'm sure many of your questions have already been asked and answered.

    PfSense runs on FreeBSD. It is an operating system which like any other operating system can be vulnerable to attacks and viruses. I wouldn't worry about that though. You can't think of pfSense like a Windows computer. It's not the same. Your biggest concern like on any network is making sure your users don't do something dumb like go to sites they shouldn't or open email attachments they shouldn't. That is something that you can't control, but security experts try to reduce as much as possible. Right out of the box with default settings, pfSense is already more secure than any commercially available home router. I would suggest you do a lot of searching on these forums and elsewhere to learn more. It took me personally months of searching, trial and error, just to begin to understand what pfSense is capable of.

    When you have specific issues and can't find the answers after searching the forums and else, the pfSense community is usually very helpful.

    Raffi



  • ah ok all good info i wont bother using the squid then..
    as for the porn question i was just using it as a random thing for the question about url filitering.. no teens.. i do have 12 and 9 yr old.. i not worried i was just making a reference..

    but ok ill check out he other info and ill scrap the squid

    and ya 5mps is all you can get in the country.. you cant tap in the fiber in front of my driveway and they only offered on phone line.. unless i lived in town...

    thanks for the feed back.. and ill just disabled it..

    thank you



  • @comet424 I forgot to mention that the best source for information on pfSense is in the book written by the experts. Recently it has been made free to the public. Even when it was not free, it was worth every penny.
    https://www.netgate.com/docs/pfsense/book/

    Good luck
    Raffi