Squid3 problems

jwhostet

I've tried once but no success. Sad

Sad day…

so if soft updates were enabled I wouldn't be having this issue where my internet slowly stops responding?
If that is the case I may just reinstall pfsense!!! (probably not but the thought did cross my mind)

marcelloc

Try to set vfs.read_max to 128 on system->advanced->system tunables.

It will increase disk performance on most systems.

jwhostet

@marcelloc:

Try to set vfs.read_max to 128 on system->advanced->system tunables.

It will increase disk performance on most systems.

…Things just got a little more sad...

I've already increased that
@jwhostet:

nmbclusters set to 32768
vfs.read=128

No big deal. Thanks for all the help! |If I manage to start over fresh again… I'll let you know my results! (things are slow right now, so the possibility is good!) :)

jwhostet

@marcelloc:

did you enabled soft updates on /usr during install to improve disk performance?

One last question… if soft updates were turned on during the install would that have solved my problem with the speed issue I'm having?

marcelloc

I do not install any squid/dansguardian/postfix install without it.

I've never tried read_max 128.

jwhostet

UPDATE

From a live cd 20121018-0934 64bit I reinstalled pfsense I then made sure everything was working like it was. Everything is good to go no problems.
To make things better I now have soft updates on for /var and /usr:

[2.1-BETA0][admin@pfsense.pcd.domain]/root(3): mount
/dev/ad6s1a on / (ufs, local)
devfs on /dev (devfs, local)
/dev/ad6s1d on /var (ufs, local, soft-updates)
/dev/ad6s1e on /usr (ufs, local, soft-updates)
devfs on /var/dhcpd/dev (devfs, local)

I have enabled squid and things are not looking so good. :(
After enabling my speeds have dropped dramatically within the first hour of enabling squid.
Same settings as before!

What now?! Why is this happening? :(

Any other ideas?

Thanks!

jwhostet

Well I spoke to soon seems soft updates might have solved the speed dropping issue. It seems to use almost all 4gb of memory but not complaining too much.

Is it normal to use next to all of the available memory? What if I had other packages that needed to use memory too? ( As of right now I do not but soon I wish to implement others.)

marcelloc

@jwhostet:

Is it normal to use next to all of the available memory? What if I had other packages that needed to use memory too? ( As of right now I do not but soon I wish to implement others.)

yes, It's normal.

You can try to reduce squid memory cache or add more memory to keep a good performance :)

jwhostet

:(

well… apparently I spoke too soon again!

Today I get a message saying the internet is got a 2 second ping and 200kb download which is terrible... after removing squid package everything goes back to normal 20ms ping 32mb download.

I'm assuming squid has been running normal and fine because the load on the router was minimal 5 systems hitting the system hard. Whereas today with a couple more technicians I'm assuming more system continually pulling data.

My question now is why is this happening? Is it hardware related? I estimate about 10gb of bandwidth a day is used. Are we accessing so much data that the system can't keep up?

Any help is once again appreciated, hopefully we can get this solved. If you would like to know any hardware specifics or a specific configuration file let me know and I will get you what you need.

Thanks, Josh

jwhostet

Is my thread dead? :(

I have a system on the network that I thought might have been causing squid to go crazy and hang draining internet to a stop.
So I added it to the list that are supposed to bypass proxy but it did not help solve the issue.

Any body have any ideas?

marcelloc

@jwhostet:

Any body have any ideas?

Can you try to include this on squid.sh startup script after line 4.
Remember to do not save squid config on gui while you test.

sysctl kern.ipc.nmbclusters=65536
sysctl kern.ipc.somaxconn=16384
sysctl kern.maxfiles=131072
sysctl kern.maxfilesperproc=104856
sysctl kern.threads.max_threads_per_proc=4096

jwhostet

Hey thanks for getting back to me!! :)

sorry I didn't get back sooner, school got kinda hectic on me for a couple days.

I can definitely try to do that!
One question though, how do i not save squid config on gui?

Thanks again for the help…. I feel repetitive but I do truly mean it!!!!!

marcelloc

@jwhostet:

One question though, how do i not save squid config on gui?

do not press save on squid gui after changing rc.d file ;)

jwhostet

Haha…. Oh that makes sense. Thanks for the help I forgot this went on to a second page Till right now. I'll probably start testing that tomorrow and hope things go well to report back with good news!!

jwhostet

Well we are having major ISP issues here at work so it might be a couple days before I can truly test your possible solutions will post back asap with results!

marcelloc

Ok :)

quetzalcoatl

I love squid but i find too many limitations, at least with when i use it in pfsense. (i don't like how it works in other platforms)

What is going on right now is that https pages cannot be access with the squid3 package installed from pfsense.
I can use lusca and it works well with both http and https pages but it's very outdated and is not being updated an maintained anymore.

So i want to update my pfsense platform but i really feel that squid3 is useless if it doesn't allow me to work with https just like lusca and squid 2.7 without using crazy configurations.

Also squid crashes when i set the maximum object size in RAM above about 450 KB.

In fact i assign 32 GB of RAM to PFSense of which 20 GB are for dedicated for Squid RAM caching and the only way to use all that RAM and make it useful is having the maximum limit of at least 512 KB or even 1024KB for objects stored in RAM.

marcelloc

@quetzalcoatl:

What is going on right now is that https pages cannot be access with the squid3 package installed from pfsense.

check dns_v4_first option on squid config gui and try again.

quetzalcoatl

Marcellok so you say that i have to set dns_v4_first off or on?

This is a brand new command that i just discovered thanks to you Marcellok
The problem is that in the squid guide it doesn't say all the values, just the default
and talks about this command: –enable-internal-dns

I wonder if i have to stick --enable-internal-dns with all the other squid settings

http://www.squid-cache.org/Doc/config/dns_v4_first/

With the IPv6 Internet being as fast or faster than IPv4 Internet
for most networks Squid prefers to contact websites over IPv6.

This option reverses the order of preference to make Squid contact
dual-stack websites over IPv4 first. Squid will still perform both
IPv6 and IPv4 DNS lookups before connecting.

WARNING:
This option will restrict the situations under which IPv6
connectivity is used (and tested). Hiding network problems
which would otherwise be detected and warned about.

Also do you know what are maximum size for objects in RAM?

Can i safely set 4096 KB as maximum Object size in RAM?

I used to cache in disk objects big as 9 Gigabytes (a full dvd) with lusca and it works, but if i go beyond 512Kilobytes of object size in RAM sometimes quid crashes.

Marcellok do you work for the squid guys or you just work in the squid3 package for pfsense.

If you work for the squid-cache people, can you please import the ultra aggressive caching features of lusca inside squid? Because lusca is actually a tuned up squid.

Thanks for your support!

jwhostet

Well major issues here… first ISP issue then after they finally came back online I find out there is some sort of failover that my GM added me into. So now im limited to 10Mbps till a new card is installed in his failover box. So I might have to scrap my whole setup/squid may just end up being a basic router... Or nothing.

Thanks for all the help, if something changes in the next few days and I get everything I had before back I will try your fix and report back.

Or I will create another post and start over new.

Again thanks!
Josh