Varnish help on setting up for multiple internal webservers



  • Hi,
    Just wondering if someone managed to use Varnish to redirect one public IP to various internal web servers according to hostname.

    I have a very basic setup using pfSense 2.0.1 AMD64.

    I installed varnish package but I am lost as I cannot find any information on how it's done.

    I have one public IP and 3 internal web servers.

    Basically I would like:

    www.mydomain.com > 192.168.0.2 (Apache under Centos)
    news.mydomain.com > 192.168.0.3 (IIS using Host headers)
    devs.mydomain.com > 192.168.0.3 (IIS using Host headers)

    Can someone please guide me on how this can be done?

    Thanks in advance.



  • First be sure your pfsense gui is not listening on port 80 (system-> advanced)

    Varnish config begins with backend configuration

    Each backend means a webserver

    inside each configuration, you must provide:

    • Backend name

    • Ip Address

    • Port

    • Probe URL

    • Probe Interval (5)

    • Probe Timeout (1)

    • Probe window (5)

    • Probe Threshold (3)

    on backend mapping, you could choose

    map=host
    match=equal
    expression=www.mydomain.com (for first web server)

    save backend config

    on settings

    • Listen port=80

    • Managment interface=127.0.0.1:85

    • storage type=memory

    • cache storage size=1024

    on General VCL settings, check all options

    all other options on settings could be in default value




  • I've also been trying to set this up with little luck. Could you show some more detail of how you have the backend's setup?



  • Just the way I posted above:

    Each backend means a webserver

    inside each configuration, you must provide:

    • Backend name = WindowsServer

    • Ip Address = 192.168.1.4

    • Port = 80

    • Probe URL = "/"(if your server has a valid page on "/")

    • Probe Interval = 5

    • Probe Timeout = 1

    • Probe window = 5

    • Probe Threshold = 3

    on backend mapping, you could choose



  • Hi Marcellob,
    Thanks for all your support.

    Can I kindly ask if I still need to port forward port 80 in NAT options?

    Are there any settings I need to configure in NAT or Rules to make it work?

    Thanks



  • If you are listening varnish on wan port 80, then you have to

    Disable any nat on wan port 80
    Change pfsense gui port to other then 80
    Create a rule on wan to Allow traffic to varnish port.



  • @marcelloc:

    If you are listening varnish on wan port 80, then you have to

    Disable any nat on wan port 80
    Change pfsense gui port to other then 80
    Create a rule on wan to Allow traffic to varnish port.

    do i need to any other setup if my dns is handle by no-ip.com? i can't reach the internal ip server with the above setup



  • Add varnish widget to see backend status.

    If you are getting 503 errors, means That you can reach varnish But not backends.



  • @marcelloc:

    Add varnish widget to see backend status.

    If you are getting 503 errors, means That you can reach varnish But not backends.

    "Create a rule on wan to Allow traffic to varnish port."

    you mean create a rule "any to any port80" ?

    my widget shows this now
    http://img254.imageshack.us/img254/3103/pfs1.jpg



  • The wan rule is to allow traffic to pfsense on public/wan ip on varnish listen port.



  • @marcelloc:

    The wan rule is to allow traffic to pfsense on public/wan ip on varnish listen port.

    yes..im listen to port 80

    the screenshot above…is the widget functioning? i dont see any data at all... looks abit weird



  • I've published a fix to widget download during instalation, please reinstall package, add the varnish dashboard widget and see backend status.



  • @marcelloc:

    I've published a fix to widget download during instalation, please reinstall package, add the varnish dashboard widget and see backend status.

    do i need to do this again?

    cd /usr/local/www/widgets/widgets/
    fetch http://www.pfsense.com/packages/config/varnish64/varnish.widget.php



  • @kegler:

    do i need to do this again?

    cd /usr/local/www/widgets/widgets/
    fetch http://www.pfsense.com/packages/config/varnish64/varnish.widget.php

    Just if you do not want to reinstall package.

    The fix I've published was just to fix this file path.



  • @marcelloc:

    @kegler:

    do i need to do this again?

    cd /usr/local/www/widgets/widgets/
    fetch http://www.pfsense.com/packages/config/varnish64/varnish.widget.php

    Just if you do not want to reinstall package.

    The fix I've published was just to fix this file path.

    oh…i still see the same widget

    ok let me recap, i disable all my NAT, i set any port80 to any port 80 on my rules and i set my backend servers on varnish. i still do not get thru, not even getting the error503. Just showing me the connection has time out after a 1-2min loading



  • maybe there is something missing on your varnish setup.

    try to run it from console to see if there are any erros

    /usr/local/etc/rc.d/varnish.sh start



  • @marcelloc:

    maybe there is something missing on your varnish setup.

    try to run it from console to see if there are any erros

    /usr/local/etc/rc.d/varnish.sh start

    omg… u r right.... i couldnt start.... varnish.sh command not found

    but i do see varnish.sh when i ls



  • /usr/local/etc/rc.d/varnish.sh is generated when you enable varnish and save config.

    Can you check your varnish settings tab?



  • Varnish configuration file

    Automatically generated by the pfSense package system

    This file is located in /var/etc/default.vcl

    sub vcl_error {
    if (obj.status == 503 && req.restarts < 2) {
    restart;
    }

    set obj.http.Content-Type = "text/html; charset=utf-8";
    synthetic {"
    "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
    FAIL TO LOAD VARNISH"};
    return(deliver);

    }

    backend kegwebBACKEND {

    used in catch_all

    .host = "192.168.1.95";
    .port = "80";
    .first_byte_timeout = 360s;
    .connect_timeout = 5s;
    .probe = {
    .url = ""/"";
    .interval = 5s;
    .timeout = 1s;
    .window = 5;
    .threshold = 3;
    }
    }

    backend kegapp3BACKEND {

    used in catch_all

    .host = "192.168.1.100";
    .port = "80";
    .first_byte_timeout = 360s;
    .connect_timeout = 60s;
    .probe = {
    .url = ""/"";
    .interval = 5s;
    .timeout = 1s;
    .window = 5;
    .threshold = 3;
    }
    }

    backend goldallianzeBACKEND {

    used in catch_all

    .host = "192.168.1.100";
    .port = "80";
    .first_byte_timeout = 360s;
    .connect_timeout = 60s;
    .probe = {
    .url = ""/"";
    .interval = 5s;
    .timeout = 1s;
    .window = 5;
    .threshold = 3;
    }
    }

    #Enable Per user session cache.
    sub vcl_hash {
    set req.hash += req.http.cookie;
    }

    sub vcl_recv {

    #BASIC VCL RULES SETTING
    #Fix gzip compression
    if (req.http.Accept-Encoding) {
    if (req.url ~ ".(gif|jpg|jpeg|bmp|png|ico|img|tga|wmf|gz|tgz|bz2|tbz|mp3|ogg)$") {
    unset req.http.Accept-Encoding;
    }
    else if (req.http.Accept-Encoding ~ "gzip") {
    set req.http.Accept-Encoding = "gzip";
    }
    else if (req.http.Accept-Encoding ~ "deflate") {
    set req.http.Accept-Encoding = "deflate";
    }
    else {
    unset req.http.Accept-Encoding;
    }
    }
    #set client balance identity
    set client.identity = client.ip;

    #set X-forward
    set req.http.X-Forwarded-Varnish = client.ip;

    if (req.http.host == "www.example.com") {
    set req.backend = kegwebBACKEND;
    }
    else if (req.http.host == "kcrm1.example.com") {
    set req.backend = kegapp3BACKEND;
    }
    else if (req.http.host == "kcrm2.example.com ") {
    set req.backend = goldallianzeBACKEND;
    }

    #respect client wish to refresh the page
    if (req.http.Pragma ~ "no-cache")
    {
    return(pass);
    }

    #BASIC VCL RULES ACTIONS
    #Disable post cache
    if (req.request == "POST") {
    return(pass);
    }
    #Enable static cache
    if (req.request=="GET" && req.url ~ ".(css|js|txt|zip|pdf|rtf|flv|swf|html|htm)$") {
    return(lookup);
    }
    if (req.request=="GET" && req.url ~ ".(gif|jpg|jpeg|bmp|png|ico|img|tga|wmf|mp3|ogg)$") {
    return(lookup);
    }
    #Be rfc2616 compliant
    if (req.request ~ "^(GET|HEAD|PUT|POST|TRACE|OPTIONS|DELETE)$") {
    return(lookup);
    }
    else {
    return(pipe);
    }

    return(lookup);
    }

    sub vcl_pipe {

    If we don't set the Connection: close header, any following

    requests from the client will also be piped through and

    left untouched by varnish. We don't want that.

    set req.http.connection = "close";

    Note: no "pipe" action here - we'll fall back to the default

    pipe method so that when any changes are made there, we

    still inherit them.

    }

    sub vcl_fetch {

    #Disable cache when backend is starting a session
    if (beresp.http.Set-Cookie && beresp.http.Set-Cookie ~ "(PHPSESSID|phpsessid)") {
    return(pass);
    }
    if (beresp.http.Set-Cookie && beresp.http.Set-Cookie ~ "(JSESSION|jsession)") {
    return(pass);
    }

    Varnish respects the wishes of the backend application.

    if (beresp.http.Pragma ~ "no-cache" || beresp.http.Cache-Control ~ "(no-cache|no-store|private)") {
    return(pass);
    }
        ## If the request to the backend returns a code other than 200, restart the loop
        ## If the number of restarts reaches the value of the parameter max_restarts,
        ## the request will be error'ed.  max_restarts defaults to 4.  This prevents
        ## an eternal loop in the event that, e.g., the object does not exist at all.
        if (beresp.status != 200 && beresp.status != 403 && beresp.status != 404 &&
        beresp.status != 303 && beresp.status != 302 && beresp.status != 301 && beresp.status != 401 ) {
    set beresp.saintmode = 60s;
    restart;
            }

    set beresp.grace = 60s;
    return(deliver);
    }

    sub vcl_deliver {
    ##set resp.http.X-Served-By = server.hostname;
      if (obj.hits > 0) {
        set resp.http.X-Cache = "HIT";
        set resp.http.X-Cache-Hits = obj.hits;
      } else {
        set resp.http.X-Cache = "MISS";
      }
    return(deliver);
    }



  • my pfsense management port i change to https

    change domain name on conf only for security purpose



  • The conf looks fine. Are you still unable to start varnish?
    What files do you have on /usr/local/etc/rc.d ?



  • @marcelloc:

    The conf looks fine. Are you still unable to start varnish?
    What files do you have on /usr/local/etc/rc.d ?

    varnish.sh
    varnishd
    varnishlog
    varnishncsa
    clamd



  • The file is there and package is installed, it should work.  ???

    Just to check, see if varnish package is installed with pkg_info on console.

    ps ax | grep varnish will show if there is any varnish process running.



  • @marcelloc:

    The file is there and package is installed, it should work.  ???

    Just to check, see if varnish package is installed with pkg_info on console.

    ps ax | grep varnish will show if there is any varnish process running.

    18796  0  S+    0:00.00 grep varnish

    looks like not running?



  • What about pkg_info cmd?

    What does returns /usr/local/etc/rc.d/varnish.sh start



  • @marcelloc:

    What about pkg_info cmd?

    What does returns /usr/local/etc/rc.d/varnish.sh start

    sorry what is the cmd to check the above ?



  • [2.0-RELEASE][admin@kegpfs.kegnet.com]/usr/local/etc/rc.d(6): varnish.sh start
    varnish.sh: Command not found.

    seems something not installed ?



  • Do not chdir to rc.d, Run varnish.sh with full path



  • @marcelloc:

    Do not chdir to rc.d, Run varnish.sh with full path

    kern.ipc.nmbclusters: 65536
    sysctl: kern.ipc.nmbclusters: Invalid argument
    kern.ipc.somaxconn: 16384 -> 16384
    kern.maxfiles: 131072 -> 131072
    kern.maxfilesperproc: 104856 -> 104856
    kern.threads.max_threads_per_proc: 4096 -> 4096

    wondering could it be due to my domain is host by no-ip.com?



  • php: /pkg_edit.php: The command '/usr/local/etc/rc.d/varnish.sh' returned exit code '2', the output was 'kern.ipc.nmbclusters: 65536 sysctl: kern.ipc.nmbclusters: Invalid argument kern.ipc.somaxconn: 16384 -> 16384 kern.maxfiles: 131072 -> 131072 kern.maxfilesperproc: 104856 -> 104856 kern.threads.max_threads_per_proc: 4096 -> 4096 storage_malloc: max size 1024 MB. Message from VCC-compiler: Expected ';' got '/' (program line 369), at (input Line 28 Pos 26) .url = ""/""; –-----------------------#--- In backend specification starting at: (input Line 21 Pos 1) backend kegapp33BACKEND { #######------------------ Running VCC-compiler failed, exit 1 VCL compilation failed'

    saw this in my log… but i dont quite understand what went wrong

    and i kvm into my pfsense and saw this on the screen.... seems like its stucked at the loading of varnish....

    i think i messed up the pfsense somewhere



  • Remove quotes from your backend Test url

    .url = ""/"";



  • omg…it worked now...thank you very much....



  • @marcelloc:

    Each backend means a webserver

    inside each configuration, you must provide:

    • Backend name = WindowsServer

    • Ip Address = 192.168.1.4

    • Port = 80 <– Q

    • Probe URL = "/"(if your server has a valid page on "/")

    • Probe Interval = 5

    • Probe Timeout = 1

    • Probe window = 5

    • Probe Threshold = 3

    Q:
    Can I enter multiple listening ports e.g. port: "80, 443"? Or should I configure that host several times according to how many ports it is listening on?

    Thanks,
    Canefield



  • I think you need one port for each config as varnish is going to check backend health as well send traffic based on this port.

    Keep in mind That varnish implemets http balance/cache only. If you what to balance https as well, haproxy can do this.

    Pound and relayd can do client/firewall ssl comunication and then forward client request to varnish, but there is no gui options for this yet.



  • Hate to jump on someone else's thread, but I've gone threw all the same motions, but still no results.

    web1.domain.com
    web2.domain.com

    All end up pointing to the pfSense login, so my question is simply this… What IP do you point the Rule to? the pfSense box that host Varnish?



  • Did you changed  pfsense gui port?
    This current varnish gui implementation listens deamon on all interfaces.



  • I did now and it works!

    Guess next time I need to slow down and read all the helpful text.

    Thank you



  • Correction, it was working. I didnt change anything and I'm getting the 503 errors again  :(

    –---------- Update ------------

    I only get the 503 error on 1 of the 2 domains.

    ------------ Update ------------

    Removing the Backend mapping with the http:// on it seemed to fix the issue.

    ------------ Update ------------

    Down again after about 30 seconds.  >:(



  • That's probably a backend health url check problem.

    Add varnish widget to dashboard to see backend status.

    The 503 error means could not find any available backend.

    Also try to see backend configuration file on view configuration.



  • Here is my config

    
    # Varnish configuration file
    # Automatically generated by the pfSense package system
    
    # This file is located in /var/etc/default.vcl
    
    sub vcl_error {
    		if (obj.status == 503 && req.restarts < 1) {
    			restart;
    		}
    
    	set obj.http.Content-Type = "text/html; charset=utf-8";
    	synthetic {"
    "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
    
    <title>"} obj.status " " obj.response {"</title>
    
     **We are very sorry but an error occurred during this request.** 
    
    Please press refresh in your browser to try again.
    
    Varnish Error "} obj.status " " obj.response {"
    
    "} obj.response {"
    
    ### Guru Meditation:
    
    XID: "} req.xid {"
    
    "};
    	return(deliver);
    
    }
    
    backend MinecraftBACKEND {
    	# used in catch_all 
    	.host = "10.0.0.50";
    	.port = "80";
    	.first_byte_timeout = 300s;
    	.connect_timeout = 25s;
    	.probe = {
    		.url = "/index.php";
    		.interval = 5s;
    		.timeout = 3s;
    		.window = 5;
    		.threshold = 3;
    	}
    }
    
    sub vcl_recv {
    
    	#BASIC VCL RULES SETTING
    	#Fix gzip compression
    	if (req.http.Accept-Encoding) {
    	if (req.url ~ "\.(gif|jpg|jpeg|bmp|png|ico|img|tga|wmf|gz|tgz|bz2|tbz|mp3|ogg)$") {
    		unset req.http.Accept-Encoding;
    		}
    	else if (req.http.Accept-Encoding ~ "gzip") {
    		set req.http.Accept-Encoding = "gzip";
    		}
    	else if (req.http.Accept-Encoding ~ "deflate") {
    		set req.http.Accept-Encoding = "deflate";
    		}
    	else	{
    		unset req.http.Accept-Encoding;
    		}
    	}
    	#set client balance identity
    	set client.identity = client.ip;
    
    	#set X-forward
    	set req.http.X-Forwarded-For = client.ip;
    
    	if (req.http.host == "minecraft.domain.com") {
    		set req.backend = MinecraftBACKEND;
    	} 
    
    	#respect client wish to refresh the page
    	if (req.http.Pragma ~ "no-cache")
    		{
    		return(pass);
    		}
    
    	#BASIC VCL RULES ACTIONS
    	#Disable post cache
    	if (req.request == "POST") {
    		return(pass);
    		}
    	#Enable static cache
    	if (req.request=="GET" && req.url ~ "\.(css|js|txt|zip|rtf|flv|swf|html|htm)$") {
    		return(pass);
    		}
    	if (req.request=="GET" && req.url ~ "\.(gif|jpg|jpeg|bmp|png|ico|img|tga|wmf|mp3|ogg)$") {
    		return(pass);
    		}
    	#Disable session cache
    	if (req.http.Cookie && req.http.Cookie ~ "(PHPSESSID|phpsessid)") {
    		return(pass);
    		}
    	if (req.http.Cookie && req.http.Cookie ~ "(JSESSION|jsession)") {
    		return(pass);
    		}
    	if (req.http.X-Requested-With == "XMLHttpRequest" || req.url ~ "nocache") {
    		return(pass);
    		}
    	#Be rfc2616 compliant
    	if (req.request ~ "^(GET|HEAD|PUT|POST|TRACE|OPTIONS|DELETE)$") {
    		return(lookup);
    		}
    	else	{
    		return(pipe);
    		}
    
    	return(lookup);
    }
    
    sub vcl_pipe {
    
    	# If we don't set the Connection: close header, any following
    	# requests from the client will also be piped through and
    	# left untouched by varnish. We don't want that.
    	set req.http.connection = "close";
    	# Note: no "pipe" action here - we'll fall back to the default
    	# pipe method so that when any changes are made there, we
    	# still inherit them.
    
    }
    
    sub vcl_fetch {
    
    	#Disable cache when backend is starting a session
    	if (beresp.http.Set-Cookie && beresp.http.Set-Cookie ~ "(PHPSESSID|phpsessid)") {
    		return(pass);
    		}
    	if (beresp.http.Set-Cookie && beresp.http.Set-Cookie ~ "(JSESSION|jsession)") {
    		return(pass);
    		}
    	if (beresp.http.X-Requested-With == "XMLHttpRequest" || req.url ~ "nocache") {
    		return(pass);
    		}
    
    	# Varnish respects the wishes of the backend application.
    	if (beresp.http.Pragma ~ "no-cache" || beresp.http.Cache-Control ~ "(no-cache|no-store|private)") {
    	return(pass);
    	}
        ## If the request to the backend returns a code other than 200, restart the loop
        ## If the number of restarts reaches the value of the parameter max_restarts,
        ## the request will be error'ed.  max_restarts defaults to 4.  This prevents
        ## an eternal loop in the event that, e.g., the object does not exist at all.
        if (beresp.status != 200 && beresp.status != 403 && beresp.status != 404 && 
        	beresp.status != 303 && beresp.status != 302 && beresp.status != 301 && beresp.status != 401 ) {
    		restart;
            }
    
    	return(deliver);
    }
    
    sub vcl_deliver {
    	##set resp.http.X-Served-By = server.hostname;
      	if (obj.hits > 0) {
        	set resp.http.X-Cache = "HIT";	
        	set resp.http.X-Cache-Hits = obj.hits;
      	} else {
        	set resp.http.X-Cache = "MISS";	
      	}
    	return(deliver);
    }
    
    

Locked