Varnish3 package quite broken (fixed July-27-2012)

blundar

Another potential patch… The XML errors with Varnish appear to be with varnish_lb_directors.xml

One of the most recent patches on github seems to be responsible.

If I remove the following lines from the file, pfsense seems to parse it ok:


        <field><fielddescr>Rewrite Host</fielddescr>
            <fieldname>rewritehost</fieldname>
            <description>Hint image.mysite.com</description>
            <type>input</type>
            <size>40</size></field> 

        <field><fielddescr>Rewrite URL</fielddescr>
            <fieldname>rewriteurl</fieldname>
            <description>Hint /images</description>
            <type>input</type>
            <size>40</size></field>

I've looked through this code once, twice, three times and I can't find any issues with the syntax. In the absence of a syntax error, I'm a little stumped as to why it's causing the XML error.

The other patch doesn't seem like it causes any issues when it's present:


 		 <columnitem><fielddescr>Rewrite Host</fielddescr>
            <fieldname>rewritehost</fieldname></columnitem> 
        <columnitem><fielddescr>Rewrite url</fielddescr>
            <fieldname>rewriteurl</fieldname></columnitem>

blundar

And another…

My system (Amd64 PFSense 2.01) does not recognize the kern.ipc.nmbclusters sysctl.

I had to remove this line pertaining to this from /usr/local/pkg/varnish.inc in order to not generate log chatter

blundar

And another… The webgui is generating a broken /var/etc/default.vcl!


#set X-forward
	set req.http.X-Forwarded-For = client.ip;

		else	{
		set req.backend = WWW2BACKEND;
		}

	#respect client wish to refresh the page
	if (req.http.Pragma ~ "no-cache")
		{
		return(pass);
		}

The offending section specifically is:


else	{
		set req.backend = WWW2BACKEND;
		}

else statement with no 'if' preceding.

Looking over things, it looks like varnish.inc is responsible for generating this file. I'm working on fixing it so it creates sane, meaningful output. I'll post up when done.

blundar

the $urlmappings block is generating the offending code…

You need to define which hosts each backend answers for. Failure to define at least something will cause invalid VCL to be generated.

marcelloc

@blundar:

I've looked through this code once, twice, three times and I can't find any issues with the syntax. In the absence of a syntax error, I'm a little stumped as to why it's causing the XML error.

try to change hint line to

@blundar:

The offending section specifically is:
else	{
		set req.backend = WWW2BACKEND;
		}
else statement with no 'if' preceding.

IIRC, this else is the catch all(or default) backend to forward request.

blundar

@marcelloc:

@blundar:

I've looked through this code once, twice, three times and I can't find any issues with the syntax. In the absence of a syntax error, I'm a little stumped as to why it's causing the XML error.

try to change hint line to

@blundar:
The offending section specifically is:
else	{
		set req.backend = WWW2BACKEND;
		}
else statement with no 'if' preceding.
IIRC, this else is the catch all(or default) backend to forward request.

The catch all / default case is the broken one. Varnish bombs because of the 'else' statement without a preceding if. I'm going to fix this and I'll send you the patch, just haven't figured out a clean way of doing it yet. The $urlmappings is expected to add conditions for mappings beforehand, and does if you define them. I'm thinking that something as simple as


if(FALSE) {}

instead of the usual


	if (req.http.host == "blundar.com") {
		set req.backend = WWW2BACKEND;
		set req.grace=4h;
	} 
	else if (req.http.host == "accelbydave.com") {
		set req.backend = NASBACKEND;
		set req.grace=4h;
	} 
	else if (req.http.host == "acceleratedbydave.com") {
		set req.backend = NASBACKEND;
		set req.grace=4h;
	}

as the default case (i.e. no specific mappings defined) so that the default case will fall into the else {}.
The alternative would not be to generate the else {} statement and handle the catch-call condition in another manner, perhaps using something more similar to a more typical varnish setup:


backend default {
    .host = "127.0.0.1";
    .port = "8080";
    .connect_timeout = 600s;
    .first_byte_timeout = 600s;
    .between_bytes_timeout = 600s;
    .max_connections = 250;
}

backend umpqua {
    .host = "10.0.0.40";
    .port = "80";
    .connect_timeout = 600s;
    .first_byte_timeout = 600s;
    .between_bytes_timeout = 600s;
    .max_connections = 250;
}

backend coos {
    .host = "10.0.0.33";
    .port = "80";
    .connect_timeout = 600s;
    .first_byte_timeout = 600s;
    .between_bytes_timeout = 600s;
    .max_connections = 250;
}

sub vcl_recv {
    if (req.http.host ~ "umpqua") {
        # Route all umpqua traffic to umpqua
        set req.backend = umpqua;
    } else if (req.http.host ~ "oregonk-12.net$") {
        # route oregonk-12.net traffic to coos
        set req.backend = coos;
    } else if (req.http.host ~ "k12partners.org$") {
        # route k12partners.org to coos
        set req.backend = coos;
    } else if (req.http.host ~ "oregoneducationdata.org$") {
        # route oregoneducationdata.org to coos
        set req.backend = coos;
    } else {
        # Use the default backend for all other requests
        set req.backend = default;
    }

marcelloc

Can you send me you current setup/sample to test?

My varnish3 setups/tests are all working.

The only change I've not applied on my servers was this latest patch

https://github.com/bsdperimeter/pfsense-packages/commit/69da10ca7afe298684f2a4eb0f4a703651007d53

blundar

I just re-installed last night about 1am EST.

I am running on ESXi 5.0.0 build 721882
hardware = 2x Xeon L2510 quad core, 16Gb RAM, Perc RAID over SAS, separate FreeNAS exporting iSCSI

VM as configured: VM v8, 2Gb RAM, 1xCPU with 4xCores, 4x E1000 NICs, LSI scsi, 2 disks. 4Gb for OS, 4GB for swap. Swap disk independent (non-snapshot)

Installed off pfsense amd64 iso, dated 6-6-2012. It's PFSense 2.0.1:


2.0.1-RELEASE (amd64)
built on Mon Dec 12 18:16:13 EST 2011
FreeBSD 8.1-RELEASE-p6

Installed snort, snort-dashboard-widget, open-vm-tools, pfblocker, Varnish3 using the package manager build in to the web gui.

WAN, LAN, DMZ, HOMENET interfaces configured. Static WAN, DHCP on LAN. Fairly standard IP policies

Snort is active (but the issues with varnish precede me getting snort to behave correctly due to rule weirdnesses that I didn't catch)

In addition to the patch I already sent you, I had to patch /usr/local/pkg/varnish.inc and /usr/local/pkg/varnish_lb_directorx.xml to get the GUI to behave properly and not put extra warnings in logs, etc. I'm attaching these patches.

The issue with not generating valid vcl can be replicated with a very simple setup - single backend, and leave the "Backend Mappings" section blank. If you add any entries at all to the backend mappings, the vcl is ok. Remove the mappings and the xml fails to generate a sane default case.

I'll send you a PM too…

blundar

Forgot patches!!!

The patch to varnish.inc removes a sysctl call that was creating log traffic and adds a few very simple comments so I could figure out which part of the function that generates vcl_recv() was responsible for generating broken code in the case of an empty backends mapping statements.

the patch to varnish_lb_directors.xml was necessary to get the webgui to load the LB Directors page. Had to remove a statement. Didn't try the fix you suggested yet. I'll try it soon and update thread after

db-varnish.inc.patch.txt
db-varnish_lb_directors.xml.patch.txt

marcelloc

@blundar:

Forgot patches!!!

I'll merge a multi daemon varnish soon, I've applied the cdata fix on my updated files and varnish started fine.

Thanks for your tests and feedback