Varnish3 package quite broken (fixed July-27-2012)

blundar

On bootup, I get the message:


Fatal error: Cannot create references to/from string offsets nor overloaded objects in /etc/inc/xmlparse.inc on line 69

After Varnish3 tries to start

The searching I did seems to indicate there is a syntax error in the XML. Nothing shows up in any of the pfsense menus related to varnish so this would seem to make sense. Is there an explanation anywhere about debugging/troubleshooting the XML stuff? I've already read through as much as I could on the devwiki. The CoreUI and CoreUI2 stuff seemed to be about as close to pertinent as I could find.

As I said before - I'm not looking for someone else to fix this as to just point the finger at a little more explanation of how the package loads, etc. so that I can try to isolate the issue and fix it. Having a real hard time finding docs :(

marcelloc

@blundar:

On bootup, I get the message:
Fatal error: Cannot create references to/from string offsets nor overloaded objects in /etc/inc/xmlparse.inc on line 69
After Varnish3 tries to start

Did you configured the varnish options on package gui?

blundar

What gui? That's what I'm trying to get working - the GUI is entirely absent. I've rebooted the FW several times since installing the package, FYI.

![no services config.png](/public/imported_attachments/1/no services config.png)
![no services config.png_thumb](/public/imported_attachments/1/no services config.png_thumb)

marcelloc

Try to reinstall the package. I've pushed the fix yesterday, so it will install ok now.

blundar

Thankyou. Should be able to get to it in less than an hour. Was that broken "inc" file involved in processing the installation of the package? It would be really nice to read some docs on how the pfsense package system works…

blundar

Reinstall fixed gui. I'd still really like to read some more docs on the pfsense package system. Thanks a lot for your help!

marcelloc

@blundar:

I'd still really like to read some more docs on the pfsense package system.

these links may help
http://forum.pfsense.org/index.php?topic=20516.0
http://doc.pfsense.org/index.php/Developing_Packages

take a look on one small package like ipguard:
https://github.com/bsdperimeter/pfsense-packages/tree/master/config/ipguard

att,
Marcello Coutinho

blundar

Thankyou!

blundar

Another potential patch… The XML errors with Varnish appear to be with varnish_lb_directors.xml

One of the most recent patches on github seems to be responsible.

If I remove the following lines from the file, pfsense seems to parse it ok:


        <field><fielddescr>Rewrite Host</fielddescr>
            <fieldname>rewritehost</fieldname>
            <description>Hint image.mysite.com</description>
            <type>input</type>
            <size>40</size></field> 

        <field><fielddescr>Rewrite URL</fielddescr>
            <fieldname>rewriteurl</fieldname>
            <description>Hint /images</description>
            <type>input</type>
            <size>40</size></field>

I've looked through this code once, twice, three times and I can't find any issues with the syntax. In the absence of a syntax error, I'm a little stumped as to why it's causing the XML error.

The other patch doesn't seem like it causes any issues when it's present:


 		 <columnitem><fielddescr>Rewrite Host</fielddescr>
            <fieldname>rewritehost</fieldname></columnitem> 
        <columnitem><fielddescr>Rewrite url</fielddescr>
            <fieldname>rewriteurl</fieldname></columnitem>

blundar

And another…

My system (Amd64 PFSense 2.01) does not recognize the kern.ipc.nmbclusters sysctl.

I had to remove this line pertaining to this from /usr/local/pkg/varnish.inc in order to not generate log chatter

blundar

And another… The webgui is generating a broken /var/etc/default.vcl!


#set X-forward
	set req.http.X-Forwarded-For = client.ip;

		else	{
		set req.backend = WWW2BACKEND;
		}

	#respect client wish to refresh the page
	if (req.http.Pragma ~ "no-cache")
		{
		return(pass);
		}

The offending section specifically is:


else	{
		set req.backend = WWW2BACKEND;
		}

else statement with no 'if' preceding.

Looking over things, it looks like varnish.inc is responsible for generating this file. I'm working on fixing it so it creates sane, meaningful output. I'll post up when done.

blundar

the $urlmappings block is generating the offending code…

You need to define which hosts each backend answers for. Failure to define at least something will cause invalid VCL to be generated.

marcelloc

@blundar:

I've looked through this code once, twice, three times and I can't find any issues with the syntax. In the absence of a syntax error, I'm a little stumped as to why it's causing the XML error.

try to change hint line to

@blundar:

The offending section specifically is:
else	{
		set req.backend = WWW2BACKEND;
		}
else statement with no 'if' preceding.

IIRC, this else is the catch all(or default) backend to forward request.

blundar

@marcelloc:

@blundar:

I've looked through this code once, twice, three times and I can't find any issues with the syntax. In the absence of a syntax error, I'm a little stumped as to why it's causing the XML error.

try to change hint line to

@blundar:
The offending section specifically is:
else	{
		set req.backend = WWW2BACKEND;
		}
else statement with no 'if' preceding.
IIRC, this else is the catch all(or default) backend to forward request.

The catch all / default case is the broken one. Varnish bombs because of the 'else' statement without a preceding if. I'm going to fix this and I'll send you the patch, just haven't figured out a clean way of doing it yet. The $urlmappings is expected to add conditions for mappings beforehand, and does if you define them. I'm thinking that something as simple as


if(FALSE) {}

instead of the usual


	if (req.http.host == "blundar.com") {
		set req.backend = WWW2BACKEND;
		set req.grace=4h;
	} 
	else if (req.http.host == "accelbydave.com") {
		set req.backend = NASBACKEND;
		set req.grace=4h;
	} 
	else if (req.http.host == "acceleratedbydave.com") {
		set req.backend = NASBACKEND;
		set req.grace=4h;
	}

as the default case (i.e. no specific mappings defined) so that the default case will fall into the else {}.
The alternative would not be to generate the else {} statement and handle the catch-call condition in another manner, perhaps using something more similar to a more typical varnish setup:


backend default {
    .host = "127.0.0.1";
    .port = "8080";
    .connect_timeout = 600s;
    .first_byte_timeout = 600s;
    .between_bytes_timeout = 600s;
    .max_connections = 250;
}

backend umpqua {
    .host = "10.0.0.40";
    .port = "80";
    .connect_timeout = 600s;
    .first_byte_timeout = 600s;
    .between_bytes_timeout = 600s;
    .max_connections = 250;
}

backend coos {
    .host = "10.0.0.33";
    .port = "80";
    .connect_timeout = 600s;
    .first_byte_timeout = 600s;
    .between_bytes_timeout = 600s;
    .max_connections = 250;
}

sub vcl_recv {
    if (req.http.host ~ "umpqua") {
        # Route all umpqua traffic to umpqua
        set req.backend = umpqua;
    } else if (req.http.host ~ "oregonk-12.net$") {
        # route oregonk-12.net traffic to coos
        set req.backend = coos;
    } else if (req.http.host ~ "k12partners.org$") {
        # route k12partners.org to coos
        set req.backend = coos;
    } else if (req.http.host ~ "oregoneducationdata.org$") {
        # route oregoneducationdata.org to coos
        set req.backend = coos;
    } else {
        # Use the default backend for all other requests
        set req.backend = default;
    }

marcelloc

Can you send me you current setup/sample to test?

My varnish3 setups/tests are all working.

The only change I've not applied on my servers was this latest patch

https://github.com/bsdperimeter/pfsense-packages/commit/69da10ca7afe298684f2a4eb0f4a703651007d53

blundar

I just re-installed last night about 1am EST.

I am running on ESXi 5.0.0 build 721882
hardware = 2x Xeon L2510 quad core, 16Gb RAM, Perc RAID over SAS, separate FreeNAS exporting iSCSI

VM as configured: VM v8, 2Gb RAM, 1xCPU with 4xCores, 4x E1000 NICs, LSI scsi, 2 disks. 4Gb for OS, 4GB for swap. Swap disk independent (non-snapshot)

Installed off pfsense amd64 iso, dated 6-6-2012. It's PFSense 2.0.1:


2.0.1-RELEASE (amd64)
built on Mon Dec 12 18:16:13 EST 2011
FreeBSD 8.1-RELEASE-p6

Installed snort, snort-dashboard-widget, open-vm-tools, pfblocker, Varnish3 using the package manager build in to the web gui.

WAN, LAN, DMZ, HOMENET interfaces configured. Static WAN, DHCP on LAN. Fairly standard IP policies

Snort is active (but the issues with varnish precede me getting snort to behave correctly due to rule weirdnesses that I didn't catch)

In addition to the patch I already sent you, I had to patch /usr/local/pkg/varnish.inc and /usr/local/pkg/varnish_lb_directorx.xml to get the GUI to behave properly and not put extra warnings in logs, etc. I'm attaching these patches.

The issue with not generating valid vcl can be replicated with a very simple setup - single backend, and leave the "Backend Mappings" section blank. If you add any entries at all to the backend mappings, the vcl is ok. Remove the mappings and the xml fails to generate a sane default case.

I'll send you a PM too…

blundar

Forgot patches!!!

The patch to varnish.inc removes a sysctl call that was creating log traffic and adds a few very simple comments so I could figure out which part of the function that generates vcl_recv() was responsible for generating broken code in the case of an empty backends mapping statements.

the patch to varnish_lb_directors.xml was necessary to get the webgui to load the LB Directors page. Had to remove a statement. Didn't try the fix you suggested yet. I'll try it soon and update thread after

db-varnish.inc.patch.txt
db-varnish_lb_directors.xml.patch.txt

marcelloc

@blundar:

Forgot patches!!!

I'll merge a multi daemon varnish soon, I've applied the cdata fix on my updated files and varnish started fine.

Thanks for your tests and feedback