Varnish3 package quite broken (fixed July-27-2012)



  • I'm using pfsense 2.01 recently installed on a VMWare VM.  I pulled the Varnish 3 package with the package manager.

    I unfortunately need some Varnish3 features so using the 2.1 version is out of the question for me.

    "Out of the box" the Varnish3 package is totally broken.

    /usr/local/pkg/varnish.inc gives and error on startup.  I think I fixed it by adding "if(" to the beginning of line 158 so that it now reads

    
     #check rewrite options
                                    if($url['rewritehost']) {
    
    

    Now I'm getting XML parsing errors on startup.  Trying to figure out how to debug / diagnose them further.

    Any / all advice appreciated.  Looking more for a RTFM direction than anything



  • @blundar:

    I think I fixed it by adding "if(" to the beginning of line 158 so that it now reads

    Thanks for the feedback, I've applied the patch on the varnish3 code

    att,
    Marcello Coutinho



  • On bootup, I get the message:

    
    Fatal error: Cannot create references to/from string offsets nor overloaded objects in /etc/inc/xmlparse.inc on line 69
    
    

    After Varnish3 tries to start

    The searching I did seems to indicate there is a syntax error in the XML.  Nothing shows up in any of the pfsense menus related to varnish so this would seem to make sense.  Is there an explanation anywhere about debugging/troubleshooting the XML stuff?  I've already read through as much as I could on the devwiki.  The CoreUI and CoreUI2 stuff seemed to be about as close to pertinent as I could find.

    As I said before - I'm not looking for someone else to fix this as to just point the finger at a little more explanation of how the package loads, etc. so that I can try to isolate the issue and fix it.  Having a real hard time finding docs :(



  • @blundar:

    On bootup, I get the message:

    
    Fatal error: Cannot create references to/from string offsets nor overloaded objects in /etc/inc/xmlparse.inc on line 69
    
    

    After Varnish3 tries to start

    Did you configured the varnish options on package gui?



  • What gui?  That's what I'm trying to get working - the GUI is entirely absent.  I've rebooted the FW several times since installing the package, FYI.

    ![no services config.png](/public/imported_attachments/1/no services config.png)
    ![no services config.png_thumb](/public/imported_attachments/1/no services config.png_thumb)



  • Try to reinstall the package. I've pushed the fix yesterday, so it will install ok now.



  • Thankyou.  Should be able to get to it in less than an hour.  Was that broken "inc" file involved in processing the installation of the package?  It would be really nice to read some docs on how the pfsense package system works…



  • Reinstall fixed gui.  I'd still really like to read some more docs on the pfsense package system.  Thanks a lot for your help!



  • @blundar:

    I'd still really like to read some more docs on the pfsense package system.

    these links may help
    http://forum.pfsense.org/index.php?topic=20516.0
    http://doc.pfsense.org/index.php/Developing_Packages

    take a look on one small package like ipguard:
    https://github.com/bsdperimeter/pfsense-packages/tree/master/config/ipguard

    att,
    Marcello Coutinho



  • Thankyou!



  • Another potential patch…  The XML errors with Varnish appear to be with varnish_lb_directors.xml

    One of the most recent patches on github seems to be responsible.

    If I remove the following lines from the file, pfsense seems to parse it ok:

    
            <field><fielddescr>Rewrite Host</fielddescr>
                <fieldname>rewritehost</fieldname>
                <description>Hint image.mysite.com</description>
                <type>input</type>
                <size>40</size></field> 
    
            <field><fielddescr>Rewrite URL</fielddescr>
                <fieldname>rewriteurl</fieldname>
                <description>Hint /images</description>
                <type>input</type>
                <size>40</size></field> 
    
    

    I've looked through this code once, twice, three times and I can't find any issues with the syntax.  In the absence of a syntax error, I'm a little stumped as to why it's causing the XML error.

    The other patch doesn't seem like it causes any issues when it's present:

    
     		 <columnitem><fielddescr>Rewrite Host</fielddescr>
                <fieldname>rewritehost</fieldname></columnitem> 
            <columnitem><fielddescr>Rewrite url</fielddescr>
                <fieldname>rewriteurl</fieldname></columnitem> 
    
    


  • And another…

    My system (Amd64 PFSense 2.01) does not recognize the kern.ipc.nmbclusters sysctl.

    I had to remove this line pertaining to this from /usr/local/pkg/varnish.inc in order to not generate log chatter



  • And another…  The webgui is generating a broken /var/etc/default.vcl!

    
    #set X-forward
    	set req.http.X-Forwarded-For = client.ip;
    
    		else	{
    		set req.backend = WWW2BACKEND;
    		}
    
    	#respect client wish to refresh the page
    	if (req.http.Pragma ~ "no-cache")
    		{
    		return(pass);
    		}
    
    

    The offending section specifically is:

    
    else	{
    		set req.backend = WWW2BACKEND;
    		}
    
    

    else statement with no 'if' preceding.

    Looking over things, it looks like varnish.inc is responsible for generating this file.  I'm working on fixing it so it creates sane, meaningful output.  I'll post up when done.



  • the $urlmappings block is generating the offending code…

    You need to define which hosts each backend answers for.  Failure to define at least something will cause invalid VCL to be generated.



  • @blundar:

    I've looked through this code once, twice, three times and I can't find any issues with the syntax.  In the absence of a syntax error, I'm a little stumped as to why it's causing the XML error.

    try to change hint line to

    @blundar:

    The offending section specifically is:

    
    else	{
    		set req.backend = WWW2BACKEND;
    		}
    
    

    else statement with no 'if' preceding.

    IIRC, this else is the catch all(or default) backend to forward request.



  • @marcelloc:

    @blundar:

    I've looked through this code once, twice, three times and I can't find any issues with the syntax.  In the absence of a syntax error, I'm a little stumped as to why it's causing the XML error.

    try to change hint line to

    @blundar:

    The offending section specifically is:

    
    else	{
    		set req.backend = WWW2BACKEND;
    		}
    
    

    else statement with no 'if' preceding.

    IIRC, this else is the catch all(or default) backend to forward request.

    The catch all / default case is the broken one.  Varnish bombs because of the 'else' statement without a preceding if.  I'm going to fix this and I'll send you the patch, just haven't figured out a clean way of doing it yet.  The $urlmappings is expected to add conditions for mappings beforehand, and does if you define them.  I'm thinking that something as simple as

    
    if(FALSE) {}
    
    

    instead of the usual

    
    	if (req.http.host == "blundar.com") {
    		set req.backend = WWW2BACKEND;
    		set req.grace=4h;
    	} 
    	else if (req.http.host == "accelbydave.com") {
    		set req.backend = NASBACKEND;
    		set req.grace=4h;
    	} 
    	else if (req.http.host == "acceleratedbydave.com") {
    		set req.backend = NASBACKEND;
    		set req.grace=4h;
    	}
    
    

    as the default case (i.e. no specific mappings defined) so that the default case will fall into the else {}.
    The alternative would not be to generate the else {} statement and handle the catch-call condition in another manner, perhaps using something more similar to a more typical varnish setup:

    
    backend default {
        .host = "127.0.0.1";
        .port = "8080";
        .connect_timeout = 600s;
        .first_byte_timeout = 600s;
        .between_bytes_timeout = 600s;
        .max_connections = 250;
    }
    
    backend umpqua {
        .host = "10.0.0.40";
        .port = "80";
        .connect_timeout = 600s;
        .first_byte_timeout = 600s;
        .between_bytes_timeout = 600s;
        .max_connections = 250;
    }
    
    backend coos {
        .host = "10.0.0.33";
        .port = "80";
        .connect_timeout = 600s;
        .first_byte_timeout = 600s;
        .between_bytes_timeout = 600s;
        .max_connections = 250;
    }
    
    sub vcl_recv {
        if (req.http.host ~ "umpqua") {
            # Route all umpqua traffic to umpqua
            set req.backend = umpqua;
        } else if (req.http.host ~ "oregonk-12.net$") {
            # route oregonk-12.net traffic to coos
            set req.backend = coos;
        } else if (req.http.host ~ "k12partners.org$") {
            # route k12partners.org to coos
            set req.backend = coos;
        } else if (req.http.host ~ "oregoneducationdata.org$") {
            # route oregoneducationdata.org to coos
            set req.backend = coos;
        } else {
            # Use the default backend for all other requests
            set req.backend = default;
        }
    
    


  • Can you send me you current setup/sample to test?

    My varnish3 setups/tests are all working.

    The only change I've not applied on my servers was this latest patch

    https://github.com/bsdperimeter/pfsense-packages/commit/69da10ca7afe298684f2a4eb0f4a703651007d53



  • I just re-installed last night about 1am EST.

    I am running on ESXi 5.0.0 build 721882
    hardware = 2x Xeon L2510 quad core, 16Gb RAM, Perc RAID over SAS, separate FreeNAS exporting iSCSI

    VM as configured: VM v8, 2Gb RAM, 1xCPU with 4xCores, 4x E1000 NICs, LSI scsi, 2 disks.  4Gb for OS, 4GB for swap.  Swap disk independent (non-snapshot)

    Installed off pfsense amd64 iso, dated 6-6-2012.  It's PFSense 2.0.1:

    
    2.0.1-RELEASE (amd64)
    built on Mon Dec 12 18:16:13 EST 2011
    FreeBSD 8.1-RELEASE-p6
    

    Installed snort, snort-dashboard-widget, open-vm-tools, pfblocker, Varnish3 using the package manager build in to the web gui.

    WAN, LAN, DMZ, HOMENET interfaces configured.  Static WAN, DHCP on LAN.  Fairly standard IP policies

    Snort is active (but the issues with varnish precede me getting snort to behave correctly due to rule weirdnesses that I didn't catch)

    In addition to the patch I already sent you, I had to patch /usr/local/pkg/varnish.inc and /usr/local/pkg/varnish_lb_directorx.xml to get the GUI to behave properly and not put extra warnings in logs, etc.  I'm attaching these patches.

    The issue with not generating valid vcl can be replicated with a very simple setup - single backend, and leave the "Backend Mappings" section blank.  If you add any entries at all to the backend mappings, the vcl is ok.  Remove the mappings and the xml fails to generate a sane default case.

    I'll send you a PM too…



  • Forgot patches!!!

    The patch to varnish.inc removes a sysctl call that was creating log traffic and adds a few very simple comments so I could figure out which part of the function that generates vcl_recv() was responsible for generating broken code in the case of an empty backends mapping statements.

    the patch to varnish_lb_directors.xml was necessary to get the webgui to load the LB Directors page.  Had to remove a statement.  Didn't try the fix you suggested yet.  I'll try it soon and update thread after

    db-varnish.inc.patch.txt
    db-varnish_lb_directors.xml.patch.txt



  • @blundar:

    Forgot patches!!!

    I'll merge a multi daemon varnish soon, I've applied the cdata fix on my updated files and varnish started fine.

    Thanks for your tests and feedback


Log in to reply