reestablish relayd

xeba

I've compiled relayd on '12.2-STABLE FreeBSD 12.2-STABLE r369445 GENERIC amd64 and copied to pfSense, however i get this error:

[2.5.0-RELEASE][admin@pfSense25.home.arpa]/root: /usr/local/sbin/relayd -df /var/etc/relayd.conf
startup
unused protocol: dnsproto
host 192.168.56.1, check icmp (0ms), state unknown -> up, availability 100.00%
table prova: 1 added, 0 deleted, 0 changed, 0 killed
fatal: cannot add rule: Operation not supported by device
hce exiting, pid 94294
lost child: pfe exited abnormally
relay exiting, pid 94345
relay exiting, pid 94785
relay exiting, pid 94763
ca exiting, pid 94813
ca exiting, pid 94437
parent terminating, pid 93818

rdr-anchor "relayd/*" all and anchor "relayd/*" all present in pf config.

Any idea on how to fix this?

mrhaamann

@xeba pls post the config you are working with. I did not test all features. This might give us a hint what type of action is not supported.

xeba

Seems to be pfSense related, since the same config works on FreeBSD.
I've compiled it on 12.2-STABLE, I don't know if it is possible to compile directly on pfSense, maybe that would solve the issue.

This is the config:

log updates 
timeout 1000 
table <provapool> {
	192.168.56.1
}
dns protocol "dnsproto" {
	tcp { nodelay, sack, socket buffer 1024, backlog 1000 }
}
redirect "prova" {
  listen on 192.168.0.38 port 7777
  forward to <provapool> port 7776 check icmp 
}

EDIT: The error seems to be generated at pfe_filter.c:

		if (ioctl(env->sc_pf->dev, DIOCADDRULE, &rio) == -1)
			fatal("cannot add rule");

xeba

I'll try to compile it using this guide:
https://github.com/Augustin-FL/building-pfsense-iso-from-source

but I'ld like to avoid to compile the entire system.
Has any of you tried to compile a single port?

Netznico

This post is deleted!

xeba

@xeba said in reestablish relayd:

I'll try to compile it using this guide:
https://github.com/Augustin-FL/building-pfsense-iso-from-source

but I'ld like to avoid to compile the entire system.
Has any of you tried to compile a single port?

I confirm that the version compiled in this way works, i don't know when I'll get around to build a fully functioning package. Anyway, anyone willing to do some testing?

jackdig

@xeba I'd be happy to test it. We are using pfSense 2.4 and cannot upgrde to 2.5 because of missing relayd. HAproxy does not cover all relayd use cases.

xeba

@jackdig
Ok, I should be able to work again on this a month from now.
I'll keep you updated

xeba

Sorry for the delay, theese are are the two experimental packages:

relayd-openssl-5.5.20140810_8.zip
pfSense-pkg-relayd-0.1_3.zip

At the moment you must redefine the config including the default monitor, since the format of the xml has changed.

Needless to say this not suitable for production, especially since some changes for openssl were not trivial and may contain bugs.

However this is a starting point, happy to have your feedback

Mike115

Any new progress in this? Having a working published relayd package is still of interest.

xeba

@mike115

I'll probably publish something on github in a while, with both the compiled package and the instructions on how to build it.

However I cannot force people to test it, so unless I have some feedback I doubt it will ever come back as an official pacakge.

jackdig

@xeba
I appreciate your work on this, but I haven't been able to test the package yet.
We wrote a cron job to mimic the relayd functionality and are currently happy with that solution.

xeba

@jackdig
Yes that is "my" problem as well, we use HAproxy for webserver and other type of solutions for pure tcp connections, so I cannot really do the testing myself.

Mike115

@xeba I'll test it! I have some lab environments. I've also been looking for a reason to break into some BSD dev. Do you have some samples of what a proper HAProxy config should look like for failover web clustering with port 80 redirecting to 443 on the backend with SSL?

Mike115

@jackdig do you have some samples of this HA cron job?

jackdig

@mike115 said in reestablish relayd:

@jackdig do you have some samples of this HA cron job?

I do, but this is not for the faint of heart :-)

Some background:
We are using a cron script for our production environment.
We have two pfSense firewalls in a HA setup (using CARP IPs) and we have two internal web servers serving different sites (both servers are active). When either internal web server fail, we want to redirect all traffic to the working web server.

Some problems:

Up until pfSense 2.5 (I think), there was no way of controlling fw rules (aliases) using scripts. Then /usr/local/bin/aliasmod came along and allowed us to add/remove IPs to aliases. This was not perfect, so we modified aliasmod to a new script (aliastool) that allow us to read or change the IP of an alias.

We want the script to work when our primary pfSense fw (CARP master) is down, so we need to run the script on all (master and slave) pfSense servers. We added checks in our script so that nothing will be done if the current server is not CARP master.

Cron will only allow us to run a script every minute, but we want to monitor our web servers more often (every 5 sec). We wrote the cron script so that it will run for just under 1 minute and then exit. It will not start a second instance if it is already running, but it will kill an old running instance after some time. All this code is needed to ensure that the monitoring script will never (fully) fail.

This script is written entirely for our own need and use case. The code is not perfect and the functionality might not fit you. You are, however, free to use it or enhance it anyway you'd like.

I uploaded our script (and the php based aliasool) here:
https://github.com/jackfagner/pfsense-ha-using-cron

Mike115

@jackdig Thanks for posting and good work! Unfortunately my exposure to shell scripts is somewhat limited. I understand the format and functions well enough but my knowledge of grep, sed, and awk is severely underdeveloped (my professor would be disappointed )...something I'm working on.

Anyways, have you tried an interface trigger? It may be a little redundant. I like those in my C# programs because then I don't have to write conditional tests at the granular level, although checking the IP like you did is good too because the interface may not always be reported as down. I also imagine you could further simplify this by using the built in watchdog.

I still have a lot to learn. This is a little more adventurous than I feel like getting into this morning, but thanks again for posting. I'll save it for later exploration.

jackdig

@mike115 said in reestablish relayd:

Anyways, have you tried an interface trigger? It may be a little redundant. I like those in my C# programs because then I don't have to write conditional tests at the granular level, although checking the IP like you did is good too because the interface may not always be reported as down. I also imagine you could further simplify this by using the built in watchdog.

I'm not sure if we are talking about the same thing here?
Interface triggers could possible be used on the FW to determine if/when the FW interface changes (for tracking the CARP IP, for example), but it can not be used to determine if a web server on the internal network is working properly. For that functionality we need to use ping och http requests.
The reason for not using interface triggers for CARP monitoring is mainly simplicity. I'm not sure if interface triggers will survive a pfSense version upgrade.

ClaraGordon

This post is deleted!