reestablish relayd

[mrhaamann]

Hi all,
I would like to have the relayd back for some purposes and stated to work on the existing code to port it to openssl1.1.1.
I already have a compilable version running on my machine (freebsd 12),
need to test some things to make sure ssl is working.
I also added support for socket option dont_fragment, which was asked for in the original github section.
Would someone be interested in reintroducing it as a separate package with the "old" integration ?
I am not very familiar with packaging, but I would work on the code to make it stable.
The php integration is already present in the pfsense code and should be extracted to be in a separate package.
I did not take a look how deep the relayd was integrated in the core classe, but most of the stuff should be present
and needs to be re-packaged only.

Marcus

[jimp]

I'm not sure if we'd be interested in bringing it back in. There were a few edge cases where it was useful, but mostly it was a source of problems for people who have been much, much happier using HAProxy for that role.

If it were packaged up and isolated to where it didn't require any extra libraries, we could maybe consider it.

Most of the built-in relayd code was taken out in this commit: https://github.com/pfsense/pfsense/commit/586c623a943f59486a461c1af9873dd6cc11a3b3

That would be a good starting point for identifying the code needed for making it a package.

[xeba]

@jimp

I've modified the relayd implementation that currently requires to be statically linked to libressl in order to work.
It now compiles with openssl 1.1.1 and seems to work on basic cases.

I wanted to create a package to test it further, but it isn't in the ports tree, so how can I include its binaries into a pfsense package?

[xeba]

I've compiled relayd on '12.2-STABLE FreeBSD 12.2-STABLE r369445 GENERIC amd64 and copied to pfSense, however i get this error:

[2.5.0-RELEASE][admin@pfSense25.home.arpa]/root: /usr/local/sbin/relayd -df /var/etc/relayd.conf
startup
unused protocol: dnsproto
host 192.168.56.1, check icmp (0ms), state unknown -> up, availability 100.00%
table prova: 1 added, 0 deleted, 0 changed, 0 killed
fatal: cannot add rule: Operation not supported by device
hce exiting, pid 94294
lost child: pfe exited abnormally
relay exiting, pid 94345
relay exiting, pid 94785
relay exiting, pid 94763
ca exiting, pid 94813
ca exiting, pid 94437
parent terminating, pid 93818

rdr-anchor "relayd/*" all and anchor "relayd/*" all present in pf config.

Any idea on how to fix this?

[mrhaamann]

@xeba pls post the config you are working with. I did not test all features. This might give us a hint what type of action is not supported.

[xeba]

Seems to be pfSense related, since the same config works on FreeBSD.
I've compiled it on 12.2-STABLE, I don't know if it is possible to compile directly on pfSense, maybe that would solve the issue.

This is the config:

log updates 
timeout 1000 
table <provapool> {
	192.168.56.1
}
dns protocol "dnsproto" {
	tcp { nodelay, sack, socket buffer 1024, backlog 1000 }
}
redirect "prova" {
  listen on 192.168.0.38 port 7777
  forward to <provapool> port 7776 check icmp 
}

EDIT: The error seems to be generated at pfe_filter.c:

		if (ioctl(env->sc_pf->dev, DIOCADDRULE, &rio) == -1)
			fatal("cannot add rule");

[xeba]

I'll try to compile it using this guide:
https://github.com/Augustin-FL/building-pfsense-iso-from-source

but I'ld like to avoid to compile the entire system.
Has any of you tried to compile a single port?

[Netznico]

This post is deleted!

[xeba]

@xeba said in reestablish relayd:

I'll try to compile it using this guide:
https://github.com/Augustin-FL/building-pfsense-iso-from-source

but I'ld like to avoid to compile the entire system.
Has any of you tried to compile a single port?

I confirm that the version compiled in this way works, i don't know when I'll get around to build a fully functioning package. Anyway, anyone willing to do some testing?

[jackdig]

@xeba I'd be happy to test it. We are using pfSense 2.4 and cannot upgrde to 2.5 because of missing relayd. HAproxy does not cover all relayd use cases.

[xeba]

@jackdig
Ok, I should be able to work again on this a month from now.
I'll keep you updated

[xeba]

Sorry for the delay, theese are are the two experimental packages:

relayd-openssl-5.5.20140810_8.zip
pfSense-pkg-relayd-0.1_3.zip

At the moment you must redefine the config including the default monitor, since the format of the xml has changed.

Needless to say this not suitable for production, especially since some changes for openssl were not trivial and may contain bugs.

However this is a starting point, happy to have your feedback

[Mike115]

Any new progress in this? Having a working published relayd package is still of interest.

[xeba]

@mike115

I'll probably publish something on github in a while, with both the compiled package and the instructions on how to build it.

However I cannot force people to test it, so unless I have some feedback I doubt it will ever come back as an official pacakge.

[jackdig]

@xeba
I appreciate your work on this, but I haven't been able to test the package yet.
We wrote a cron job to mimic the relayd functionality and are currently happy with that solution.

[xeba]

@jackdig
Yes that is "my" problem as well, we use HAproxy for webserver and other type of solutions for pure tcp connections, so I cannot really do the testing myself.

[Mike115]

@xeba I'll test it! I have some lab environments. I've also been looking for a reason to break into some BSD dev. Do you have some samples of what a proper HAProxy config should look like for failover web clustering with port 80 redirecting to 443 on the backend with SSL?

[Mike115]

@jackdig do you have some samples of this HA cron job?

[jackdig]

@mike115 said in reestablish relayd:

@jackdig do you have some samples of this HA cron job?

I do, but this is not for the faint of heart :-)

Some background:
We are using a cron script for our production environment.
We have two pfSense firewalls in a HA setup (using CARP IPs) and we have two internal web servers serving different sites (both servers are active). When either internal web server fail, we want to redirect all traffic to the working web server.

Some problems:

Up until pfSense 2.5 (I think), there was no way of controlling fw rules (aliases) using scripts. Then /usr/local/bin/aliasmod came along and allowed us to add/remove IPs to aliases. This was not perfect, so we modified aliasmod to a new script (aliastool) that allow us to read or change the IP of an alias.

We want the script to work when our primary pfSense fw (CARP master) is down, so we need to run the script on all (master and slave) pfSense servers. We added checks in our script so that nothing will be done if the current server is not CARP master.

Cron will only allow us to run a script every minute, but we want to monitor our web servers more often (every 5 sec). We wrote the cron script so that it will run for just under 1 minute and then exit. It will not start a second instance if it is already running, but it will kill an old running instance after some time. All this code is needed to ensure that the monitoring script will never (fully) fail.

This script is written entirely for our own need and use case. The code is not perfect and the functionality might not fit you. You are, however, free to use it or enhance it anyway you'd like.

I uploaded our script (and the php based aliasool) here:
https://github.com/jackfagner/pfsense-ha-using-cron

[Mike115]

@jackdig Thanks for posting and good work! Unfortunately my exposure to shell scripts is somewhat limited. I understand the format and functions well enough but my knowledge of grep, sed, and awk is severely underdeveloped (my professor would be disappointed )...something I'm working on.

Anyways, have you tried an interface trigger? It may be a little redundant. I like those in my C# programs because then I don't have to write conditional tests at the granular level, although checking the IP like you did is good too because the interface may not always be reported as down. I also imagine you could further simplify this by using the built in watchdog.

I still have a lot to learn. This is a little more adventurous than I feel like getting into this morning, but thanks again for posting. I'll save it for later exploration.