SSL_write() failed (SSL:) issues on Secondary node

[wast3gat3]

Hi pfSense Community,

I have 4 pfSense firewalls on an ESXi 6.0.0 host, two are configured as rear firewalls in HA running DHCP on them and two are configured as forward facing firewalls also in HA.
The rear firewalls are connected to 3 vSwitches (Internal, Secured and DMZ) and the forward firewalls are connected to 2 vSwitches (DMZ and External).
The forward firewalls have no issues in HA and CARP addresses are working fine.
The rear secondary firewall is very slow to respond to https requests from my browser (Firefox) and in the system logs I am seeing the following.

nginx: 2016/08/03 14:42:34 [crit] 22820#100111: *841 SSL_write() failed (SSL:) (1: Operation not permitted) while sending to client, client: {IP address of client}, server: , request: "GET /getstats.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "{IP address of firewall}", referrer: "https://{IP address of firewall}/"

This happens with almost every request to the secondary node and the logs are full of the nginx errors for SSL_write.

I have rebuilt the firewall from the ground up twice, once restoring the config and the second time adding all setting back manually and when it joins the HA cluster the errors start.

Any help greatly appreciated as I have compared the forward firewalls config to the rear and they are near identical except for IP addressing in their setup and they have zero errors.

Anyone seen this?

Warm regards,
Wastey

[lshiry]

I am also seeing this problem on my secondary pfsense.  The nginx errors:

nginx: 2016/09/16 10:13:52 [crit] 37510#100191: *14209 SSL_write() failed (SSL:) (1: Operation not permitted) while sending to client, client: x.x.x.x, server: , request: "GET /ifstats.php?if=lagg0_vlanx HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: “xxxxxxxxxxx”, referrer: "https://xxxxxxxxxxxxx/graph.php?ifnum=opt4&ifname=DB&timeint=1&initdelay=10"

Also seeing similar error with ssh, and ssh sessions hang up and get disconnected:

fatal: Fssh_packet_write_poll: Connection from x.x.x.x port 58926: Operation not permitted

Running v2.3.2.

[Righter]

i have exactly the same issue :-(

Running 2.3.2p1 on ESXi 6.0 with OpenVMTools

As soon as I disable HA-Sync on the secondary it's workin well again

[ralph.ratenan]

Hello community,
i have exactly the same issues with two servers, in version 2.3.2-RELEASE-p1 (amd64) .
:-[ :-[

Any help, advise or solution would be appreciated  :-\

Thank youuuuu

[diegoqueiroz]

@lshiry:

I am also seeing this problem on my secondary pfsense.  The nginx errors:

nginx: 2016/09/16 10:13:52 [crit] 37510#100191: *14209 SSL_write() failed (SSL:) (1: Operation not permitted) while sending to client, client: x.x.x.x, server: , request: "GET /ifstats.php?if=lagg0_vlanx HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: “xxxxxxxxxxx”, referrer: "https://xxxxxxxxxxxxx/graph.php?ifnum=opt4&ifname=DB&timeint=1&initdelay=10"

Also seeing similar error with ssh, and ssh sessions hang up and get disconnected:

fatal: Fssh_packet_write_poll: Connection from x.x.x.x port 58926: Operation not permitted

Running v2.3.2.

I am also having the very simular issue, and all connections seem unstable.

When I start a SSH session, it starts smoothly, but the connection drops after some seconds.
The only difference is that I see "Permission denied" instead of "Operation not permitted".

My errors:

fatal: Fssh_packet_write_poll: Connection from 192.168.9.13 port 10743: Permission denied

nginx: 2017/04/08 13:04:21 [alert] 57749#100122: *14319 writev() failed (13: Permission denied) while sending to client, client: 192.168.0.1, server: , request: "GET /getstats.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "192.168.0.1", referrer: "http://192.168.0.1/"

I am running 2.3.3-RELEASE-p1.

[diegoqueiroz]

I solved my problem. I'll describe the solution here, maybe this can help someone in the future.

First, my setup.
In my pfSense box, I do have a MultiWan setup with two independent links. Since my ISP gateways do not answer ping requests, I have set Google's DNS IPs 8.8.8.8 and 8.8.4.4 to monitor my links.

And the problem:
Apparently, all my ISP links were intermittent for international hosts (Google's DNS is an internation host for me, since I am in Brazil).
Due to this, all my links became unstable. The %loss in Status > Gateways were very high (like 30-40% of loss on both links). This was clearly affecting several pfSense services, like SSH and my users' internet access.

There were also some lines stating this issue in the logs:

/rc.start_packages: Gateways status could not be determined, considering all as up/active. (Group: MultiWAN)
...
/rc.filter_configure_sync: Gateways status could not be determined, considering all as up/active. (Group: MultiWAN)
...
/rc.openvpn: Gateways status could not be determined, considering all as up/active. (Group: MultiWAN)

Since there's nothing I can do to solve the intermittency of my ISPs link, I decided to change the monitor IP of my Gateways (in System > Routing > Gateways) to an IP that wasn't intermittent (just to cite, I changed them to www.uol.com.br IPs, 200.221.2.45 and 200.147.67.142).

This immediately solved the problem.