Thanks for the clarification on a few things.
It's helped me get my mind round everything better and giving me a clearer idea of what direction I need to go to get everything as I want.

OK. I found it. Under System/Advanced/Admin Access.

[I am unable to delete my post. Sorry for the noise.]

Send the logs to a syslog server. Once they're off the FW you're only limited by disk space on the syslog server.

https://www.netgate.com/resources/videos/bandwidth-monitoring-on-pfsense.html

-Rico

@Cool_Corona

for me this cannot be a case of debate ☺
these are unreasonable things at SOHO

surely there is also a nuclear power plant in the garage to serve this muscular firewall + router unit

you can’t hunt sparrows with a cannon, but it’s your decision
I look at these things with a professional eye and you only experiment with pfSense

it has been my job for a long time and I think you're dealing with it as a hobby
this is not a problem anyway, but like I said - we are different

Bug ?
Setup !

pfSense handles ICMP as per user settings.
If not, this forum would be swamped by angry user posts ^^

@johnpoz Sorry mod, you are right and I have edited my post. I'm not using it forever, I have just installed it to test it on Unraid but I will use pfsense following spaceinvader tutorial for Unraid. Thanks anyway for your help.

Generally speaking in a site-to-site scenario the OpenVPN network (tunnel network) doesn't really matter to the clients on both sites, it's transparent for them. It's used by OpenVPN internally and routes the traffic to your real networks on both sites.
There is a LOT really good official documentation around for VPNs:
https://www.netgate.com/resources/videos/site-to-site-vpns-on-pfsense.html
https://www.netgate.com/resources/videos/advanced-openvpn-on-pfsense-24.html
https://docs.netgate.com/pfsense/en/latest/book/openvpn/site-to-site-example-configuration-shared-key.html
https://docs.netgate.com/pfsense/en/latest/book/openvpn/site-to-site-example-configuration-ssl-tls.html

-Rico

Additional info, system logs show several:

kernel vm_thread_new: kstack allocation failed

And several

kernel sonewconn: pcb 0xc7274790: Listen queue overflow: 193 already in queue awaiting acceptance (1 occurrences) nginx 2020/06/12 12:39:47 [error] 937#100185: *5059 connect() to unix:/var/run/php-fpm.socket failed (61: Connection refused) while connecting to upstream, client: xx.xx.xx.xx, server: , request: "GET / HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket:", host: "xx.xx.xx.xx:xxxx"

@CodeNinja

in this case, DMZ + WAF will be your good friend
something like this that I can suggest:

• OS: Debian 10.x (Buster) 64bit
• Apache Worker, factory package
• Mod Security apache module with OWASP rules, factory package
• PHP-FPM 7.3 or rather 7.4 if it goes with everything but definitely 1 version
• PHP can only write where we allow it, ie it stays on the www-data user
• firewall inbound to CF IPs is limited to http and https, just as SSH access is also severely limited (http can be completely
disabled by likely, CF solves http-> https redirect)
• SSH access is password protected + Cert.
• firewall to the outside, by default everything that is needed (external APIs and their counterparts) is enabled separately
• hosting-type access via SFTP, SSH, although shell access may be possible

CF = CloudFlare (https://www.cloudflare.com/plans/)

edit: we have had such web servers for years, nothing is secure, but we try to make it that way

@twoj

it is clear what you need:

xFinity Router in bridge mode, if it exists for this type and your ISP allows it
or you mention a modem (Arris modem) that does not contain NAT per se and you get a public IP directly

the difference between the measurements is very large approx. 900 and 400

we didn't get ahead professionally, because this difference is not justified by the dual -NAT throughput, so there is still a cat hiding somewhere in the bag ☺

if you have the opportunity to exchange, please come back to us afterwards (the curiosity moves the whole world )

@guardian Thanks for your time and support. We already have this problem for weeks no so my boss decided to make a "big bang" and just shut off the old network and go to the new one as we run out of time to make the switch. It will be a sh*tstorm but we have 4 days as yesterday was a free day here and today most employees are not in the office and off course we have the saturday and sunday.

Till now it looks not that bad and there is a lot of progress. I wil mark this question as closed.

Well after lots of testing and trying here is why.

I had DNS Resolver options checked for:

'Enable Forwarding Mode'
'Use SSL/TLS for outgoing DNS Queries to Forwarding Servers'

Un-checking them and checking back fixed the problem!

I suspect that reboot will help as well, but I not very often reboot my router.

Hope maybe beneficial to somebody else.

@kiokoman Saved my bacon! Thank you! And, despite @stephenw10's suggestion, @kiokoman had it right: date yymmddhhmm (two digit year and no seconds).

@yakatz Also a word of warning, as some who deals with PHP's LDAP bindings on a regular basis, ldap_connect is incredibly picky about TLS/SSL connections. And until about PHP 7.3, they are very hard to override and allow insecure connection even for testing.

@Gertjan @Rico

Yes yes thanks !
I also found a very short and great video on the subject, so sharing for all people.

https://youtu.be/AZ_ju6pCbow