Pfsense after some time starts to block all outgoing traffic except 443/80

[tonis.taht]

I have a weird situation where everything works as it should, but after some time (couple of hours) no outgoing traffic is going through anymore, except HTTPS/HTTP. And I also think this happens if the number of clients in subnet increases. All the firewall rules are default, except last LAN rule that was manually added to allow all traffic (to see if this resolves the issue).
For example:

• Doing git pull over ssh times out
• AWS VPN client has TLS handshake issues
• Email fetching does not work (IMAP)
• VoIP phones have connection issues

I have tried to factory reset everything, but it still happens. Started to happen couple if months ago all of a sudden. I am new to Pfsense and I have not figured out what is causing it.

Anyone has any idea what could cause it or how to troubleshoot.