pfSense randomly blocking access to gmail from VPS?

pftdm007

Hoping someone can help me with this irritating issue...

I have this strange problem that re-occurs once every few months where for a few days out of the blue, one of my locally hosted VPS has severe issues (or simply can't) connect to imap.gmail.com. This situation usually resolves by itself without my intervention... Very strange as these things usually dont get better by themselves so naturally I blamed a misconfig or something depending on periodic changes or updates (i.e. pfblocker or snort, or something that updates blocklists on a periodic basis...)

Meanwhile connecting to imap.gmail.com from another machine (via Thunderbird or Outlook, etc) or simply accessing "gmail.com" via Firefox, etc all work flawlessly so I know the issue is between that specific VPS and pfsense (or between that VPS and gmail?).

I tried running a traceroute from the VPS to imap.gmail.com:

traceroute to imap.gmail.com (142.250.31.109), 30 hops max, 60 byte packets
 1  * * *
 2  * * *
 3  * * *
 4  * * *
 5  * * *
 6  * * *
 7  * * *
 8  * * *
 9  * * *
10  * * *
11  * * *
12  * * *
13  * * *
14  * * *
15  * * *
16  * * *
17  * * *
18  * * *
19  * * *
20  * * *
21  * * *
22  * * *
23  * * *
24  * * *
25  * * *
26  * * *
27  * * *
28  * * *
29  * * *
30  * * *

Running the same traceroute from pfsense (obfuscated my personal IP's):

 1  XXX.XXX.XXX.XXX  14.805 ms  13.352 ms  21.513 ms
 2  10.170.154.238  28.997 ms  12.929 ms  14.042 ms
 3  XXX.XXX.XXX.XXX  12.995 ms  12.986 ms  14.069 ms
 4  72.14.205.46  13.923 ms  14.085 ms  19.590 ms
 5  108.170.251.54  21.299 ms
    108.170.251.50  12.989 ms
    108.170.251.2  14.988 ms
 6  216.239.59.125  21.064 ms
    142.250.227.190  23.139 ms
    142.251.69.176  20.300 ms
 7  142.251.66.30  31.547 ms  23.955 ms
    142.251.69.191  26.732 ms
 8  142.251.49.199  32.357 ms
    142.250.209.75  25.889 ms  26.986 ms
 9  142.250.62.15  1410.976 ms
    142.251.77.144  30.548 ms  27.728 ms
10  172.253.67.52  27.012 ms
    142.251.52.182  27.749 ms
    142.251.52.170  28.108 ms
11  172.253.66.157  30.985 ms  26.984 ms
    172.253.66.159  28.000 ms
12  * * *
13  * * *
14  * * *
15  * * *
16  * * *
17  * * *
18  * * 172.253.122.109  26.155 ms

Ping from the VPS:

PING imap.gmail.com (142.251.163.108) 56(84) bytes of data.
64 bytes from wv-in-f108.1e100.net (142.251.163.108): icmp_seq=1 ttl=106 time=26.7 ms
64 bytes from wv-in-f108.1e100.net (142.251.163.108): icmp_seq=2 ttl=106 time=29.8 ms
64 bytes from wv-in-f108.1e100.net (142.251.163.108): icmp_seq=3 ttl=106 time=25.7 ms
64 bytes from wv-in-f108.1e100.net (142.251.163.108): icmp_seq=4 ttl=106 time=30.1 ms
64 bytes from wv-in-f108.1e100.net (142.251.163.108): icmp_seq=5 ttl=106 time=29.10 ms
64 bytes from wv-in-f108.1e100.net (142.251.163.108): icmp_seq=6 ttl=106 time=31.2 ms
64 bytes from wv-in-f108.1e100.net (142.251.163.108): icmp_seq=7 ttl=106 time=27.4 ms
64 bytes from wv-in-f108.1e100.net (142.251.163.108): icmp_seq=8 ttl=106 time=30.4 ms
64 bytes from wv-in-f108.1e100.net (142.251.163.108): icmp_seq=9 ttl=106 time=26.6 ms
^C
--- imap.gmail.com ping statistics ---
9 packets transmitted, 9 received, 0% packet loss, time 8012ms
rtt min/avg/max/mdev = 25.731/28.662/31.168/1.897 ms

Ping from pfsense:

PING imap.gmail.com (142.251.163.109): 56 data bytes
64 bytes from 142.251.163.109: icmp_seq=0 ttl=107 time=30.532 ms
64 bytes from 142.251.163.109: icmp_seq=1 ttl=107 time=30.742 ms
64 bytes from 142.251.163.109: icmp_seq=2 ttl=107 time=27.335 ms
64 bytes from 142.251.163.109: icmp_seq=3 ttl=107 time=31.716 ms
64 bytes from 142.251.163.109: icmp_seq=4 ttl=107 time=28.483 ms
64 bytes from 142.251.163.109: icmp_seq=5 ttl=107 time=26.640 ms
64 bytes from 142.251.163.109: icmp_seq=6 ttl=107 time=27.430 ms
64 bytes from 142.251.163.109: icmp_seq=7 ttl=107 time=28.903 ms
64 bytes from 142.251.163.109: icmp_seq=8 ttl=107 time=30.019 ms
64 bytes from 142.251.163.109: icmp_seq=9 ttl=107 time=31.404 ms

--- imap.gmail.com ping statistics ---
10 packets transmitted, 10 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 26.640/29.320/31.716/1.722 ms

For the life of me I cannot see if pfblockerNG or DNSBL are causing this issue as I don't see anything relevant in their logs... I nevertheless deactivated them and did a force reload, to no avail.

Snort also has no blocked hosts in its list and no alerts.

This is what I tried so far:

pfSense firewall: deactivated and ran in DMZ. No noticeable improvements or changes.
Deactivated Snort, pfblockerNG and DNSBL and reloaded (even rebooted pfsense to be sure...): No changes or improvements

The logs on the VPS are showing connection errors and delays:

Fri, 28 Oct 2022 12:50:22 -0400
Connection to: imap://imap.gmail.com:993/
Server connection took 0.1599 seconds.
S: * OK Gimap ready for requests from xx.xx.xx.xx XXXXXXXXXXXXXXXXXXXXx
C: 1 CAPABILITY
S: * CAPABILITY IMAP4rev1 UNSELECT IDLE NAMESPACE QUOTA ID XLIST 
CHILDREN X-GM-EXT-1 XYZZY SASL-IR AUTH=XOAUTH2 AUTH=PLAIN 
AUTH=PLAIN-CLIENTTOKEN AUTH=OAUTHBEARER AUTH=XOAUTH
S: 1 OK Thats all she wrote! XXXXXXXXXXXXXXXXXXXXx
Command 1 took 0.0364 seconds.
C: 2 AUTHENTICATE PLAIN [INITIAL CLIENT RESPONSE (username: xxxxxxxxxxxxxx)]
Slow Command: 30.03 seconds
ERROR: read/timeout error.
Slow Command: 30.047 seconds
ERROR: read/timeout error.
S: * CAPABILITY IMAP4rev1 UNSELECT IDLE NAMESPACE QUOTA ID XLIST 
CHILDREN X-GM-EXT-1 UIDPLUS COMPRESS=DEFLATE ENABLE MOVE CONDSTORE 
ESEARCH UTF8=ACCEPT LIST-EXTENDED LIST-STATUS LITERAL- SPECIAL-USE 
APPENDLIMIT=XXXXXXXXXXXX
S: 2 OK blablabla@gmail.com authenticated (Success)
Command 2 took 63.6592 seconds.

The "slow command" messages seem to indicate some network delays, etc. This VPS is running on a Proxmox host. Other VM's on this host have zero issues of that nature.

How would I go about troubleshooting this? Following Troubleshooting Website Access I went thru the list one item at the time and AFAIK everything is in order. If it was a misconfig on my side I'd imagine this issue would happen all the time and not on a periodic basis only for a few hours/days......

Ideas?

SteveITS

@pftdm007 If it was DNSBL the device would have the DNSBL IP. I do see imap.gmail.com is being resolved to different IPs in your examples. Could it be only one IP is blocked?

pftdm007

@steveits Hello! thanks for replying ;)

If one of google's IP would be blocked, it would not be on my side but again I am trying to find a way to confirm this. TO be transparent, my pfsense config has been rolled on for more than a decade now meaning each time I rebuild the system (and that was propably 10 years ago now thanks to pfsense's legendary stability) I use my previous config....

So perhaps there's a small setting or block item somewhere I would have forgot about... How would you dig deeper into this?

EDIT: I have another issue which I initially wanted to adress separately but perhaps after all they're related... Some time ago, and I cannot remember when this began, but I started seeing this error message from several VPS connecting to a local MariaDB server:

php_network_getaddresses: getaddrinfo failed: Name or service not known

The message is clear: the service cant resolve the DB server. A refresh usually works which indicates a temporary issue. This may not be related to the initial issue I describe here with Gmail and if it is NOT I will open a separate forum post to address it.

SteveITS

@pftdm007 The fact the traceroute doesn't return the LAN IP of the pfSense implies the VPS loses its network connection? Can you ping out from the VPS while the traceroute is failing?

Is it only 142.250.31.109 or also 142.250.31.108? (ping/trace by IP)

You could try continuously pinging your pfSense IP from the VPS and see if there is packet loss when the problem happens.

pftdm007

@steveits

Just logged in the VPS, ran traceroute to imap.gmail.com and its still failing, however ping works perfectly to imap.gmail.com and pfsense's LAN IP.

I tried tracerouting from another machine (a desktop computer) and traceroute to reddit.com is also failing.... I think this is a separate issue with pfsense? What in pfsense could be blocking traceroutes??? I may have done something bad in the firewall rules.... I need to check this.

To add to your comments, if the VPS was losing its network (LAN) connection I would imagine it would not be possible to login to the VPS via SSH right? Also pinging would not work...?

SteveITS

@pftdm007 I had a situation recently in our data center where we were trying to add a second public IPv4 subnet to the LAN. We could ping the data center's gateway IP but traceroute anywhere past that IP failed. They had set up inbound routing but forgot to allow outbound routing from this subnet. Not sure if that helps you but maybe. It sounds a bit different though if you can ping imap.gmail.com and not traceroute...that seems very odd unless those two commands were using different IPs.

Yeah if you're using SSH then it would have to have network access, but you also said you could ping out.

pftdm007

@steveits

Found that the issue was caused by Snort blocking Google IP's for various reasons. What I cannnot explain is why I needed (for an entirely unrelated reason) to re-config the snort interfaces to be able to actually see that Snort was the culprit. At least I am pretty sure its the case because since I last posted on this thread, I've had two episodes of connectivity issues and both times it was clear as day that Snort was blocking Google IP's. Unblocking them made my VPS reconnect almost instantly.

For now I consider this solved!