Outbound/Inbound blocked after a few minutes

SirIrish

I decided to make a new post as I had marked the other one as solved.

I done an upgrade and I had issues. Solved them all, bar one.

I have two wan gateways. One is set to default. I had pfsense setup, before the upgrade, to allow one server to use the non default gateway. After the upgrade this would not work. I made some changes in the out bound nat and I got it to work. The next day it was gone again. Both inbound and outbound.
I accidentally :-[ changed the subnet mask(on the gateway interface Static IPv4 configuration) and it started to work for a while ??? and then stopped again. Then I changed the mask back and it worked again for a while. I can continue to do this back and forth and it keeps temporarily working. It lasts about 20 minutes.

If anyone has any ideas what is going on I would appreciate the knowledge being shared with me. ;)

stephenw10

More details please. ;)
What changes did you make to the NAT? What subnet mask? How are you filtering the server traffic? Why are you using static IPs? What WANs do you have? Anything else you might think relevant.

Steve

SirIrish

Technically, I didn't make any changes I had to put a rule back into the outbound :
interface, source, * , ,,nat address,*,no

The sub net mask on the wan gateway interface. The interface of the non default gateway I am trying to go out/in. If I change that to 26 I have a connection for 20 mins.I lose connection then I change it back to 27 and again I have a connection for 20 mins. Rinse and repeat. ;D

The firewall rule to use the non default gateway.

I'm using static IP's to run 2x servers and web sites.
I have 2 gateways connected to two different broadband providers.
Everything worked fine until the upgrade. After the upgrade I had issues with rules being changed and/or deleted.

Do you have any idea why it would work for a period of time and then stop?

stephenw10

Changing the subnet gateway like that is unlikely to make any difference. Making that change though causes pfSense to reload the interface config. I predict that if you issued this at the CLI that would also bring back connectivity:

/etc/rc.reload_interfaces

So then the question is what is being set that expires after 20 minutes.

When connectivity is lost what happens if you try and ping via that interface in Diagnostics: Ping:? What is the actual error given? What type of WAN connection is this? (cable, dsl, wifi etc)

Steve

SirIrish

Thanks for the help.

You are correct, running that command from the CLI had the same effect.

Pinging from Diagnostics
PING www.google.com (74.125.228.241) from myipaddress: 56 data bytes

–- www.google.com ping statistics ---
1 packets transmitted, 0 packets received, 100.0% packet loss

It is a cable connection. The only thing that has changed is the upgrade.

stephenw10

Sorry been away for a few days.
Does the dashboard mark the second gateway as down? Anything in the system logs?
Some people seem to have been having trouble with the apinger process which monitors gateways in recent updates. This seems to often be accompanied by clearly incorrect stats in the rrd graphs and the dashboard widget like >100% packet loss or ping times in the 10s of seconds.

Steve

SirIrish

<phew>Thought you had given up :) Welcome Back.

No the dashboard all looks like it should.
I will go over the logs again tomorrow to check.
There are no other problems or issues. Just this strange one. :'(</phew>

stephenw10

Hopefully the logs will show something.
So you still have DNS resolution when it goes down. Do you have DNS servers added in pfSense on both WANs? It isn't seeing the route as down, it's still sending the packets but just not receiving anything. Hmm.

Can you repair the connection by unplugging and re-plugging the cable to the WAN? Without making any config changes.

Steve

SirIrish

The logs show nothing that happens around the time when it starts to fail.

Unplugging the Wan cable did not repair the connection.

I have 2 internal DNS server configured on the pfSense for my main domain. Which is configured in general setup. It only allows one domain in there. Should I have DNS Servers configured elsewhere as well?

stephenw10

Hmm. The DNS still functioning iS probably nothing then if it's using your internal servers.

Do your rrd quality graphs show the connection dropping out? I'm surprised there are no apinger entries in the logs.

Steve

SirIrish

OK I've been keeping an eye on it the last couple of days. There is nothing in the logs to help when the connection goes down. rrd quality does not show me anything either.

So as a test I set up another server to do the exact same thing and it works. All the time.

So what is changing in the 15 to 20 minutes that is blocking the server ??

SirIrish

I just removed and re done the rule to send it out the non default gateway(for the 100th time) and now it is allowing me to ping out but I cannot connect to any websites and cannot also connect to it from outside.

stephenw10

So when you replaced the server with another box it worked perfectly?

Steve

SirIrish

Sorry. I didn't replace the box I just created a new rule for a lab server. Now the lab server has in/out on the non default gateway but the server I want to work still does not.

stephenw10

Hmm, tricky.
So what's the difference? Is this a server issue or something related to the forwarding rules? (hard to know what that might be though).
Not really sure what to suggest. :-\

Steve