ATT Internet AIr

[ahole4sure]

@stephenw10

Yes, sorry I meant to send screenshot
It is static

[stephenw10]

Hmm, well hard to say what's happening then. Is there any logging in the modem showing what the second client is?

[Gblenn]

I'm pretty sure it's the TPLink switch showing up as the second client...

@stephenw10 said in ATT Internet AIr:

Just to confirm though the switch admin interface itself is configured using a static IP address?
If that is set to DHCP it could be leaking requests out of VLAN10 causing problems.

I remember having an issue with a TPLink switch of similar type (TL1016D). It was changing it's management IP over to one of my VLAN's, intermittently. But I'm afraid I can't remember now how I resolved it...

[ahole4sure]

@Gblenn @stephenw10

THIS has been an ordeal from hell .....

So for sure things have been better with a Linksys Smartswitch - GS105E-200NAS
However, the MAIN issue has been finding out that the second modem of the 2 that I received from ATT was not configured properly - that is one of the reasons that the gateway kept going offline.

ATT is working on confiuguring modem #2
With modem #1 I have succesfully gotten the DHCP for the ATT wan to go to the public IP address !

The questions -- do my VLAN settings look correct?

Also if I configure the ATT modem as a failover and then fail my fiber modem (by disconnecting it)
My LAN internet (delivered mostly by Eero wireless) was not existent until I created a rule for the ATT modem to beable to access any source and any destination? Is that acceptable?
The network seemed a bit squirrely but I wasn't sure if I needed any other settings like - should I normally be able to get by with "auto-created" Outbound NAT? Or do I need to have my Outbound rules in hybrod mode? They are a MESS currently from pst attempts to "fix" problems!

Last question - after I reconnect my fiber modem - it took like 5 mintues or longer for my LAN (and wireless) internet to come back up properly suing my fiber ---- it seemed like it was stuck in some sort of limbo land trying to convert back to fiber from the ATT backup. Wasn't sure how to troubleshoot that - or do you have suggestions?

THANK YOU GUYS AGAIN!!!

[Gblenn]

Sounds like it may have been the modem and not the switch? Once you get it working, I'd change back to the TPLink switch to see if it is also ok, which I suspect it is. I use TPLink Omada switches but have one older switch with the same interface as yours, and it's been really stable. I think the one thing that I had to do was to set it's management IP manually so it wouldn't pick up an IP from one of the VLAN's. I also have that 5-port Netgear switch and I had some problems with it where I had to restart it now and then because the UI didn't work. It was switching traffic and VLAN's worked fine but for some reason the web interface locked up.

@ahole4sure said in ATT Internet AIr:

The questions -- do my VLAN settings look correct?

I think they do, for the most part, except that ID 1 (default) should not be changed from the std setting (untagged). It looks like you changed it to tagged on port 1? The only time you make changes involving ID 1 is when you want to exclude it from one of the ports. Like when you connect your NAS or other servers to a port in order to isolate them from the rest of the networks.

During testing it could be a good idea to keep one of the other ports at default setting so you have an alternative port to access the UI...

Also if I configure the ATT modem as a failover and then fail my fiber modem (by disconnecting it)
My LAN internet (delivered mostly by Eero wireless) was not existent until I created a rule for the ATT modem to beable to access any source and any destination? Is that acceptable?

Check your rules on your default LAN, the one at the bottom, that is your default any source to any network rule. Remember, this sits below any other blocking or routing rules, and is Internal to External. You want any devices on your LAN to be able to access the world, and that includes everything. So each VLAN needs to have that rule at the bottom. But in order to really isolate a VLAN from the rest of your networks, you have to add Blocking rules above that. One rule per the other networks that you want to block access to. So on VLAN 10 you will have a Block rule with source Any and destination VLAN 20. Another one with dest VLAN 30 and of course your LAN. On the LAN network you do want to be able to access the VLAN's I suppose, since you want to reach your NAS and whatever servers or devices you have. So typically no blocking access to the VLAN's.

The network seemed a bit squirrely but I wasn't sure if I needed any other settings like - should I normally be able to get by with "auto-created" Outbound NAT? Or do I need to have my Outbound rules in hybrod mode? They are a MESS currently from pst attempts to "fix" problems!

You should be able to keep your outbound rules to Auto, and not mess with hybrid and adding rules manually there.

Last question - after I reconnect my fiber modem - it took like 5 mintues or longer for my LAN (and wireless) internet to come back up properly suing my fiber ---- it seemed like it was stuck in some sort of limbo land trying to convert back to fiber from the ATT backup. Wasn't sure how to troubleshoot that - or do you have suggestions?

When you unplug your fiber, you should notice a short interruption. Like if you are on a Teams call, it will freeze for a few (7-10 seconds) and then get back up again when it has switched over to the failover connection.
When you reattach the fiber you should not notice anything. The default setting (I think) is not to flush states, which means that connections remain on the failover gateway until you close them. So your Teams meeting will continue on the failover gateway until you close the meeting. Only when you start a new meeting, will it end up on the fiber again. You can change this so pfsense will Kill states also at recovery, which means that you will get that short interruption and reconnect when you recover from a failover.

The time it takes depends on your settings under Routing where you define the "decision criteria" for switching between gateways. Packet loss or member down for example as well as the threshold numbers.

THANK YOU GUYS AGAIN!!!

[stephenw10]

That looks correct. What firewall rule did you have to add though?

I wouldn't expect any firewall rule to be needed. Nor any outbound NAT rules as long as outbound NAT is still in auto or hybrid mode. The new WAN is DHCP so they will be added automatically.

[Gblenn]

@stephenw10 said in ATT Internet AIr:

I wouldn't expect any firewall rule to be needed.

I'm not sure the any to any rule is actually created automatically when you create a VLAN?

[stephenw10]

The outbound NAT rules are created automatically.

Firewall rules are not but shouldn't be required. Hence I'm curious about exactly what rule had to be added manually.

[Gblenn]

@stephenw10 said in ATT Internet AIr:

The outbound NAT rules are created automatically.

Yes but the NAT reference was

@ahole4sure said in ATT Internet AIr:

I wasn't sure if I needed any other settings like - should I normally be able to get by with "auto-created" Outbound NAT?

And the rule I was referring to at least was related to this question, where a rule is needed for internet to be accessible.

My LAN internet (delivered mostly by Eero wireless) was not existent until I created a rule for the ATT modem to beable to access any source and any destination? Is that acceptable?

But now I'm wondering if there is something that is not right in the setup, since one VLAN at least, is only for WAN2 (or 3?). And in this case, there shouldn't be rule, other than for testing that the VLAN is actually working...

[stephenw10]

Yup exactly. That rule shouldn't be required. So lets see it.

[ahole4sure]

@stephenw10 @Gblenn

I have been so frustrated with the whole process I have not acted very systematically
I know the any rule depicted here was forgotten by me and for sure not added automatically. So I know it has to be manually added (I guess for VLANs)

But in my frustration I added a NAT rule that most likely wasn’t needed.

I have until the end of the week. Thank you both for the replies. I’ll go back to auto rules on the outbound NAT and test
Then I’ll go to my firewall rules and try to get them cleaned up (maybe send some screenshots later)
Hopefully you both won’t mind chiming in on my mess of rules. lol don’t be judgmental lol

[stephenw10]

Hmm, nope you absolutely shouldn't need that rule on a WAN. That passes traffic from the modem side into the firewall which should not be needed.

[Gblenn]

@ahole4sure A rule like that will be needed for your NAS- or Guest-VLANs only.

But not for the VLAN you have for the ATT modem (rosegate...). Not sure anymore which VLAN is used for what though...

[ahole4sure]

@Gblenn @stephenw10

The saga continues -- it appears that the second (in my discussions) of my two ATT modems may be bad. The back end ATT people swear that it is provisioned correctly. They are overnighting a replacement device with new SIM tomorrow.

On another note - I did as @Gblenn suggested and set up an additional test scenario and I was able to get Modem #1 to work through the TP- Link switch
So her is the current problem -- I have simulated power failures and reboots of the pfsense box. The modem and switch boot quicker on power failure AND if I just do a reboot of the pfsense box without booting the modem - I am unable to reegain connection. The connection is restored after modem manual reboot. During the time of trying to regain connection the modem just cycles through connection and disconnection to the pfsense box. (screenshots are 5 sec apart)
I assume it is just not renewing the lease - but can I force it???
Have you ever seen this behavior before? Any fix or workaround? I am trying to make this as self fixable as possible since I will eventiually deploy 5 physical hours away from me with no tech savvy on site employees.

[stephenw10]

What do the pfSense logs show when that's happening? Check the system and dhcp logs.

[ahole4sure]

@stephenw10
Soi strangely enough , while testinng the TP-Link switch, and this time without power failure or reboot - the gateway just went down (not sure exactly when) but has remaained down for several hours

When I checked the modem it was doing that cycling connecting , disconnecting thing

The only relevant entries in the log (as far as current time-wise) were int he DHCP log
see attached

[ahole4sure]

@stephenw10
I rebooted the modem and connected to the Linksys switch

The modem shows connected to the pfsense igb3 mac address , but the interface never showed the IP address this time, and the gateway never showed coming online
BUT the cmd ping lets me ping google.com from the OPT6VLAN10 interface that doesn't show up as online ???

Also at the end -- do you have any idea waht those numerous "default deny" things are in my firewall logs -- for both my WAN2 and my LAN. ?? There are just so many !!
I didn't even know I had a "default deny" rule

[stephenw10]

I don't think that ping is real. It doesn't show a source address in the output. That should appear like:

But since it doesn't it implies OPT6VLAN10 doesn't have a valid address.

The DHCP logs there simply show no servers responding.

[ahole4sure]

@stephenw10
So does that mean that the ATT servers are "to blame" in this case?

I need to make a decision soon -- I have enjoyed learning and pushing through the process but sooner or later I gotta decide --

1. failover internet at my second location is not an option

2. I need a different gateway than the Nighthawk (the odd think here is that if I stay away from VLAN connection the Nighthawk seems to be stable (and survive reboots and simulated power failures)
   So on the one hand it seems like the Nighthawk>VLAN>pfSense scenario is to blame , while on the other hand is it just the Nighthawk to blame??

Any thoughts on how I might should proceed to getting to the source of the issue?
Running another ethernet cable to my proposed modem location is just not an option - it has about a 10ft run UNDER concrete floor to get to the outer wall and that run is what is feeding the cameras

[Gblenn]

@ahole4sure That blocked device that you have showing in the picture from the ATT modem is your TPLink switch, right? I wonder if that may play a part in this? The ATT modem is connected to the only device it's trying to block?!

I think you should set the IP manually and try removing that entry in the ATT modem. If you haven't done it already, it's under System - IP Setting and there you set DHCP to disable and enter the IP you want when accessing it.