ATT Internet AIr
-
Would there be anything else to explore , that would make the device intermittently connect and disconnect from the Ethernet??
Ordered new switches to trial that
Could it be anything in pfsense that would cause that?
And I guess it could be the ATT modem, but both modem # 1 and 2 do it so it would have to be a widespread issue
-
Hard to imagine it could be the switch.
Just to confirm though the switch admin interface itself is configured using a static IP address?
If that is set to DHCP it could be leaking requests out of VLAN10 causing problems. -
Yes, sorry I meant to send screenshot
It is static -
Hmm, well hard to say what's happening then. Is there any logging in the modem showing what the second client is?
-
I'm pretty sure it's the TPLink switch showing up as the second client...
@stephenw10 said in ATT Internet AIr:
Just to confirm though the switch admin interface itself is configured using a static IP address?
If that is set to DHCP it could be leaking requests out of VLAN10 causing problems.I remember having an issue with a TPLink switch of similar type (TL1016D). It was changing it's management IP over to one of my VLAN's, intermittently. But I'm afraid I can't remember now how I resolved it...
-
THIS has been an ordeal from hell .....
So for sure things have been better with a Linksys Smartswitch - GS105E-200NAS
However, the MAIN issue has been finding out that the second modem of the 2 that I received from ATT was not configured properly - that is one of the reasons that the gateway kept going offline.ATT is working on confiuguring modem #2
With modem #1 I have succesfully gotten the DHCP for the ATT wan to go to the public IP address !The questions -- do my VLAN settings look correct?
Also if I configure the ATT modem as a failover and then fail my fiber modem (by disconnecting it)
My LAN internet (delivered mostly by Eero wireless) was not existent until I created a rule for the ATT modem to beable to access any source and any destination? Is that acceptable?
The network seemed a bit squirrely but I wasn't sure if I needed any other settings like - should I normally be able to get by with "auto-created" Outbound NAT? Or do I need to have my Outbound rules in hybrod mode? They are a MESS currently from pst attempts to "fix" problems!Last question - after I reconnect my fiber modem - it took like 5 mintues or longer for my LAN (and wireless) internet to come back up properly suing my fiber ---- it seemed like it was stuck in some sort of limbo land trying to convert back to fiber from the ATT backup. Wasn't sure how to troubleshoot that - or do you have suggestions?
THANK YOU GUYS AGAIN!!!π Log in to view π Log in to view π Log in to view π Log in to view
π Log in to view π Log in to view π Log in to view π Log in to view π Log in to view π Log in to view
-
Sounds like it may have been the modem and not the switch? Once you get it working, I'd change back to the TPLink switch to see if it is also ok, which I suspect it is. I use TPLink Omada switches but have one older switch with the same interface as yours, and it's been really stable. I think the one thing that I had to do was to set it's management IP manually so it wouldn't pick up an IP from one of the VLAN's. I also have that 5-port Netgear switch and I had some problems with it where I had to restart it now and then because the UI didn't work. It was switching traffic and VLAN's worked fine but for some reason the web interface locked up.
@ahole4sure said in ATT Internet AIr:
The questions -- do my VLAN settings look correct?
I think they do, for the most part, except that ID 1 (default) should not be changed from the std setting (untagged). It looks like you changed it to tagged on port 1? The only time you make changes involving ID 1 is when you want to exclude it from one of the ports. Like when you connect your NAS or other servers to a port in order to isolate them from the rest of the networks.
During testing it could be a good idea to keep one of the other ports at default setting so you have an alternative port to access the UI...
Also if I configure the ATT modem as a failover and then fail my fiber modem (by disconnecting it)
My LAN internet (delivered mostly by Eero wireless) was not existent until I created a rule for the ATT modem to beable to access any source and any destination? Is that acceptable?Check your rules on your default LAN, the one at the bottom, that is your default any source to any network rule. Remember, this sits below any other blocking or routing rules, and is Internal to External. You want any devices on your LAN to be able to access the world, and that includes everything. So each VLAN needs to have that rule at the bottom. But in order to really isolate a VLAN from the rest of your networks, you have to add Blocking rules above that. One rule per the other networks that you want to block access to. So on VLAN 10 you will have a Block rule with source Any and destination VLAN 20. Another one with dest VLAN 30 and of course your LAN. On the LAN network you do want to be able to access the VLAN's I suppose, since you want to reach your NAS and whatever servers or devices you have. So typically no blocking access to the VLAN's.
The network seemed a bit squirrely but I wasn't sure if I needed any other settings like - should I normally be able to get by with "auto-created" Outbound NAT? Or do I need to have my Outbound rules in hybrod mode? They are a MESS currently from pst attempts to "fix" problems!
You should be able to keep your outbound rules to Auto, and not mess with hybrid and adding rules manually there.
Last question - after I reconnect my fiber modem - it took like 5 mintues or longer for my LAN (and wireless) internet to come back up properly suing my fiber ---- it seemed like it was stuck in some sort of limbo land trying to convert back to fiber from the ATT backup. Wasn't sure how to troubleshoot that - or do you have suggestions?
When you unplug your fiber, you should notice a short interruption. Like if you are on a Teams call, it will freeze for a few (7-10 seconds) and then get back up again when it has switched over to the failover connection.
When you reattach the fiber you should not notice anything. The default setting (I think) is not to flush states, which means that connections remain on the failover gateway until you close them. So your Teams meeting will continue on the failover gateway until you close the meeting. Only when you start a new meeting, will it end up on the fiber again. You can change this so pfsense will Kill states also at recovery, which means that you will get that short interruption and reconnect when you recover from a failover.The time it takes depends on your settings under Routing where you define the "decision criteria" for switching between gateways. Packet loss or member down for example as well as the threshold numbers.
THANK YOU GUYS AGAIN!!!
-
That looks correct. What firewall rule did you have to add though?
I wouldn't expect any firewall rule to be needed. Nor any outbound NAT rules as long as outbound NAT is still in auto or hybrid mode. The new WAN is DHCP so they will be added automatically.
-
@stephenw10 said in ATT Internet AIr:
I wouldn't expect any firewall rule to be needed.
I'm not sure the any to any rule is actually created automatically when you create a VLAN?
-
The outbound NAT rules are created automatically.
Firewall rules are not but shouldn't be required. Hence I'm curious about exactly what rule had to be added manually.
-
@stephenw10 said in ATT Internet AIr:
The outbound NAT rules are created automatically.
Yes but the NAT reference was
@ahole4sure said in ATT Internet AIr:
I wasn't sure if I needed any other settings like - should I normally be able to get by with "auto-created" Outbound NAT?
And the rule I was referring to at least was related to this question, where a rule is needed for internet to be accessible.
My LAN internet (delivered mostly by Eero wireless) was not existent until I created a rule for the ATT modem to beable to access any source and any destination? Is that acceptable?
But now I'm wondering if there is something that is not right in the setup, since one VLAN at least, is only for WAN2 (or 3?). And in this case, there shouldn't be rule, other than for testing that the VLAN is actually working...
-
Yup exactly. That rule shouldn't be required. So lets see it.
-
I have been so frustrated with the whole process I have not acted very systematically
I know the any rule depicted here was forgotten by me and for sure not added automatically. So I know it has to be manually added (I guess for VLANs)π Log in to view
But in my frustration I added a NAT rule that most likely wasnβt needed.I have until the end of the week. Thank you both for the replies. Iβll go back to auto rules on the outbound NAT and test
Then Iβll go to my firewall rules and try to get them cleaned up (maybe send some screenshots later)
Hopefully you both wonβt mind chiming in on my mess of rules. lol donβt be judgmental lol -
Hmm, nope you absolutely shouldn't need that rule on a WAN. That passes traffic from the modem side into the firewall which should not be needed.
-
@ahole4sure A rule like that will be needed for your NAS- or Guest-VLANs only.
But not for the VLAN you have for the ATT modem (rosegate...). Not sure anymore which VLAN is used for what though...
-
The saga continues -- it appears that the second (in my discussions) of my two ATT modems may be bad. The back end ATT people swear that it is provisioned correctly. They are overnighting a replacement device with new SIM tomorrow.
On another note - I did as @Gblenn suggested and set up an additional test scenario and I was able to get Modem #1 to work through the TP- Link switch
So her is the current problem -- I have simulated power failures and reboots of the pfsense box. The modem and switch boot quicker on power failure AND if I just do a reboot of the pfsense box without booting the modem - I am unable to reegain connection. The connection is restored after modem manual reboot. During the time of trying to regain connection the modem just cycles through connection and disconnection to the pfsense box. (screenshots are 5 sec apart)
I assume it is just not renewing the lease - but can I force it???
Have you ever seen this behavior before? Any fix or workaround? I am trying to make this as self fixable as possible since I will eventiually deploy 5 physical hours away from me with no tech savvy on site employees. -
What do the pfSense logs show when that's happening? Check the system and dhcp logs.
-
@stephenw10
Soi strangely enough , while testinng the TP-Link switch, and this time without power failure or reboot - the gateway just went down (not sure exactly when) but has remaained down for several hoursWhen I checked the modem it was doing that cycling connecting , disconnecting thing
The only relevant entries in the log (as far as current time-wise) were int he DHCP log
see attachedπ Log in to view π Log in to view π Log in to view π Log in to view π Log in to view
-
@stephenw10
I rebooted the modem and connected to the Linksys switchThe modem shows connected to the pfsense igb3 mac address , but the interface never showed the IP address this time, and the gateway never showed coming online
BUT the cmd ping lets me ping google.com from the OPT6VLAN10 interface that doesn't show up as online ???Also at the end -- do you have any idea waht those numerous "default deny" things are in my firewall logs -- for both my WAN2 and my LAN. ?? There are just so many !!
I didn't even know I had a "default deny" rule
π Log in to view π Log in to view π Log in to viewπ Log in to view π Log in to view π Log in to view π Log in to view π Log in to view π Log in to view
-
I don't think that ping is real. It doesn't show a source address in the output. That should appear like:
π Log in to viewBut since it doesn't it implies OPT6VLAN10 doesn't have a valid address.
The DHCP logs there simply show no servers responding.
-
@stephenw10
So does that mean that the ATT servers are "to blame" in this case?I need to make a decision soon -- I have enjoyed learning and pushing through the process but sooner or later I gotta decide --
-
failover internet at my second location is not an option
-
I need a different gateway than the Nighthawk (the odd think here is that if I stay away from VLAN connection the Nighthawk seems to be stable (and survive reboots and simulated power failures)
So on the one hand it seems like the Nighthawk>VLAN>pfSense scenario is to blame , while on the other hand is it just the Nighthawk to blame??
Any thoughts on how I might should proceed to getting to the source of the issue?
Running another ethernet cable to my proposed modem location is just not an option - it has about a 10ft run UNDER concrete floor to get to the outer wall and that run is what is feeding the cameras -
-
@ahole4sure That blocked device that you have showing in the picture from the ATT modem is your TPLink switch, right? I wonder if that may play a part in this? The ATT modem is connected to the only device it's trying to block?!
I think you should set the IP manually and try removing that entry in the ATT modem. If you haven't done it already, it's under System - IP Setting and there you set DHCP to disable and enter the IP you want when accessing it.
-
@Gblenn Are you suggesting that I go back to trying to manually set the IP address for the VLAN interface to the static address I have form ATT? I hasn't worked in the past but I'm up for anything -- I had hoped that I could get DHCP to work and it DOES when connected directly to the pfsense (but the issuess start when I thow the VLAN into the mix)
-
@ahole4sure No, I meant the management IP for the TPLink switch. I believe you set that block in the ATT modem so it wouldn't pick up that MAC instead of pfsense.
So keep everything as it is, set the correct MAC (for pfsense) in the ATT modem, and remove the blocking. AND, set the IP of the TPLink switch to whatever it is that you want it to be. I suppose you have already set it as static in pfsense DHCP, but still. Just to make sure it doesn't try to get an IP from the ATT modem. -
Like this
-
@Gblenn
So in reesonse to your initial reply I switch my interface to static - (so far it is staying pretty stable) I actually think that I had not gone back to that AFTER I found out one of my modems was "bad"
I would like to be able to keep using DHCP ( like appears to have been working well with dorect connection to the pfssense interface as compared to the VLAN connecting through the switch
But at this point , just getting it working is all I care about!
And it appears that ATT has no problem with providing my static IPI had already set both the Linksys and the TP-Link to static as per your pic -- that didn't really change anything
I just don't know why connecting through the VLAN screws up the DHCP delivery and stable connection ??That said - I can connect to my ATT modem after I added the virtual IP address in that subnet to the VLAN interface (that address is 192.168.2.1)
For some reason - with a ethernet cable connected as a trunk to port one of the switch and the ATT modem connecteed to port 2 of the switch I can't connect to the management interface of the switch (192.168.3.100) -- any suggestions for that ?THANKS againπ Log in to view π Log in to view π Log in to view π Log in to view π Log in to view π Log in to view
-
Those IPAlias VIPs are all conflicting. You can't have the same subnet defined on different interfaces.
-
@stephenw10
Oh crap , my bad
I thought you had said I couldSo I can have multiple subnets on one interface, but not the same subnet on two different interfaces??
-
Exactly. Otherwise the system doesn't know where to route traffic. The interface must be unique for each subnet in the table.
-
@stephenw10
So I guess a VLAN general question
If I try to go through the igb3 interface connection the connection to the switch 192.168.3.100 would not connect at all
The trunk (port 1 of the switch) is connected to the igb3 port of pfsense
The VIP configured in way # 1 doesnβt allow connection at all
The VIP configured in way #2 works it connects. But the connection is so slow itβs almost unusable![alt text] ![IMG_0336.jpeg]π Log in to view π Log in to view (/assets/uploads/files/1733452318123-img_0336.jpeg)π Log in to view -
@ahole4sure I did not mean for you to set the pfsense interface to static, I meant only the TPLink switch.
The problem you were having before was that the TPLink was "stealing" the IP that was handed out by ATT meant for pfsense. It seems you already had it set as static now.
The other way to secure that pfsense would get the IP, was to enter the pfsense MAC into the ATT interface and set the Passthrough mode to static there. In this context in the ATT modem, static means the IP should only go to one single device (the one defined with the MAC). So even though it seemed like that setting didn't really survive a reboot, it should secure that you get your Public IP via DHCP as desired.So basically the only thing I was thinking you should be testing was to remove this Blocked device, from the ATT settings you did some time earlier (I suppose as a way to make sure the IP was handed out to the right interface).
My thinking was that this strange and continous disconnect and reconnect that the ATT modem is doing, is because you have the MAC of the switch set as Blocked. So the ATT modem blocks it, and then there is no connection and it also no longer see's the MAC so it tries to connect and discovers that MAC again... and then it cycles again....
The way to reach the ATT interface is going to be via the WAN port of pfsense which is back to that discussion above. One way is to set a static route in System > Routing and just have the 192.168.3.100 IP set as going out that gateway.
-
Yup you'd need that VIP on the VLAN not the parent NIC because I think you have removed VLAN1 from the trunk port in the switch? Otherwise it could be on igb3 directly.
So really it depends what VLANs the switch gui is configured to listen on.
-
@Gblenn
I can't thank. oyu enough for continuing to try to help me resolve this issue -- and yes this strange connecting and disconnecting has to have a sourceSome addn't info that I may not have made clear -- I HAVE had the MAC address in the ATT device sice it was discussed before. Of note, if you connect to a different device the MAC address changes so I have had to enter the MAC address (if the device was out of the needed environment) and the apply the change and then disconnect the device. Currently the MAC address appears to be "sticking" and having it there does not seem to help the weird connect/disconnect issue when trying to connect through VLAN and still use DHCP at the pfsense level. (the ATT dhcp has been off as well).
The other reason that kinda made me think the "block device" was not a source of the problem is that it happened on the Linksys switch as well (and that was not the blocked MAC address - the blocked MAC address was form the TP-Link).I can defintely try to unblock the "block" but I don't think it is involvedπ Log in to view π Log in to view
And @Gblenn and @stephenw10
At this point - having spent tens of hours on this -- I wonder if I should just quit trying to include DHCP into the mix since ATT has given me a ststic IP for now (I was originally trying to future prrof things in case I didn't always have a static and tryign to accept the challenge to "make it work") -
Mmm, if it works reliably with a static configuration I'd go with it. Maybe AT&T don't expect DHCP to work in that setup.
-
@stephenw10 said in ATT Internet AIr:
Mmm, if it works reliably with a static configuration I'd go with it. Maybe AT&T don't expect DHCP to work in that setup.
@ahole4sure Alternatively, change it so you only have VLANs on your LAN (igb1) interface and keep the WAN interfaces clean and directly attached to the ATT modem(s).
-
Hopefully one of you guys are around -- I was still trying to experiment and get an understanding of why things wouldn't work
(also noticed some pretty long reboots - weird behavior)NOW THE BOX IS UNACCESSIBLE
It's been so long since I set it up. I have it out of the rack setup and on the bench. Only one cable connected.to LAN. No access.
It will be a disaster if all my setup is gone (spent hours settting up Wireguard VPN, and even more hours setting up HaProxy stuff) - but I have no backup!! dumb I get it
Whats best way to troubleshoot and hopefully save something
-
@ahole4sure Hmm, try connecting a WAN cable to see if that helps. I'm not sure but I think I have seen that happen when it's disconnected. Also, check your IP settings in the PC you are using. Does it have an IP in the correct subnet?
BTW, I think the long boot time may be due to it waiting to see if WAN is there, so there is a pause in the boot process...
-
-
@Gblenn Was able to get an HDMI connected and here is the error -- looks like a disater
I ordeered a new box on Amazon at 330 amBut if I have lost all my config the I am SUNK
-
@ahole4sure Check if there is something in this thread that can help you?
https://forum.netgate.com/topic/185312/config-xml-empty-subsequent-inability-to-restart-properly-and-what-i-did/2
In particular that he was able to log in via console (SSH?) and replace the empty file with another one. I guess though that if you don't have any backup at all, it will be really tough if it is empty. It is /cf/conf/config.xml that you are looking for.
I "consoled" in and did some snooping.
The error I saw was a PHP stack trace that ultimately pointed at config.xml being empty (pasted after my closing)
I looked and saw that...it was an empty file (zero length)
I found a recent backup that was non-empty (the latest was empty) and copied that into place.
I then rebooted and things returned to normal.