Starlink - No Internet when "Reject leases from" Configured
-
@stephenw10 said in Starlink - No Internet when "Reject leases from" Configured:
To allow the upstream device to boot fully before pfSense you can create the file /boot/loader.conf.local and add to it:
autoboot_delay="30"Or whatever the minimum delay required is.
What is the unit of the value? seconds, minutes?
Do I set this value AND the Protocol timing/Timeout under the Interfaces/WAN -
It's in seconds. By default it's set to 3s to allow you to select something from the bootloader menu but can be set much higher.
Where that helps is if the upstream device bounces the link during it's boot. If pfSense tries to pull a lease at the point the link is down the dhclient just fails out and no delay setting within the client config will help. Setting a delay before boot prevents hitting that.Steve
-
@stephenw10 Thanks for the clarification on the boot delay seconds. I probably won't leave it at 30 but I do like having more than 3.
Here is the scenario. Working system with a valid WAN connection via the Starlink Brick. Reboot either device and pfSense gets a valid online condition with a v4/DHCP4 address.
Pull the WAN connection from the Starlink brick and plug in the Starlink router. Let it fully come up, the ethernet connection router is valid and workig. The WiFi connections are valid, everything works. Reboot the Starlink router, everything comes back online. Reboot the Starlink dish, everything comes back online.
Unplug the Starlink router from the brick and plug in the SG-4860 WAN port.
But, even with the Timeout value set and the delay, rebooting together, rebooting separately with one first then the other, it just does not want to connect if the 192.168.100.1 is set in the "Reject Leases from" field. I was monitoring both the console port on the SG-4860 and the GUI via a raspberry pi I had running there. After the Starlink brick is rebooted or power cycled, I would get an "online" condition that lasted about 10 second then dropped out. I would even see the v4/DHCP4 address on the console, but as I refreshed the screen, it too would drop. Then it is back to physically unplugging the wan connection at the Starlink brick multiple times and eventually you get a online condition. Once you have a valid connection you can reboot either and the system comes back online with a valid WAN address via DHCP. Which oddly enough is on the 100.91.143.x/10 network.
Just to add another piece to this puzzle. Spoke to someone local who (like Midi) is also running a Netgate 6100 with Starlink as her primary and a 5G wireless carrier as a back up WAN connection setup in failover. IF she loses Starlink and pfSense switches to the 5G wireless. It will never switch back to Starlink. Even if she physically unplugs the 5G connection, it will not reattach Starlink... until she plays with reboots and plug/unplug game. Before she had AT&T Uverse as her primary, same setup and the failover worked fine. She has not moved to 23.01 yet.
It's just getting that initial connection to work is the key. I know its probably something in the Starlink negotiation... but they say if it comes up OK on their router, it's not their problem.
Let me know if I can do anything or pull data for you.
Thanks,
Rick -
Hmm, so to be clear if you reject leases from 192.168.100.1 it never pulls a lease? Even with reconnecting the WAN etc?
But if you don't reject those leases pfSense ends up with a local lease from the Starlink brick and doesn't pull the CGN IP that allows it to connect out?
I think we tried this earlier in the thread but it would be good to check the logs to see what the DHCP server IP is that hands out the lease in each case.
It's hard to see why refusing those leases would prevent pulling leases from another server, if that's what's actually happening.
Steve
-
@stephenw10
I was watching the boot cycle during re-connect issue today. I kept refreshing the console and it did try to connect to 192.168.100.100. Then it dropped, when it finally connected it was to 100.91.143.xx/10.
The gateway shows 100.64.0.xx
My DDNS cached address is 98.97.61.xxI posted the real data to a direct chat.
There was another post on Reddit last night with a Starlink user having trouble with pfSense. Is anyone on the Dev team running Starlink?
Rick
-
@ramosel said in Starlink - No Internet when "Reject leases from" Configured:
Is anyone on the Dev team running Starlink?
Not yet as far as I know. That would make this a lot easier!
-
@stephenw10 said in Starlink - No Internet when "Reject leases from" Configured:
@ramosel said in Starlink - No Internet when "Reject leases from" Configured:
Is anyone on the Dev team running Starlink?
Not yet as far as I know. That would make this a lot easier!
It would, but I'm willing to help if anyone wants to look.
Otherwise, Austin is not that far from Boca Chica.
-
Hmm, there's enough Starlink users this has got to be a known issue.
Just to be clear if you remove the reject leases line from the config it ends up with a lease from that IP that cannot connect externally?
-
@stephenw10 Correct. It gets a lease on 192.168.100.1 for a short time, probably no more than 10 seconds (no subnet delineation) drops it and gets one on 100.91.143.1/10. The final one works.
-
Hmm, maybe I've lost track here then. What's the problem with not rejecting that then?
I thought that was required to prevent it ending up with a bad lease that could not connect.
-
@stephenw10 That's OK, I get lost and I'm in it....
If the reject address is there, it never gets the 192.168.100.100 address... so it just sits. Never connects.
If it's removed, it gets the 192.168.100.100, drops it and then you go into this mode where you need to reboot, unplug, replug... then it finally gets an address on 100.91.143.xx/10 and works. Then it will reboot and reconnect just fine.
But if there is a prolonged outage, or I plug in the Starlink Native router (to keep it up to date), then you start this whole dance over again.
What I'm hearing from the lady running Starlink as a Primary WAN and a Verizon 5g Home wireless gateway as a secondary in a failover is that if Starlink Fails over to Verizon, it never comes back unless she does the same reboot/unplug dance I'm doing on the Starlink only system
Hope that clears it up...
When it's working, it works great!
-
@ramosel Just FYI, the outage on Friday that started the latest round of reconnect fun was legit. Starlink had an expired cert on one of their ground stations. They were offline for half an hour or more. That was the cause of the lost connection.
-
Hmm, so it's as though it must pull that 192.168.100.100 lease in order to reach the second stage but it fails to retry the new lease without manual intervention?
I wonder if the FreeBSD dhclient is not respecting the given lease time? Does it eventually connect when the 192.168.100.100 lease tries to renew?
Does it connect if you just manually renew it from the Status > Interfaces page?Steve
-
@stephenw10 said in Starlink - No Internet when "Reject leases from" Configured:
so it's as though it must pull that 192.168.100.100 lease in order to reach the second stage
Added to that question : if a PC was hooked up directly to the disk, would it establish a working connection ?
I understand that , normally, the disk (the 'thing' outside) is connected to a router inside.
All network devices are connected to this router, directly, or indirectly with a switch.That a 'disk/modem/etc' comes up with a RFC1918 LAN network first upon boot : that's a known behavior.
As soon as the disk establishes its uplink - and this time the word uplink isn't an understatement, it really links upwards - it will obtain a 'real' WAN IP.
At that moment, it - the disk ! - should take it's LAN interface down, and then UP, so the DHCP client on the WAN interface of the spacelink router - or pfSense being the router here - asks again for an IP. This time a routable IP is obtained.For what I know :
I never actually saw a SpaceX Starlink kit.
I understand that many Starlink don't 'need' the Starlink supplied router, as it is just a router with (probably) wifi capabilities.
I'm pretty sure 'thousands' removed the (startlink router) router and use their own router instead.
The real question is : @Ramosel why do you have an issue ? What's different with your setup ?
Are Starlink devices region depended ? Is this because you have a type A setup and all the others are using a type B setup ?
Like A = home and B is a pro setup ?
Why would Starlink make (your) live harder ?@ramosel said in Starlink - No Internet when "Reject leases from" Configured:
Correct. It gets a lease on 192.168.100.1 for a short time, probably no more than 10 seconds (no subnet delineation) drops it and gets one on 100.91.143.1/10.
During that time :
What happens when you look here during the entire sequence :
Status > System Logs > DHCP
Status > System Logs > System GeneralPersonally, myself, I don't bother looking at the gui.
I open two SSH connections.
In the first : tail -f /var/log/dhcpd.log
In the second : tail -f /var/log/system.logI would be watching for the 'general' system log because that's the place interface DOWN and UP will take place. Upon WAN interface event, a whole lot of lines will be logged, as many processes will get restarted if something happens with WAN. The one of interest is called "dhcpd".
WAN Interface events of interest will use the interface device name, like 're0' or "em0' or 'igc0' where 0 can also be 1 etc.The logs will help you to see what happens when. YOU can create all the scenarios, the logs will tell you what happens.
"DHCP" (IPv4) is a very mature process, and 'normally' no user interaction is needed to make it work.In your case, where only a RFC1918 type is available upon 'disk' boot (and as soon as the DHCP server build in the disk start up right after it brought up electrically its 'LAN' port), pfSense will notice that it's WAN port became active (having electric signals) and kicks of a dhcp client on the WAN interface, and starts requests (broadcasting) "DHCPDISCOVER".
the disk DHCP server will receive these, and it has been set up like : give an RFC1918 in the 192.168.100.x range (himself probably being 192.168.100.1).
At that moment, even when the disk DHCP server is running, the other part of the disk is still aligning, looking for satellites, and doing whatever is needed to make all the satellite connection. You already get a RFC1918 IP 192.168.100.x IP so you can use that to connect yourself to the 'disk' GUI (the disk build in web server it probably has one) to see internal data like satellites found, position, connection passwords etc etc.Satellite finding and connecting takes time, that's easy to understand.
So, at this moment, the WAN IP of pfSense is set : 192.168.100.x and a gateway is known (192.168.100.1 - but check this : is this the case ??)
The gateway IP is important as pfSense, by default, will ping (the so called 'dpinger' process) this IP 192.168.100.1 to check it's connection. If no ping then it will take action - and that means : resetting the WAN interface, so the pfSense DHCP client will kick in again etc.After some time, several minutes probably, the disk obtains a working satellite link.
At that moment :
It will shut down its own local 'disk build in' DHCP server on the LAN port.
It will bridge its LAN port to the disk-internal-uplink-port (using radio waves, satellites, probably some fiber back on the ground, the whole thing).
It will bring up its LAN port. All traffic coming into the disk LAN port will now be send to the disk == send to the satellite == send to the ISP called 'spacex'.pfSense doesn't know 'sh*t' about all this, all it senses its WAN being brought down : ok, link down so it does all the WAN down housekeeping stuff like destroying existing routes and all other things routers like to do.
And then its WAN link come sup again.
The pfSense DHCP client is started and sends as usual a first DHCPISCOVER. This request will not get intercepted by the DHXCP server in the disk - as it doesn't exist anymore. It will get send to the disk => stallite => etc etc deep into the spacex network.
Somewhere in the ISP spacex network (on the satellite, at Elon's home ? Dono) a SpaceX DHCP server will receive the DHCP request from the pfSense DHCP client and answer it.
This time, this DHCP server will propose a 'real' routable WAN IP, probably a DNS or two, a gateway IP (important - what is it ?) and maybe more.The gateway IP is again used by pfSense, as a dpinger process is started :
ps ax | grep 'dpinger'so it can check the WAN (up) link - and take actions (== reset interface) if the connection goes bad.
Why do I tell you all this ? Two reasons :
- As soon as you understand what you see (the logs), you can find a way out, aka : the solution.
- Understanding DHCP means DHCP will not give you issues anymore (I can't prove this scientifically, but somehow I know it applies)
-
@gertjan said in Starlink - No Internet when "Reject leases from" Configured:
Thanks for ALL that, gertjan... I notice avatar is a chevron. Military background? I was a US Navy Pilot. I have a brain injury. I'm still all here but can only bring my focus to bear for about 20min at time, so I'll try to digest all you sent in pieces over the next day or so.
Added to that question : if a PC was hooked up directly to the disk, would it establish a working connection ?
I understand that , normally, the disk (the 'thing' outside) is connected to a router inside.
All network devices are connected to this router, directly, or indirectly with a switch.For what I know :
I never actually saw a SpaceX Starlink kit.
I understand that many Starlink don't 'need' the Starlink supplied router, as it is just a router with (probably) wifi capabilities.
I'm pretty sure 'thousands' removed the (startlink router) router and use their own router instead.Lets take this part first. I have not tried a PC direct, but will today. Interesting thought.
Because of my IT background (not on the network side of the house, but I've certainly worked with them) and a family FAA connection to SpaceX, I was brought in on the Starlink Beta early. I have the V1 dish (round), there is now a V2 (rectangle).
V1 consists of a Dish, a power brick+ (it's more than just a power brick) and the wireless router - The wireless router does have a single RJ45 jack, for local ethernet past the router. You are right in that in this V1 configuration the wireless router can be removed from the setup.
V2 consists of a Dish and a wireless router. This router acts as a termimus for the dish and a power cord. The V2 router cannot be removed from the system and natively has no ethernet port. That is added via a "dongle" of sorts that strips off a physical port from the dish connection prior to the router. It has to be ordered and paid for as an extra.
The V1 power brick is not just a power source, there is some networking capability in this device which has an AC power input and two POE ports. One (black) goes up to the dish, the other (white) goes to the wireless router. In this case, I use the white connector to go directly to the WAN port on by SG-4860. All the rest of my network is then under pfSense control, LAN, OPTLANs, wireless backhauls, switching, ipPhones, alarm, etc.
If you want to DM me an email address, I'll gladly shoot you pics of all this if that would help. Or you can see most of it here: https://youtu.be/Zsuf7pdl_1M
So while I am certainly not unique, this Starlink/pfSense setup being used in small numbers. Most V1 users just use the supplied router. Likewise, most V2 users do bemoan the lack of native ethernet but some are ordering the dongle and I do know of just a few running pfSense off the dongle, such as the young lady running the failover with a 5G service that will never switch back. I personally don't know anyone using a plastic box router behind Starlink V1 or V2. I know a few who have tried, but gave up. But I'm sure they are out there and working.
Though Starlink is being used in every locale... Starlink is designed for rural users. Most rural users just want working internet, never unplug their Starlink router and are happy with wireless everything.
I absolutely agree with your 2 reasons at the end. It's why I'm here.
-
Mmm, it appears that either the Starlink router is doing something special with the dhcp client or the pfSense dhclient behaves differently to other routers/hosts.
Connecting a host directly would be an interesting test there.
The fact that Starlink don't exclude 3rd party routers entirely implies their own router is not doing something special like, for example, AT&T do.
-
S stephenw10 referenced this topic on
-
S stephenw10 referenced this topic on
-
If you have a moment : https://forum.netgate.com/topic/179959/starlink-and-pfsense/4?_=1683716772366
My question is :
Does the 'disk' pulls down it's LAN for a brief moment, when it has finished creating a stable uplink/downlink ?Can you read what I've said over there ?
Anything to add ?
Take note : I'm not a 'starlink' user, so everything is pure speculation on my side.On one side : I can't imagine that starlink (the disk) uses something utterly none-RFC compliant solution.
On the other side : If I was an ISP, I would force users to equipment I give them, so support is way easier (= read : way cheaper ) -
I have a developer that lives in the sticks in Wyoming. His only option was line of sight wifi or Starlink. The current setup is similar to this one. The v2 dish to pfsense. the problem with starlink is it uses cIP address in the DHCP pools to hand out to all the dishes and do all the routing on a virtual layer. Software defined Networking at its finest. cIP addresses are Carrier grade IP addresses which are non route-able publicly. So in order to get his pfsense box to show it was online was to set the monitor ip in the wan gateway to 8.8.4.4. So it looks like this-
we also have 3 VPN site to site clients running fulltime on this box. Mainly do to this user not being a network person at all, So I have to remotely manage it via these tunnels.
I did not see anyone else ask this, but did you reboot the pfsense after rebooting the dish? because if you didn't it will not work. Or if you did not enter the Starlink as the hostname in the dhcp options screen.
-
Hmm, so simply setting a different monitoring IP allowed it to work?
-
@stephenw10 That with adding the hostname here.
