More Starlink "arpresolve: can't allocate llinfo..." error issues
-
I've been running pfSense on a Netgate 1100 for about 18 months.
I have a dual-WAN configuration with Starlink and T-Mobile in a load-balancing configuration and six VLANs. I'm running v23.05.1-RELEASE.
I have defined the firewall LAN IP as 192.168.2.1 because one of the Starlink IP addresses occupies the same default of 192.168.1.1.
I have Starlink configured as recommended in various posts to reject leases from 192.168.100.1 and 192.168.1.1 (I've tried with and without the 1.1 address rejection).
I have at various times successfully and unsuccessfully configured the virtual IP etc. to access the Starlink statistics page. Currently the system is not configured for this. My Starlink is a v2 square dish and router with the ethernet adapter and the router in bypass mode.
Recently (at least since updating to 23.05, not so much before but cause-effect is unclear), I have been receiving the following errors in the System/General log related to my Starlink connection:
Jul 14 10:27:23 check_reload_status 457 Updating all dyndns
Jul 14 10:27:23 check_reload_status 457 Reloading filter
Jul 14 10:27:19 php-fpm 10305 /system_gateways.php: Removing static route for monitor 8.8.4.4 and adding a new route through 100.64.0.1
Jul 14 10:27:19 php-fpm 10305 /system_gateways.php: Removing static route for monitor 8.8.8.8 and adding a new route through 192.168.12.1
Jul 14 10:27:15 check_reload_status 457 Syncing firewall
Jul 14 10:27:15 php-fpm 48414 /system_gateways_edit.php: Configuration Change: admin@192.168.10.110 (Local Database): Gateway settings changed
Jul 14 10:15:00 sshguard 42489 Now monitoring attacks.
Jul 14 10:15:00 sshguard 25870 Exiting on signal.
Jul 14 10:10:10 check_reload_status 457 Reloading filter
Jul 14 10:10:10 php-fpm 75511 /rc.newwanip: rc.newwanip: on (IP address: 100.68.183.215) (interface: STARLINK[opt1]) (real interface: mvneta0.4092).
Jul 14 10:10:10 php-fpm 75511 /rc.newwanip: rc.newwanip: Info: starting on mvneta0.4092.
Jul 14 10:10:08 check_reload_status 457 rc.newwanip starting mvneta0.4092
Jul 14 10:10:08 kernel arpresolve: can't allocate llinfo for 100.64.0.1 on mvneta0.4092
Jul 14 10:10:08 kernel arpresolve: can't allocate llinfo for 100.64.0.1 on mvneta0.4092
...[repeat above line 31 more times over preceding 2 seconds]...
Jul 14 10:04:00 sshguard 25870 Now monitoring attacks.
Jul 14 10:04:00 sshguard 84695 Exiting on signal.
Jul 14 10:02:48 check_reload_status 457 Updating all dyndns
Jul 14 10:02:48 check_reload_status 457 Reloading filter
Jul 14 10:02:44 php-fpm 84303 /system_gateways.php: Removing static route for monitor 8.8.4.4 and adding a new route through 100.64.0.1
Jul 14 10:02:44 php-fpm 84303 /system_gateways.php: Removing static route for monitor 8.8.8.8 and adding a new route through 192.168.12.1
Jul 14 10:02:41 check_reload_status 457 Syncing firewall
Jul 14 10:02:40 php-fpm 10305 /system_gateways_edit.php: Configuration Change: admin@192.168.10.110 (Local Database): Gateway settings changed
Jul 14 09:59:53 php-fpm 24578 /index.php: Successful login for user 'admin' from: 192.168.10.110 (Local Database)
Jul 14 09:51:51 check_reload_status 457 Reloading filter
Jul 14 09:51:50 php-fpm 84303 /rc.newwanip: rc.newwanip: on (IP address: 100.68.183.215) (interface: STARLINK[opt1]) (real interface: mvneta0.4092).
Jul 14 09:51:50 php-fpm 84303 /rc.newwanip: rc.newwanip: Info: starting on mvneta0.4092.
Jul 14 09:51:49 check_reload_status 457 rc.newwanip starting mvneta0.4092
Jul 14 09:51:49 kernel arpresolve: can't allocate llinfo for 100.64.0.1 on mvneta0.4092
...[repeat above groups of errors every 10-45 minutes]...I've searched for "arpresolve: can't allocate llinfo" and various permutations and read posts for many hours on this forum and reddit etc., experimented with various approaches suggested to no effect at all.
I'm not a routing expert, though I have a degree in CS and 33+ years of experience in industry though I'm now retired. Many of the discussions go into details I'm not familiar with.
Unlike in some of the discussions, the above errors are do not appear to coincide with Starlink going down or disconnecting from the router. It may be possible that the error coincide with Starlink switching satellites though I don't know how to find the data to figure this out.
Although the error doesn't seem to be coincident with Starlink being down, it is a recent change the behavior of my system and disconcerting. It's obviously not running trouble-free, and seems like a router of this sophistication should be able to.
The T-Mobile Home Internet never exhibits these issues even though it also appears to use CG-NAT, albeit the TMO gateway address and the interface address are in the same subnet, whereas the Starlink gateway and subnets are in different subnets.
Finally, here some various configuration items that might help anyone who might be able to help resolve this:
Thank you!
-M-G -
I finally fixed it, it seems.
A few weeks ago I put outbound NAT in as a fix for NTP that was throwing IPv6 errors into the system log on both the Starlink and T-Mobile interfaces. NTP was also failing to consistently determine time or serve it. All this despite IPv6 being blocked/unused on my pfSense.
However, the outbound NAT was for interface traffic going from This Firewall to anywhere on that interface translated to the interface address.
Looking at that today it finally dawned on me that the outbout NAT could be capturing the ARP traffic to the Starlink gateway. I narrowed the outbound NAT down to just port 123 and the llinfo errors stopped ocurring.
That said, holy cow the error messages in FreeBSD are frequently far, far away from being intuitively obvious. Hours of searching failed to even find a definition of what llinfo was, let alone what the error meant.
Amazingly, at one point I seem to have found the original code by the original author who commented that he'd be back around to figure this out later. Later appears to be many, many years later. Still waiting, I guess.
Dating myself, I really miss the formality of OpenVMS and the incredibly helpful Error Messages and Codes manual. Loose software engineering, standardization, phrasing and undefined error messages make trouble-shooting admittedly lame user errors incredibly hard.
I love the stability, flexibility and UI of pfSense, but better error messages and codes documentation would save many dumb old former CS professionals like me many, many hours of frustration along the way. That in turn might sell more products and support the future of the company better.