23.01 install results in no internet
-
I don't know that without digging into it. But it does vary depending on what the interface is.
It could be a race condition, yes. That's going to be considerably faster on a VM with one interface than an Atom. Check the logs from each. Do you see entries in a different order?
-
@stephenw10 so from what i can see, on the working system a log entry comes up:
May 23 21:18:31 FW1 php-fpm[74460]: /rc.newwanip: rc.newwanip: on (IP address: x.x.x.x) (interface: WAND[opt5]) (real interface: pppoe0).whereas on the other host:
May 23 21:23:41 FW2 php-fpm[854]: /rc.newwanip: rc.newwanip: on (IP address: ) (interface: WAND[opt5]) (real interface: pppoe0).so neither find_interface_ip nor get_interface_ip is returning an address, but can't figure out why not..
If i hard code it to return my static address whenever the interface is pppoe0, it comes online and everything else seems to work, but obviously this is quite a nasty hack.
-
@stephenw10 so i seem to have tracked it down...
The ISP gives me a small (/28 x.x.x.64/28) block of legacy IP, and the WAN address assigned to the PPPOE session (x.x.x.65) is the same address as the first usable one in the routable /28 block.I have the same address assigned to a DMZ interface, which other devices on this VLAN use as their gateway. When this address is active on the DMZ interface, rc.newwanip doesn't detect it on pppoe0. If i disable DMZ, rc.newwanip detects pppoe0 just fine.
With 22.05 this worked fine. It's a somewhat weird setup, but i guess the ISP doesn't want to waste another expensive legacy address by assigning the pppoe0 interface a separate address. Any idea what might have changed post 22.05 that could have triggered this?
The IPv6 setup is a little different, there is no GUA assigned to the WAN interface, but it doesn't need one as it routes via the link-local addresses, and thus doesn't have any conflict.
-
You can't have the same subnet on two interfaces unless they are bridged. If the ISP was handing you a /32 for the WAN it could work as a more specific route. PPPoE is a point to point protocol so it doesn't actually care what the IPs are.
It probably shouldn't have worked in 22.05. -
@stephenw10 The ISP does hand a /32 for the WAN address, but then the only way to make use of the /28 is to assign the same address with the /28 mask to one of the other interfaces. It's worked fine for many years (since 2.1.x) and all ISPs in the country seem to hand out routable legacy addressing this way.
Is there any other way to configure it?Speaking of same addressing, the logic in the UI also prevents using the same IPv6 link-local address on multiple interfaces - but this is explicitly allowed with IPv6, for instance its quite common to assign fe80::1 on each VLAN as the primary gateway.
-
You can add the additional public IPs as VIPs on the WAN and 1:1 NAT them to internal hosts. But that is a significant change. I'm not sure what might have changed in 23.01 that prevents it pulling the more specific route for a /32.
Does this work without the HA part?
Are you able to test 23.05? -
@stephenw10 23.05 behaves the same as 23.01..
Removing HA for the pppoe parent makes no difference...
Removing HA on the DMZ interface works, so it seems having the WAN address matching a CARP VIP has somehow broken between 22.05 and 23.01.
NAT breaks a lot of stuff so i'd want to avoid it.
-
What's actually logged when it tries to use an IP from a subnet that's already assigned to a different interface?
I expect to see an error in the system or routing logs. -
@stephenw10 If the PPP session is already active and i try to assign the /28 afterwards, it rejects it in the UI saying the address conflicts.
However when the PPP interface is down, it doesn't know what address it's going to assign when it comes online so it accepts the configuration.When it connects, the two functions used in rc.newwanip fail to find the address assigned to pppoe0 and return empty results, which causes the script to restart the interface thinking it doesn't have an address assigned.
This configuration worked in 22.05.
-
Hmm, so the PPP session connects and is passed an IP in the /28 but somehow just does nothing with it? And no routing error is logged? Hmm
-
@stephenw10 The PPP session connects and gets the address, it shows up if you run ifconfig at the right time and it even routes packets for a few seconds...
Then rc.newwanip tries to work out the address which has been assigned using find_interface_ip and/or get_interface_ip, but these functions don't return any address for pppoe0. (running these same functions against another interface such as igb1 is successful).
The correct address is present in /var/db/pppoe0_ip but it seems that $interface_ip_arr_cache doesn't get populated, the array exists but is empty which means this branch is never taken:
if (!isset($interface_ip_arr_cache[$interface]) or $flush) {so it never reads the file from /var/db.. i have yet to track down where $interface_ip_arr_cache is coming from
-
Ah, OK. Interesting. I'm not aware of anything specific that would have affected that but it could well have been an unintended effect of some other change. That would be unlikely to have been tested since it's considered an invalid setup.
I would open bug report with the specific findings you have there.
https://redmine.pfsense.org/IMO it should be valid since PPP is point-to-point and can work in a setup like that. It may just need to be excluded from a check somewhere.
Steve
-
This post is deleted!
