Virgin Media SuperHub3 - Internet accessible from WAN interface of pfSense, but not from LAN

[dinth]

Hi. Ive got pfSense on ESXi VM running on a construction site - so the whole system - ESXi machine, switches, cable modem, etc is often switched on and off at same time from a fuse board.
Sometimes when the power is switched on, everything works out of the box, but on average every second time the system starts, im getting no routing between WAN and LAN/OVPN interfaces in pfSense. PfSense gets a correct ip addresses from DHCP server, i can ping from WAN interface in pfSense, i can even VPN to pfSense network from outside location, but there's no outbound traffic from internal network to the Internet. Usually it can be fixed by restarting pfSense and/or ESXi a couple of times. Anyone can advise how to troubleshoot this?

[johnpoz]

Why is your outbound nat in Hybrid? And why do you have outbound nat for your LAN address?

[dinth]

I believe that all those rules were auto generated when I was setting up pfSense.
I changed from automatic to hybrid when I was setting up OVPN and assumed that i will need hybrid if additional Nat rules are added

Kind regards

[johnpoz]

Well you have NO added rules... And NO you should not have outbound nat rules on your LAN... Did you set a gateway on your lan interface??? Which it tells you like 100 times during setup not to do..

Do you have the same network on your wan and lan?

Why do you have 10.11.0.1/32 on your em1? And you understand your tunnel network is overlapping with that pfblocker vip which created a route for 10.10/16 out em1??

Yeah you have a gateway set on your lan this 10.10.255.2??? Why would you set that? Its no longer a lan network when you do that but another wan interface..

Im surprised it works at all!!

I would start over ;)

[dinth]

@johnpoz said in No routing (?) LAN<>WAN every 2nd-3rd restart:

Do you have the same network on your wan and lan?

No, WAN interface is connected to a Virgin Media cable model running in "semi-bridge" mode.

Did you set a gateway on your lan interface???

I believe i only have one gateway.

Why do you have 10.11.0.1/32 on your em1?

That's something pfBlockerNG automatically set up. 10.11.0.1 is the virtual IP address it uses (i have chosen it, because documentation says that vIP should be outside of the ip range which is 10.10.x.x for me)

Yeah you have a gateway set on your lan this 10.10.255.2??

10.10.255.x is range for OVPN clients, 10.10.255.2 was probably my machine when i've been making those screenshots.

[dinth]

After around 20 restarts i believe i found the problem - which surprisingly doesn't lay in pfSense configuration at all.
Ive got a Virgin Media UK SuperHub 3 router/modem set up in "modem mode", which in reality is not a real bridge mode, but a "semi-bridge mode". When a computer connects to modem WAN interface it receives two DHCP leases - one from Virgin Media's network and one from the modem itself. Only the first one is correct thought, the second one is meant only for modem management. I read about this issue before and i had "Reject leases from 192.168.100.1" set up in my WAN interface.

Now, when i was comparing screenshots from pfSense made when it's working with ones made when only WAN interface has internet connectivity i noticed that "Reject leases" option is not always working:
When everything is ok, WAN gateway IP is "dynamic".
When only WAN interface has internet connectivity, WAN Gateway IP is "192.168.100.1".

So i still dont have an idea how to fix it, but it's definitely a SuperHub 3 problem.

[stephenw10]

It looks like you have no default route. I would resave the WAN as your default gateway and recheck the routing table to make sure a default route is present.

Steve

[UKdude78]

@dinth said in Virgin Media SuperHub3 - Internet accessible from WAN interface of pfSense, but not from LAN:

After around 20 restarts i believe i found the problem - which surprisingly doesn't lay in pfSense configuration at all.
Ive got a Virgin Media UK SuperHub 3 router/modem set up in "modem mode", which in reality is not a real bridge mode, but a "semi-bridge mode". When a computer connects to modem WAN interface it receives two DHCP leases - one from Virgin Media's network and one from the modem itself. Only the first one is correct thought, the second one is meant only for modem management. I read about this issue before and i had "Reject leases from 192.168.100.1" set up in my WAN interface.

Now, when i was comparing screenshots from pfSense made when it's working with ones made when only WAN interface has internet connectivity i noticed that "Reject leases" option is not always working:
When everything is ok, WAN gateway IP is "dynamic".
When only WAN interface has internet connectivity, WAN Gateway IP is "192.168.100.1".

So i still dont have an idea how to fix it, but it's definitely a SuperHub 3 problem.

Hello I believe you are the first person on the entire internet to solve this riddle, the SH3.0 is just plain faulty. I must have read through 50 sites and 100s of pages till I stumbled off your post confirming what I too have noticed.

I am guessing you did not find a fix, but I think the only thing to try is go back to virgin media and ask for a SH 2.0, not sure how easy that will be but maybe get one off ebay or virgin and see if they can re-sync you back up. If you are on the same bb package it maybe possible.

Otherwise I do not believe SH 3.0 and pfsense is ever going to work sadly.

[dinth]

Hi UKdude76.
I have moved my PfSense instance to a dedicated machine (an SFFPC made by Dell, before my PfSense was running in a ESXi VM) and now it's working fine.
But even before this migration i have managed to get pretty close to sorting this issue out. At some point i have realized that changing the order and the delay in which ESXi host and VM modem boot up fixes the issue and i was using a Sonoff S26 plug to delay start of VM modem by 1-2 minutes.

[UKdude78]

@dinth said in Virgin Media SuperHub3 - Internet accessible from WAN interface of pfSense, but not from LAN:

Hi UKdude76.
I have moved my PfSense instance to a dedicated machine (an SFFPC made by Dell, before my PfSense was running in a ESXi VM) and now it's working fine.
But even before this migration i have managed to get pretty close to sorting this issue out. At some point i have realized that changing the order and the delay in which ESXi host and VM modem boot up fixes the issue and i was using a Sonoff S26 plug to delay start of VM modem by 1-2 minutes.

That is very interesting I did already attempt to boot up the modem in modem only mode let it sync then reboot the pfsense box but it still had the same fault with dhcp not appearing correctly it just flicks to n/a and then displays the real isp 81xx etc and then off again.

I tried vice versa also to same results.

Some others suggesting that boot order to fix on Virgin forums and other places, so its still a mystery how your set up is working and a few others is not. I may try it on a virtual or another pfsense box, maybe its just the pfsense hardware.

But I see what your saying with the modem being semi bridge mode and having 2 dhcp addresses since sometimes I get the 192 address and others the proper 81 ip address, its like the modem is half way up the ladder before falling down it. Normal router mode 100% works fine with 192 address syncing 10x out of 10 but then for connections its capped severely with large downloads.

Anyhow I begged Virgin media to send me a SH 2.0 and told them otherwise Id have to cancel everything, Virgin staff said let me see what I can do, their now sending out an another SH 2.0 router so I should be back in business, SH 2.0 100% works with pfsense and has done for past 5 years.

[UKdude78]

Tested the issue further, added another ethernet card and guess what worked flawless with dhcp getting picked up, tried it many times and it and I don't need to add any Reject leases from 192.168.100.1 entry.

Maybe pfsense just hates my onboard ethernet ports, either way issue sorted.