1 Gig Fios and PFSense

jbeez

@sstatjm Can you tcpdump your wan interface when its connected to the ONT and see what happens?

How about trying a pass all on wan just to see if it stays up, could be something weird with it not accepting dhcp packets or something? Just spitballing here, that must be frustrating.

sstatjm

@jbeez Thank. I will give that a try. I was thinking of doing a MAC address spoofing. What do yo think?

jbeez

@sstatjm IME with my own FiOS, that lease is very sticky. I've had to call support in the past and ask them to break my lease before it would re-assign, I wouldn't mess w/ the mac address spoofing at all, just break it and never think about it again once it's working

sstatjm

@jbeez When i just got the service it worked fine for about a month. Then everything changed. I had to put their router infront of the pfsense machine. I could get it to work for like 5 -10 mins at a time when connected directly to the ONT.

jbeez

@sstatjm I'd love to see a pcap of what it's doing when that happens.

Does the WAN interface lose its IP?
Can you ping the def gw from there?

Do you have ipv6 configured as well?

Can you ping out to the internet? Traceroute?

sstatjm

@jbeez I can try and do it when I get home later on. I just experience packet loss at the gateway. IP address is still assigned to pfsense. I don't have ipv6 configured. I can ping out to the internet. traceroute. WAN doesnt lose it's IP. Just have packet loss and then no connection

sstatjm

@jbeez I turned off the ONT for over 2 hrs for the gateway to release I hope. Connect it back. Internet only stay up for 5 mins before I experience packet lost again.

sstatjm

I was looking in the verizon router and saw that it calls out every hour to check the gateway for an IP. However when pfsense is connected it seems to be checking every minute or 5 mins. And thats when the gateway loses packet/ping. Also I am not using the verizon dns servers. Now I am wondering if that could be the issue also.

jbeez

@sstatjm I don't use the verizon DNS servers at all

jbeez

@sstatjm said in 1 Gig Fios and PFSense:

@jbeez I turned off the ONT for over 2 hrs for the gateway to release I hope. Connect it back. Internet only stay up for 5 mins before I experience packet lost again.

Did you try a pass all on the WAN interface to see what happens?

Your pfsense will send out a DHCP DISCOVER. 0.0.0.0 src udp 68 to dst 255.255.255.255 udp 67
Verizon DHCPd will send a DHCP OFFER. verizonDHCPd IP src udp 68 to dst port 68 (not sure if it addresses your new IP or just via mac?)
pfsense will send DHCPREQUEST. src 0.0.0.0 udp 68 dst 255.255.255.255 udp 67
Verizon will send DHCPACK. verizon IP src udp 67 to your new ip dst udp 68

I'm serious about calling them to release the lease. I can't say if it's changed but in the past it was mandatory for me to call and have them break that lease. They know exactly what you're asking when you call too, it's not an uncommon request and they've trained them to field that request.

sstatjm

@jbeez I am just at a lost as to what to do. Just want to remove this verizon provided router and avoid the double NAT.
Pass all a new rule on the WAN

jbeez

@sstatjm two suggestions,

first in the wan interface https://pfsense/interfaces.php?if=wan at the bottom are two checkboxes, they shouldn't be causing an issue but you have a strange issue, so uncheck the blocking of private networks. If that doesn't solve it its easy to revert

second, you can add your firewall rule in the floating tab to have it process before all other rules. your rule looks good as posted but I'm not sure where you have it in the loading order

Are you able to pcap the traffic with verizon on your wan interface to see what's happening after that 5 minutes?

Can you post how your WAN interface is configured? Here are the relevant sections from mine:

If you want to ssh in, I just looked at my /var/db/dhclient.leases.igb0 here's an example of my last entry

lease {
  interface "igb0";
  fixed-address {redacted}.95;
  option subnet-mask 255.255.255.0;
  option routers {redacted}.1;
  option domain-name-servers 71.242.0.12,71.250.0.12;
  option domain-name "verizon.net";
  option dhcp-lease-time 7200;
  option dhcp-message-type 5;
  option dhcp-server-identifier 96.227.128.1;
  renew 5 2023/1/13 14:21:34;
  rebind 5 2023/1/13 15:06:34;
  expire 5 2023/1/13 15:21:34;
}

sstatjm

@jbeez

Gertjan

@sstatjm said in 1 Gig Fios and PFSense:

Pass all a new rule on the WAN ....

Remove this WAN rule as it should not and will not repair or explain your issue.
It's pfSense that "connects to" the upstream gateway through the WAN.
Never ever some outside source should be able to connect to your WAN so it can access some WAN (or LAN) hosted service.

The exception would be a natted port, like a VPN hosted on pfSense. That WAN rule would only accept connection using UDP, using port 1194. The answering process would be (only) the VPN server running on pfSense.

There are not much details in this thread, so I'll add my own interrogations.
What if the DHCP handshake isn't done correctly ?
My own ISP uses DHCP, packets should be encapsulated in a VLAN channel '835'. Special DHCP options should be added so my ISP 'login' and ISP 'password' are (encoded) send with the initial DHCP-REQUST command. If not done so, I get an IP and a degraded connection, I guess to re negotiate the access ?

Your ISP router will work because ( ? ) it will update itself, and adapt if a new login procedure is needed. So its normal it would work just fine.
Btw : it isn't the ISP that updates the ISP router. It will be the ISP router that requests if an update is available, and update itself if that is the case.

Again, I mention my own experiences. I can't help with what your ISP 'wants' so you get a good connection.
The connection (the link) is fine, as when you use the ISP router, all is well.
So I tend to say : your ISP router does something that pfSense doesn't and that's why the ISP closes the door.

Btw : don't even think of calling your ISP about this.
Because you know what they will say "use OUR router, others are not supported by us (your router is supported by you)".
The path to the solution is : Go 'Google' and look who is using your ISP and is also using pfSense.

sstatjm

@gertjan When I just got the service I had no issues for weeks and then one day. I started to experience packet loss and then no internet. If all else fails I will just put the ip assign to pfsense from the router and put it in DMZ.
I notice that DCHP updates itself every hour when i was going thru a log i found when logged into the router

Gertjan

@sstatjm said in 1 Gig Fios and PFSense:

When I just got the service I had no issues for weeks and then one day

What works yesterday just fine can change tomorrow.
As I explained above : what if the ISP changes the connect procedure ?
You'll agree with me that something has changed ;)
You, as a end user, won't notice anything of all this because the ISP can 'instruct" (update) your ISP router (that is : the router will ask for an update, and get one if it's available).
The result is that your connection always works when you use the ISP router.

The thing is : pfSense can't know what changed unless you changed settings.

jbeez

@gertjan said in 1 Gig Fios and PFSense:

@sstatjm said in 1 Gig Fios and PFSense:

Pass all a new rule on the WAN ....

Remove this WAN rule as it should not and will not repair or explain your issue.
It's pfSense that "connects to" the upstream gateway through the WAN.
Never ever some outside source should be able to connect to your WAN so it can access some WAN (or LAN) hosted service.

The exception would be a natted port, like a VPN hosted on pfSense. That WAN rule would only accept connection using UDP, using port 1194. The answering process would be (only) the VPN server running on pfSense.

There are not much details in this thread, so I'll add my own interrogations.
What if the DHCP handshake isn't done correctly ?
My own ISP uses DHCP, packets should be encapsulated in a VLAN channel '835'. Special DHCP options should be added so my ISP 'login' and ISP 'password' are (encoded) send with the initial DHCP-REQUST command. If not done so, I get an IP and a degraded connection, I guess to re negotiate the access ?

Your ISP router will work because ( ? ) it will update itself, and adapt if a new login procedure is needed. So its normal it would work just fine.
Btw : it isn't the ISP that updates the ISP router. It will be the ISP router that requests if an update is available, and update itself if that is the case.

Again, I mention my own experiences. I can't help with what your ISP 'wants' so you get a good connection.
The connection (the link) is fine, as when you use the ISP router, all is well.
So I tend to say : your ISP router does something that pfSense doesn't and that's why the ISP closes the door.

Btw : don't even think of calling your ISP about this.
Because you know what they will say "use OUR router, others are not supported by us (your router is supported by you)".
The path to the solution is : Go 'Google' and look who is using your ISP and is also using pfSense.

As far as I know he's on the same ISP as I am, and there is no VLAN encapsulated traffic happening there. a GPON connection goes to an ONT device, and then hands off to either MOCA or an rj45 ethernet for data. As long as his ONT is provisioned for data over the ethernet port it does a standard DHCP request/response proceedure.

I suggested the wan rule as a troubleshooting step to see if the dhcp packets during the renewal phase were getting caught up. I have several unanswered questions still so I can't help much further here.

jbeez

@sstatjm What network interfaces are you using(hardware). Do you have any log messages? Maybe a hardware issue on the pfsense box?

Do you have anything else to test with?

sstatjm

@jbeez Running on a ASRock H370M-ITX motherboard with a Intel(R) Core(TM) i5-9500 CPU, NICs are Dual Intel LAN.
Dont have anything else at the moment to test with.

Currently I am getting - dpinger 99776 WAN_DHCP 192.168.1.1: sendto error: 13

2 those every sec

jbeez

@sstatjm In system-> Routing. WAN_DHCP ->

See if the issue persists. Not sure what it can do if your gateway goes down, you don't have a backup WAN link right?