1 Gig Fios and PFSense

jbeez

@sstatjm I'd love to see a pcap of what it's doing when that happens.

Does the WAN interface lose its IP?
Can you ping the def gw from there?

Do you have ipv6 configured as well?

Can you ping out to the internet? Traceroute?

sstatjm

@jbeez I can try and do it when I get home later on. I just experience packet loss at the gateway. IP address is still assigned to pfsense. I don't have ipv6 configured. I can ping out to the internet. traceroute. WAN doesnt lose it's IP. Just have packet loss and then no connection

sstatjm

@jbeez I turned off the ONT for over 2 hrs for the gateway to release I hope. Connect it back. Internet only stay up for 5 mins before I experience packet lost again.

sstatjm

I was looking in the verizon router and saw that it calls out every hour to check the gateway for an IP. However when pfsense is connected it seems to be checking every minute or 5 mins. And thats when the gateway loses packet/ping. Also I am not using the verizon dns servers. Now I am wondering if that could be the issue also.

jbeez

@sstatjm I don't use the verizon DNS servers at all

jbeez

@sstatjm said in 1 Gig Fios and PFSense:

@jbeez I turned off the ONT for over 2 hrs for the gateway to release I hope. Connect it back. Internet only stay up for 5 mins before I experience packet lost again.

Did you try a pass all on the WAN interface to see what happens?

Your pfsense will send out a DHCP DISCOVER. 0.0.0.0 src udp 68 to dst 255.255.255.255 udp 67
Verizon DHCPd will send a DHCP OFFER. verizonDHCPd IP src udp 68 to dst port 68 (not sure if it addresses your new IP or just via mac?)
pfsense will send DHCPREQUEST. src 0.0.0.0 udp 68 dst 255.255.255.255 udp 67
Verizon will send DHCPACK. verizon IP src udp 67 to your new ip dst udp 68

I'm serious about calling them to release the lease. I can't say if it's changed but in the past it was mandatory for me to call and have them break that lease. They know exactly what you're asking when you call too, it's not an uncommon request and they've trained them to field that request.

sstatjm

@jbeez I am just at a lost as to what to do. Just want to remove this verizon provided router and avoid the double NAT.
Pass all a new rule on the WAN

jbeez

@sstatjm two suggestions,

first in the wan interface https://pfsense/interfaces.php?if=wan at the bottom are two checkboxes, they shouldn't be causing an issue but you have a strange issue, so uncheck the blocking of private networks. If that doesn't solve it its easy to revert

second, you can add your firewall rule in the floating tab to have it process before all other rules. your rule looks good as posted but I'm not sure where you have it in the loading order

Are you able to pcap the traffic with verizon on your wan interface to see what's happening after that 5 minutes?

Can you post how your WAN interface is configured? Here are the relevant sections from mine:

If you want to ssh in, I just looked at my /var/db/dhclient.leases.igb0 here's an example of my last entry

lease {
  interface "igb0";
  fixed-address {redacted}.95;
  option subnet-mask 255.255.255.0;
  option routers {redacted}.1;
  option domain-name-servers 71.242.0.12,71.250.0.12;
  option domain-name "verizon.net";
  option dhcp-lease-time 7200;
  option dhcp-message-type 5;
  option dhcp-server-identifier 96.227.128.1;
  renew 5 2023/1/13 14:21:34;
  rebind 5 2023/1/13 15:06:34;
  expire 5 2023/1/13 15:21:34;
}

sstatjm

@jbeez

Gertjan

@sstatjm said in 1 Gig Fios and PFSense:

Pass all a new rule on the WAN ....

Remove this WAN rule as it should not and will not repair or explain your issue.
It's pfSense that "connects to" the upstream gateway through the WAN.
Never ever some outside source should be able to connect to your WAN so it can access some WAN (or LAN) hosted service.

The exception would be a natted port, like a VPN hosted on pfSense. That WAN rule would only accept connection using UDP, using port 1194. The answering process would be (only) the VPN server running on pfSense.

There are not much details in this thread, so I'll add my own interrogations.
What if the DHCP handshake isn't done correctly ?
My own ISP uses DHCP, packets should be encapsulated in a VLAN channel '835'. Special DHCP options should be added so my ISP 'login' and ISP 'password' are (encoded) send with the initial DHCP-REQUST command. If not done so, I get an IP and a degraded connection, I guess to re negotiate the access ?

Your ISP router will work because ( ? ) it will update itself, and adapt if a new login procedure is needed. So its normal it would work just fine.
Btw : it isn't the ISP that updates the ISP router. It will be the ISP router that requests if an update is available, and update itself if that is the case.

Again, I mention my own experiences. I can't help with what your ISP 'wants' so you get a good connection.
The connection (the link) is fine, as when you use the ISP router, all is well.
So I tend to say : your ISP router does something that pfSense doesn't and that's why the ISP closes the door.

Btw : don't even think of calling your ISP about this.
Because you know what they will say "use OUR router, others are not supported by us (your router is supported by you)".
The path to the solution is : Go 'Google' and look who is using your ISP and is also using pfSense.

sstatjm

@gertjan When I just got the service I had no issues for weeks and then one day. I started to experience packet loss and then no internet. If all else fails I will just put the ip assign to pfsense from the router and put it in DMZ.
I notice that DCHP updates itself every hour when i was going thru a log i found when logged into the router

Gertjan

@sstatjm said in 1 Gig Fios and PFSense:

When I just got the service I had no issues for weeks and then one day

What works yesterday just fine can change tomorrow.
As I explained above : what if the ISP changes the connect procedure ?
You'll agree with me that something has changed ;)
You, as a end user, won't notice anything of all this because the ISP can 'instruct" (update) your ISP router (that is : the router will ask for an update, and get one if it's available).
The result is that your connection always works when you use the ISP router.

The thing is : pfSense can't know what changed unless you changed settings.

jbeez

@gertjan said in 1 Gig Fios and PFSense:

@sstatjm said in 1 Gig Fios and PFSense:

Pass all a new rule on the WAN ....

Remove this WAN rule as it should not and will not repair or explain your issue.
It's pfSense that "connects to" the upstream gateway through the WAN.
Never ever some outside source should be able to connect to your WAN so it can access some WAN (or LAN) hosted service.

The exception would be a natted port, like a VPN hosted on pfSense. That WAN rule would only accept connection using UDP, using port 1194. The answering process would be (only) the VPN server running on pfSense.

There are not much details in this thread, so I'll add my own interrogations.
What if the DHCP handshake isn't done correctly ?
My own ISP uses DHCP, packets should be encapsulated in a VLAN channel '835'. Special DHCP options should be added so my ISP 'login' and ISP 'password' are (encoded) send with the initial DHCP-REQUST command. If not done so, I get an IP and a degraded connection, I guess to re negotiate the access ?

Your ISP router will work because ( ? ) it will update itself, and adapt if a new login procedure is needed. So its normal it would work just fine.
Btw : it isn't the ISP that updates the ISP router. It will be the ISP router that requests if an update is available, and update itself if that is the case.

Again, I mention my own experiences. I can't help with what your ISP 'wants' so you get a good connection.
The connection (the link) is fine, as when you use the ISP router, all is well.
So I tend to say : your ISP router does something that pfSense doesn't and that's why the ISP closes the door.

Btw : don't even think of calling your ISP about this.
Because you know what they will say "use OUR router, others are not supported by us (your router is supported by you)".
The path to the solution is : Go 'Google' and look who is using your ISP and is also using pfSense.

As far as I know he's on the same ISP as I am, and there is no VLAN encapsulated traffic happening there. a GPON connection goes to an ONT device, and then hands off to either MOCA or an rj45 ethernet for data. As long as his ONT is provisioned for data over the ethernet port it does a standard DHCP request/response proceedure.

I suggested the wan rule as a troubleshooting step to see if the dhcp packets during the renewal phase were getting caught up. I have several unanswered questions still so I can't help much further here.

jbeez

@sstatjm What network interfaces are you using(hardware). Do you have any log messages? Maybe a hardware issue on the pfsense box?

Do you have anything else to test with?

sstatjm

@jbeez Running on a ASRock H370M-ITX motherboard with a Intel(R) Core(TM) i5-9500 CPU, NICs are Dual Intel LAN.
Dont have anything else at the moment to test with.

Currently I am getting - dpinger 99776 WAN_DHCP 192.168.1.1: sendto error: 13

2 those every sec

jbeez

@sstatjm In system-> Routing. WAN_DHCP ->

See if the issue persists. Not sure what it can do if your gateway goes down, you don't have a backup WAN link right?

jbeez

Also... any dpinger messages in my log about WAN_DHCP show the public IP of my Verizon FiOS upstream gateway. Your log message is showing an RFC1918 address.

Physically are you connecting just your wan nic right to your ONT? You aren't bridging those interfaces in pfsense at all are you?

mikeinnyc

WAN_DHCP Gateway ??? with your PFS Box, you plugged it in and just got a working WAN IP DHCP lease right away...until they changed it

Most likely it's not your PFSense router. It's most likely that Verizon has changed your Public IP DHCP block along with the DNS servers that Verizon uses to resolve. You did nothing wrong so don't worry. I am assuming you are willing to have your IP address changed X times per month in exchange for a lower-rate nonstatic IP. Verizon "may" change your IP faster than Toliet Paper in the NYC Subways but only if 1) you get flagged by pattern high burst traffic usage or by multiple open ports for longer than normal periods. 2) OR, It could just be your power failed and it changed the "WAN IP and the DNS servers."

I know it's hard but try to pay up and Go Static IP if you are running more than cameras outside specifically file sharing aka web hosting. Otherwise, you'll be unplugging that ONT box and trying to get a new lease may be hourly even with ddns. Best of luck and I hope you get a DHCP ... static lease for at least one year.

sstatjm

@jbeez at this point I am just exhausted. I am currently connected to the verizon router that's why you are seeing that ip address.
To get a stable connection I have to connect to the verizon router. the moment i connect directly the ONT all hell breaks lose and service drops. Yes, i did call them to release the DCHP lease........... 4hrs and 45 mins later. Back to square one.

jbeez

@mikeinnyc said in 1 Gig Fios and PFSense:

WAN_DHCP Gateway ??? with your PFS Box, you plugged it in and just got a working WAN IP DHCP lease right away...until they changed it

Most likely it's not your PFSense router. It's most likely that Verizon has changed your Public IP DHCP block along with the DNS servers that Verizon uses to resolve. You did nothing wrong so don't worry. I am assuming you are willing to have your IP address changed X times per month in exchange for a lower-rate nonstatic IP. Verizon "may" change your IP faster than Toliet Paper in the NYC Subways but only if 1) you get flagged by pattern high burst traffic usage or by multiple open ports for longer than normal periods. 2) OR, It could just be your power failed and it changed the "WAN IP and the DNS servers."

I know it's hard but try to pay up and Go Static IP if you are running more than cameras outside specifically file sharing aka web hosting. Otherwise, you'll be unplugging that ONT box and trying to get a new lease may be hourly even with ddns. Best of luck and I hope you get a DHCP ... static lease for at least one year.

idk if this is the same everywhere, but I'm in the Phila area, and my verizon wan IP has changed maybe three times in the last 2yrs, its very stable for me.

Again I don't work at verizon, IDK what they do, how consistent it is, but I can speak to my personal experience, I also have some unifi gateways and edgerouters on maybe 4 or 5 other verizon networks that I admin and they are also stable, although most of them are in this area as well.