Any chance this is related to the way the modem is connected to the pfSense box? The ISP's modem doesn't have a Bridge-mode…

If you are on satellite they are notorious for hijacking DNS and blocking all other DNS traffic. Make sure either DNS Forwarder or DNS resolver is enabled but NOT both. Most use DNS Resolver. Make sure you've selected your network interfaces correctly and selected Localhost but not Localhost on the Outgoing Network Interfaces. Outgoing is for querying up stream DNS servers for Internet related stuff. Go to System -> General Setup and at the bottom tick "Allow DNS server list to be overridden by DHCP/PPP on WAN" and test again and see if you get any traffic. Restart the machine to be sure all services fired correctly if you still don't get traffic. If you still don't get traffic thing do something like… ping 8.8.8.8 and ping google.com and see if one gets traffic. If no on both then you have another issue. Could be rule related. If yes on the IP but no on google.com then you have a DNS resolution issue.

@johnpoz: I would prob release it at the client and let it do a new discover, etc. Hard to do sometimes on devices that have no UI.  Many are typically set up via an app on a phone/tablet or sometimes via on-device web server page.  But few, if any, offer actual release/renew features on what little UI they offer. I've shelved this into the "it ain't broke, don't fix it" slot for now.  At some point I'd like to figure it out, but stuff's working so I'm going to stop looking at the log…

Well, it's a hypervisor server, right?  So are there multiple VMs running in it that might be asking for IP addresses? I'm not up on the right terminology for how VMWare handles virtual switching, bridging, NAT and the like.  But eventually it boils down to the one hardware interface can end up fielding traffic for multiple virtual machines within it.  Is there the possibility that you have multiple VMs or containers on the machine?  That'd be one explanation.

I got another look on DHCP sequence and compared to other devices. In short, VoiP controller receiving what was requested in DHCP Discover poll. VoiP set Broadcast flag 0x8000 and pf sense replied to broadcast address accordingly. Other PC and devices do not set Broadcast flag and pfSense replying to machine IP address as mentioned by JKnott.   The other thing, Wireshark shows End option missing in DHCP Discover which is normally (255) End. But most important, for some reason VoiP do not reply on received DHCP Offer basically not following protocol. Correct protocol handling happening during soft/hard restart only which make me pretty much sure it's VoiP device software problem. Thanks for tips.

There are two DHCPs.  (1)ISP DHCP which provides your FW with a public IP and allows u to connect to the Internet, and (2) Your own internal DHCP which provides LAN IP to your internal clients, and both should be independent of each other.  Lets say (1) fails, your internal clients can (should) still be able to talk to each other.  If (2) fails all by itself, that's totally strange and I'd suspect something wrong how u have it set up. When you say you cannot connect to the Internet, you should:  (1) Can ping FW? and IPCONFIG /ALL on client shows normal? (Intranet issue), and (2) Does pFsense WAN interface has acquired a public IP? (ISP/WAN side issue).

Unless the DNS requests made by said gizmo can only come from their custom DNS server, there isn't really a downside to redirecting them. This is what I prefer to do: https://doc.pfsense.org/index.php/Redirecting_all_DNS_Requests_to_pfSense Any client that requests DNS from a remote address other than the firewall, gets redirected to the DNS service on the firewall (resolver or forwarder, pick your poison)

@Derelict: I am pretty limited to what I can do after this. Everything looks fine on the pfSense side. Maybe try to ping something on the 192.168.2.0/24 network from Diagnostics > Ping. That should put the pfSense MAC address in the switch's table. If not you need to figure out why not. That switch certainly supports mirroring. Mirror that port to something else and wireshark it. You are fortunately/unfortunately dealing with a 10G SFP+ port or I'd suggest putting a laptop interface on vlan 10 and plugging in directly. Assign OPT1 to a gig-e port and try that? After all your help i'm just stuck. Right now I'm returning my T320 card and buying a T520 10gb card and see if that was may problem. I'm have a really good feeling it it. Will post when i have tested with the new card! Thanks all for your help so far!

@johnpoz: Being worried about something going offline because your dhcp server is offline for your servers.. Hmmm.. Why not just run a longer lease.. If you ran a lease for 24 hours.. You would be sure - unless you rebooted them during your dhcp outage for good 12 hours..  Leases renew at the 1/2 mark normally, so any client should always be able to run for 1/2 of your lease time with dhcp server offline.. The advantage of running dhcp for stuff like servers via reservations is you can facilitate a change across a huge amount of devices with a simple dhcp change.. Say you want to point to different dns, or new gateway, or change your ntp server, etc. etc.. There are multiple options that can be handed out via dhcp that setting client to static would force you to touch that client on such changes. Shoot you can change the IP range on your whole network with a simple dhcp server change without actually have to touch a device, etc. If your worried about dhcpd going down - its also very simple to just run a failover setup for dhcp..  Couple of devices sure static right on the device.. But as you ramp up  the number of devices actually setting static on the clients becomes a PITA if something needs to be changed on the network. As your network grows the only thing that should be static should be your routers/firewall server handing out dhcpd ;) What is your current dhcp lease time? I do agree with this for larger environments. There a still a number of cases where static is a must (at least for me), in fact there are a number of services that require it. Of course as with may things in IT, there is no one right answer or one solution to rule them all.

In normal DHCP operation, the client requests the previously held address and the server provides it, if available.  Does Wireshark show the client requesting the previous address?  If so, then it should receive it, if it hasn't been assigned to another device.  Also, the DHCP server will not give an address to another device, until the lease expires. BTW, where are you deleting the lease?

I'm with Derelict on this. While I would love to seem better/more options for managing DHCP in pfSense, if your managing 2000+ clients, you should be running redundant DHCP servers dedicated for the task. You would need to setup your DHCP servers and configure the DHCP Relay/helper on your pfSense router/switch gear. On that note, I did just check and I see you cant configure the pfSense DHCP Relay agent when DHCP is enabled on ANY interface… Odd. So if you wanted to use pfSense DHCP anywhere, you would need to configure the DHCP relay/helper on the switches.

Ah yes, this was indeed the problem. Thank you very much for your help!

It does not ping my NJI DNS server

Thanks johnpoz for the detailed response and doubly so for taking the time to test. I didn't realise quite how out of my depth I was with this; thought it would would be relatively simple. I don't really want to run a second DNS server if I can avoid it and I'm a bit concerned Bind might not be as easy to configure as I've found the forwarder (with built in wizards) until I have a lot of time to play and learn. What I'd hoped would be possible was to configure the pfsense forwarder to hold the local network records and everything else to pass to the next DNS server on the list. Even if that was possible I guess that would mean all DNS requests would have to go to Internet rather than being cached locally so maybe that's not a great idea either. Thanks again for the suggestions, I'll need to consider my options I think.

I know it has been a while. I did contact them (vyprVPN). They would not provide their DNS information. So, I guess for the time being, I am kinda have to just deal with it. I have not had any time lately to continue playing with it. One of these weekends I can resume my adventure on my current path, "dark and full of terrors"

Do you have public users that hit your pfsense web gui?  If not I see no reason to use acme cert for an admin only interface - put a cert on their you signed with pfsense CA and trust that CA… Done for 10+ years... Not renew every 90 days.. As to why your getting back answers for usatoday.com.whatever - That should not happen on domain you control, unless you have a wildcard set on it.. Which normally bad practice.. If your using unbound in resolver mode - your dns server settings are pretty pointless... Unbound would be resolving not forwarding out of the box..

flush your cache… Remember your still going to see NS listed with IPv6 for domains that have IPv6 NS... But if pfsense has no IPv6 address that unbound can use for outbound queries then there would be no way for pfsense to talk to them... So in the cache you will see none have been talked too..