Intermittent brief LAN connection issues
-
Some background on my setup:
I'm running several VMs, including pfSense, in Proxmox on an old PC. I realize this is not the best way to do things. I'm using a 4-port Intel NIC for pfSense. Ethernet comes from my modem to the port on the NIC I've designated for WAN, and a second cable runs from the port I've designated LAN to a 5-port unmanaged PoE switch. That switch also connects to my personal PC, my WAP, and (importantly) to the onboard Ethernet port on my Proxmox machine's motherboard.
I haven't messed around with pfSense a whole lot other than adding some static IPs, adding pfBlockerNG, and configuring OpenVPN.Now the problem:
In short, every 10-20s certain LAN connections are getting interrupted.My WAN access is unaffected.I'm not sure whether WAN is affected. Just had a short disconnect; not sure the cause.
Longer version: One of the other VMs running on the Proxmox machine is TrueNAS. I have a SMB share set up so I can add the network drive in Windows. At first this was working perfectly. However, yesterday I went to move a large file onto the drive, and it kept disconnecting ~10s into the transfer. I'm also finding myself getting disconnected from the web UI. On Proxmox, I've also noticed that I'm getting intermittently disconnected from the VNC connection to my VMs. I've had pfSense running for several months and Proxmox running for about twice that long without experiencing this issue until now. The only thing I can think of that I've recently changed that might contribute is changing from using my old router in AP mode as my WAP and switch to using a new switch and new dedicated WAP.Troubleshooting steps I've tried:
First, let me say I'm a total noob when it comes to networking. I'm slowly learning, but 6mo ago I barely new what a router did, had no idea what a port was, etc.- I've ruled out the new switch being an issue by reinstalling my old router in its place - the issue persisted.
- I've also ruled out interference by another device on the network (IP collision I guess?) by disconnecting the WAP from the switch so it's just my PC, pfSense, and Proxmox. No difference.
- Lastly, I've further confirmed pfSense as the issue by reverting to my old router with my PC and Proxmox connected to LAN ports. The issue was gone - I was able to transfer a 62GB file without issue.
The strange thing is that my pfSense config history doesn't show any changes since 8 days ago when I set up TrueNAS, and it definitely was working without issue for a couple days at least.
Any thoughts? Please be patient with my limited understanding here. I'd even love suggestions on free tutorials and such that would equip me with the knowledge to track this issue down.
-
So you have the WAN and LAN physical NICs passed through to pfSense in Proxmox and a separate connection for the host itself?
Check the pfSense system logs. Check the monitoring graphs.
If something is happening every 20s there should be some evidence of it there.Steve
-
@stephenw10 that's correct. Not sure if it matters that the host is using a port on the mobo rather than on the PCIe card that pfSense uses. Honestly, setting up pfSense in Proxmox was a bit of a crash course in networking, and I'm not sure whether the way I set it up was the best.
I spent a good bit yesterday looking through the logs, but I was a bit overwhelmed. I found a few firewall entries related to my NAS IP, but a bit of research indicated those entries were basically business as usual (something about the default rule, think I read a dropped packet could be the cause).
Do you have any idea what I might be looking for in the logs, and/or which tab I should be looking in?
Thanks for the help!
-
I would first look on the main system log for something repeating continually. If it's logged at all it will be filling the log completely at once every 20s!
It would be very easy to get some sort of traffic loop with a setup like that if the NICs are not isolated as intended. I could definitely imagine something like that happening though I can't see why I would start after some time unless you made some networking changes.
Steve
-
@stephenw10 Thanks!
I just reproduced the issue (both the TrueNAS disconnect and the Proxmox VNC disconnect) and combed through the logs. I looked through System (General, Gateways, Routing, and DNS Resolver), Firewall, and DHCP logs, and most of them didn't have any entries in the last 10 minutes. The only thing I saw that might be relevant is a repeated "Default deny rule IPv4 (1000000103)" for a connection from my PC to my Proxmox machine - roughly 1 entry per second for 15 seconds.
I'm trying to think of anything that might have changed between everything working and this problem arising. The only thing I can think is maybe rebooting devices or disconnecting/reconnecting various ethernet cables, but I don't see why that would have caused an issue to pop up.
One other thing that might be relevant is that I use a VPN for my work, but I've been using that on and off the entire time I've had pfSense set up, and I've never encountered issues once I disable the VPN.
-
Can we see a screenshot of the firewall logs showing that blocked traffic?
-
Sure thing!
-
Ok it looks like you might have a bad subnet mask somewhere, probably on you client at 192.168.1.10. Traffic from there to other clients in the LAN like 192.168.1.100 or 192.168.1.124 should not go though pfSense at all it should go directly to those hosts.
If that client has a small subnet mash that excludes those hosts the client will send traffic to pfSense which will forward it to the hosts but the hosts will reply directly to the client so pfSense doesn't see it. So it sees and passes a TCP SYN then it doesn't see the SYN-ACK packet then the client replies ACK and pfSense blocks it as out of state.Check the subnet mask on the client.
Steve
-
Awesome, thanks! I guess I really do need to educate myself on networking - I was thinking everything on the LAN went through the router.
ipconfig /all tells me my subnet mask is 255.255.255.0, which is correct I think (and includes both devices in question: Proxmox @ 192.168.1.100 and TrueNAS @ 192.168.1.5). Here's the output in case you can see something I can't.
However, I realized what might be causing this. I can't access my NAS while logged into the VPN, so last week I was attempting to add a static route on my PC to 192.168.1.5. (My work VPN disallows split tunneling.) I've deleted the static route (it wasn't working anyway), but I'm still seeing the issue. I'll reboot everything and report back. If it helps, here's the output of route print.
Edit: After rebooting everything, it appears to be fixed! Thanks so much for your help Steve, never would have figured it out without you.
-
Ah, nice! I guess there was some remnant of the static route you added somewhere. I don't see it in the table though. Weird.
Everything there looks good to me otherwise.Steve
