Weirdest Issue Ever? - Experts Needed! SMB Hangs over Specific File
-
Hi all,
For the sake of not repeating details, but I have an on-going discussion about my issue HERE:
This is still a big problem for me and would very much appreciate someone telling me what the fix is here.
Many thanks,
James -
How big is the file?
Can you pcap the the transfer and see what happens when it fails?
Steve
-
@jamesobz definitely "on-access" file scanner, probably a memory resident antivirus or antimalware false+ detection
-
@stephenw10 - Thanks for your reply. I have uploaded a packet capture which can been seen HERE - Hopefully something stands out here to you.
This is the entire capture from right clicking the file on the server where the file is being grabbed from on 10.47.10.10 to pasting it onto 10.47.13.11 in a different VLAN.
The capture was stopped when explorer popped up with a message saying the network path cant be found.
-
@papdee - there is no AV or Malware tools on the server. Just the built-in Windows Defender which has been turned off when I have been testing this.
-
You are 10.47.13.11 pulling file from 10.47.10.10 here I assume?
And the file is ABCpdf10-32.dll.deploy?
Where did you run the capture? Was it filtered?
It looks like there are packets going missing and eventually the connection is reset. Repeatedly.
-
@stephenw10 - Yes, that is correct. The packet has only been filtered to remove another IP that had an RDP session onto 10.47.13.11 to reduce the noise, so all you are seeing is data between the 2 IP addresses we are interested in. Other than that, those logs have not been played with in any way. It was run on the interface that the .13.11 address was on.
Yea, it's timing out and retrying until it doesn't it seems At the moment i can only put this down to 'something' the pfsense hardware/version is doing which is a lousy guess, but as i know it works fine between addresses on the same VLAN that bypasses the pfsense that's my conclusion so far.
Really no idea what to change in order to fix this and i cant swap the firewall out to test another.
-
Ok, I would run pcaps in other locations and compare. There is probably some packet that gets sent from AWS that never arrives at the client. So where is that going missing. In the tunnel? In AWS? pcaps at other places in the path may show that.
Steve
-
@stephenw10 - When i have been testing to narrow this done i have been avoiding any connections to AWS or using IPSEC VPN tunnels.
This issue still exists communicating between 2 VLANs in the same office. The only hardware that traffic passes is an Aruba 6100 switch and the Netgate SG-2100 firewall.
If i change the addresses so traffic goes from a different VLAN in the same office going out a different interface, i get the same results.
-
@jamesobz Jumping into this. When you test within the same VLAN does the problem still occur?
-
@michmoor Please do. No, the issue does not occur on the same VLAN which is why i'm thinking this is some configuration that needs amending pfsense side. I'm open to all suggestions at this point.
-
@jamesobz said in Weirdest Issue Ever? - Experts Needed! SMB Hangs over Specific File:
o, the issue does not occur on the same VLAN whi
Let's turn the pfsense into a router only just to make sure packet filtering weirdness arent in play assuming you haven't done so
System - Advanced- Firewall & NAT - Advanced Options - Disable Firewall
Click save at the bottom of the page.
Try your test again. -
@jamesobz said in Weirdest Issue Ever? - Experts Needed! SMB Hangs over Specific File:
@stephenw10 - When i have been testing to narrow this done i have been avoiding any connections to AWS or using IPSEC VPN tunnels.
This issue still exists communicating between 2 VLANs in the same office. The only hardware that traffic passes is an Aruba 6100 switch and the Netgate SG-2100 firewall.
If i change the addresses so traffic goes from a different VLAN in the same office going out a different interface, i get the same results.
The SG-2100 does not have the hottest/fastest CPU in the world, and it cannot handle Gigabit wirespeed when doing filtering - especially not i if both VLAN’s involved are on the built-in switch in SG-2100, which connects to the SOC with one NIC.
My bet is you are loosing packets by the thousands because of queue congestion. I happen to have almost the same setup, and the Aruba CX-6100 switch - like most other switches - has flow control disabled on ports by default. Try and enable flow control on the ports that uplinks to the SG-2100 and to the client and NAS involved. Remember to enable flow control on them as well - otherwise it will not have the required effect.
I seem to remember flow control is enabled by default in the SG-2100, but I might be mistaken. -
Mmm, we are assuming you are using the switched (LAN 1-4) ports on the 2100 which seems like it could be significant. Are you able to test using VLANs on the WAN (mvneta0) port?
Or maybe between the WAN and LAN directly without any VLANs?
Steve
-
@keyser said in Weirdest Issue Ever? - Experts Needed! SMB Hangs over Specific File:
The SG-2100 does not have the hottest/fastest CPU in the world, and it cannot handle Gigabit wirespeed when doing filtering - especially not i if both VLAN’s involved are on the built-in switch in SG-2100, which connects to the SOC with one NIC.
My bet is you are loosing packets by the thousands because of queue congestion. I happen to have almost the same setup, and the Aruba CX-6100 switch - like most other switches - has flow control disabled on ports by default. Try and enable flow control on the ports that uplinks to the SG-2100 and to the client and NAS involved. Remember to enable flow control on them as well - otherwise it will not have the required effect.
I seem to remember flow control is enabled by default in the SG-2100, but I might be mistaken.@keyser - Thanks for your input. The level of traffic on this network is only a couple of users currently so very little until this gets resolved. I logged onto the 6100 switch and can see from the interface that there are only 16 dropped packets out of 61 million on my trunk port all traffic goes through.
However, i followed up on your suggestion and have enabled Flow Control on this uplink port, and also on the VMNIC port traffic is coming from as i am testing this between 2 virtual machines on my host. I am unsure where Flow Control is set within PfSense, but I have also read that it looks like this is on by default.
To confirm, having just enabled flow control on this switch ports has had no effect.
-
@michmoor said in Weirdest Issue Ever? - Experts Needed! SMB Hangs over Specific File:
@jamesobz said in Weirdest Issue Ever? - Experts Needed! SMB Hangs over Specific File:
o, the issue does not occur on the same VLAN whi
Let's turn the pfsense into a router only just to make sure packet filtering weirdness arent in play assuming you haven't done so
System - Advanced- Firewall & NAT - Advanced Options - Disable Firewall
Click save at the bottom of the page.
Try your test again.@michmoor - I have tried this and with it disabled i still cannot transfer the file successfully. Gets stuck on 60% each time as before.
-
@stephenw10 said in Weirdest Issue Ever? - Experts Needed! SMB Hangs over Specific File:
Mmm, we are assuming you are using the switched (LAN 1-4) ports on the 2100 which seems like it could be significant. Are you able to test using VLANs on the WAN (mvneta0) port?
Or maybe between the WAN and LAN directly without any VLANs?
Steve
Hi Steve, Yes i am using the default config in terms of switch setup on the SG-2100. Here are a couple of pics.
How do you mean test using VLANs on the WAN port? Are you suggesting that i host this file externally and then try and connect to that external server via SMB to see if it can be pulled down that way so the traffic goes via the WAN interface instead of an internal one?
-
The VLAN interfaces you have defined there are on mvneta1?
The switch in the 2100 is still in port-based vlan mode so all those VLAN would be available on all 4 VLAN ports. Which should be fine but is less common. Most users would put the 2100 switch in .1q mode and trunk VLANs to the required ports.
Either you are testing through the switch.
If you created VLANs on mvneta 1 and connected that to the 6100 switch that would rule out the on-board switch as a potential issue.
It seems unlikely to be an issue though.Steve
-
@stephenw10 said in Weirdest Issue Ever? - Experts Needed! SMB Hangs over Specific File:
The VLAN interfaces you have defined there are on mvneta1?
The switch in the 2100 is still in port-based vlan mode so all those VLAN would be available on all 4 VLAN ports. Which should be fine but is less common. Most users would put the 2100 switch in .1q mode and trunk VLANs to the required ports.
Either you are testing through the switch.
If you created VLANs on mvneta 1 and connected that to the 6100 switch that would rule out the on-board switch as a potential issue.
It seems unlikely to be an issue though.Steve
The VLAN interfaces are all on mvneta1, yes. I had tried to set this up initially using .1q but could not seem to get it working the way i needed. Having reached out I had a separate post a while back on that HERE for context and was advised to configure it the way it is now which works.
'If you created VLANs on mvneta 1 and connected that to the 6100 switch that would rule out the on-board switch as a potential issue.'
This is exactly how i have it configured. Lan Port 1 on the firewall going into the 6100 switch port 48 which is trunked. From there there is also a trunk port also on port 47 which goes to the host with the VMs on it i have been testing on. -
Sorry I meant on mvneta0. So as to not be passing traffic through the 2100s switch.
When I initially thought it could be a potential problem I had assumed you were using .1q mode. It seems far less likely in port vlan mode. But we are into the realms of the unlikely!
Steve