Weirdest Issue Ever? - Experts Needed! SMB Hangs over Specific File
-
@stephenw10 - When i have been testing to narrow this done i have been avoiding any connections to AWS or using IPSEC VPN tunnels.
This issue still exists communicating between 2 VLANs in the same office. The only hardware that traffic passes is an Aruba 6100 switch and the Netgate SG-2100 firewall.
If i change the addresses so traffic goes from a different VLAN in the same office going out a different interface, i get the same results.
-
@jamesobz Jumping into this. When you test within the same VLAN does the problem still occur?
-
@michmoor Please do. No, the issue does not occur on the same VLAN which is why i'm thinking this is some configuration that needs amending pfsense side. I'm open to all suggestions at this point.
-
@jamesobz said in Weirdest Issue Ever? - Experts Needed! SMB Hangs over Specific File:
o, the issue does not occur on the same VLAN whi
Let's turn the pfsense into a router only just to make sure packet filtering weirdness arent in play assuming you haven't done so
System - Advanced- Firewall & NAT - Advanced Options - Disable Firewall
Click save at the bottom of the page.
Try your test again. -
@jamesobz said in Weirdest Issue Ever? - Experts Needed! SMB Hangs over Specific File:
@stephenw10 - When i have been testing to narrow this done i have been avoiding any connections to AWS or using IPSEC VPN tunnels.
This issue still exists communicating between 2 VLANs in the same office. The only hardware that traffic passes is an Aruba 6100 switch and the Netgate SG-2100 firewall.
If i change the addresses so traffic goes from a different VLAN in the same office going out a different interface, i get the same results.
The SG-2100 does not have the hottest/fastest CPU in the world, and it cannot handle Gigabit wirespeed when doing filtering - especially not i if both VLAN’s involved are on the built-in switch in SG-2100, which connects to the SOC with one NIC.
My bet is you are loosing packets by the thousands because of queue congestion. I happen to have almost the same setup, and the Aruba CX-6100 switch - like most other switches - has flow control disabled on ports by default. Try and enable flow control on the ports that uplinks to the SG-2100 and to the client and NAS involved. Remember to enable flow control on them as well - otherwise it will not have the required effect.
I seem to remember flow control is enabled by default in the SG-2100, but I might be mistaken. -
Mmm, we are assuming you are using the switched (LAN 1-4) ports on the 2100 which seems like it could be significant. Are you able to test using VLANs on the WAN (mvneta0) port?
Or maybe between the WAN and LAN directly without any VLANs?
Steve
-
@keyser said in Weirdest Issue Ever? - Experts Needed! SMB Hangs over Specific File:
The SG-2100 does not have the hottest/fastest CPU in the world, and it cannot handle Gigabit wirespeed when doing filtering - especially not i if both VLAN’s involved are on the built-in switch in SG-2100, which connects to the SOC with one NIC.
My bet is you are loosing packets by the thousands because of queue congestion. I happen to have almost the same setup, and the Aruba CX-6100 switch - like most other switches - has flow control disabled on ports by default. Try and enable flow control on the ports that uplinks to the SG-2100 and to the client and NAS involved. Remember to enable flow control on them as well - otherwise it will not have the required effect.
I seem to remember flow control is enabled by default in the SG-2100, but I might be mistaken.@keyser - Thanks for your input. The level of traffic on this network is only a couple of users currently so very little until this gets resolved. I logged onto the 6100 switch and can see from the interface that there are only 16 dropped packets out of 61 million on my trunk port all traffic goes through.
However, i followed up on your suggestion and have enabled Flow Control on this uplink port, and also on the VMNIC port traffic is coming from as i am testing this between 2 virtual machines on my host. I am unsure where Flow Control is set within PfSense, but I have also read that it looks like this is on by default.
To confirm, having just enabled flow control on this switch ports has had no effect.
-
@michmoor said in Weirdest Issue Ever? - Experts Needed! SMB Hangs over Specific File:
@jamesobz said in Weirdest Issue Ever? - Experts Needed! SMB Hangs over Specific File:
o, the issue does not occur on the same VLAN whi
Let's turn the pfsense into a router only just to make sure packet filtering weirdness arent in play assuming you haven't done so
System - Advanced- Firewall & NAT - Advanced Options - Disable Firewall
Click save at the bottom of the page.
Try your test again.@michmoor - I have tried this and with it disabled i still cannot transfer the file successfully. Gets stuck on 60% each time as before.
-
@stephenw10 said in Weirdest Issue Ever? - Experts Needed! SMB Hangs over Specific File:
Mmm, we are assuming you are using the switched (LAN 1-4) ports on the 2100 which seems like it could be significant. Are you able to test using VLANs on the WAN (mvneta0) port?
Or maybe between the WAN and LAN directly without any VLANs?
Steve
Hi Steve, Yes i am using the default config in terms of switch setup on the SG-2100. Here are a couple of pics.
How do you mean test using VLANs on the WAN port? Are you suggesting that i host this file externally and then try and connect to that external server via SMB to see if it can be pulled down that way so the traffic goes via the WAN interface instead of an internal one?
-
The VLAN interfaces you have defined there are on mvneta1?
The switch in the 2100 is still in port-based vlan mode so all those VLAN would be available on all 4 VLAN ports. Which should be fine but is less common. Most users would put the 2100 switch in .1q mode and trunk VLANs to the required ports.
Either you are testing through the switch.
If you created VLANs on mvneta 1 and connected that to the 6100 switch that would rule out the on-board switch as a potential issue.
It seems unlikely to be an issue though.Steve
-
@stephenw10 said in Weirdest Issue Ever? - Experts Needed! SMB Hangs over Specific File:
The VLAN interfaces you have defined there are on mvneta1?
The switch in the 2100 is still in port-based vlan mode so all those VLAN would be available on all 4 VLAN ports. Which should be fine but is less common. Most users would put the 2100 switch in .1q mode and trunk VLANs to the required ports.
Either you are testing through the switch.
If you created VLANs on mvneta 1 and connected that to the 6100 switch that would rule out the on-board switch as a potential issue.
It seems unlikely to be an issue though.Steve
The VLAN interfaces are all on mvneta1, yes. I had tried to set this up initially using .1q but could not seem to get it working the way i needed. Having reached out I had a separate post a while back on that HERE for context and was advised to configure it the way it is now which works.
'If you created VLANs on mvneta 1 and connected that to the 6100 switch that would rule out the on-board switch as a potential issue.'
This is exactly how i have it configured. Lan Port 1 on the firewall going into the 6100 switch port 48 which is trunked. From there there is also a trunk port also on port 47 which goes to the host with the VMs on it i have been testing on. -
Sorry I meant on mvneta0. So as to not be passing traffic through the 2100s switch.
When I initially thought it could be a potential problem I had assumed you were using .1q mode. It seems far less likely in port vlan mode. But we are into the realms of the unlikely!
Steve
-
@stephenw10 said in Weirdest Issue Ever? - Experts Needed! SMB Hangs over Specific File:
Sorry I meant on mvneta0. So as to not be passing traffic through the 2100s switch.
When I initially thought it could be a potential problem I had assumed you were using .1q mode. It seems far less likely in port vlan mode. But we are into the realms of the unlikely!
Steve
I Concur with Steve, if you can test it using mvneta0 as the uplink port, then we at least could eliminate the built-in 4 port switch, and we would have proper interface counters (NIC only) to see if a queue is flooding and dropping packets in the proces.
-
@jamesobz I think we’re getting close to just swapping our the hardware (firewall) and see if problem follows.
-
Mmm, I thought that had in fact been tried but unless it was buried somewhere in the reddit comments I don't see it now.
Edit: The files passes other pfSense installs fine. Though possible not the same config.
-
So what is the size of this file? 1MB, 1GB, 10GB, 1TB?
Talk of queue issues, etc.. If that was the case I would assume you would run into this issue whenever you copy same sort of large file, etc..
-
The file is not especially large. Like of the order of 10MB.
It really does seem to be the file since in the original use case the rest of the folder can be passed fine without it.
