SSH broken pipe / upload issues
I'm running a 3 node Proxmox (KVM) cluster with 2 virtualized pfSense 2.3.4 (amd64, virtio, no VLAN for the primary NIC).
After upgrading from 2.3.2 to 2.3.4 we're experiencing strange SSH/SFTP behaviours when connecting to any of our virtual Debian servers via SSH/SFTP on the internal IP's through an OpenVPN Access Server (virtualized, no VLAN):
- No upload (32kb file size max)
- Timeouts / disconnected out of the sudden ("Broken pipe").
Virtual machines beeing connected to:
- Debian 8, all latest updates
- all on different subnets and VLANs (via Proxmox VLAN on the virtio NIC)
- Each pfSense has virtual virtio NICs allowing routing to the different subnets/VLANs.
The problem only exists when connecting via the VPN. The VPN server (OpenVPN AS) runs in subnet 192.168.100.0/24, so the routing goes trough firewall1 (192.168.100.253) via its NIC 10.20.0.2 to the corresponding clients in the subnet via firewall2 (10.20.0.253). Connections on the servers SSH daemon always come from 10.20.0.2.
SSH/SFTP works just fine when connecting to each of the servers through a public IP from the outside (no timeouts, no upload problem).
I'm stuck here. Actually we have not done any changes to the configuration of the pfSense setups at all after or before the upgrades.
I smell some routing problems as it only happens internally but where should I start looking?
Any ideas where to start searching?
OK, just for fun I set up another OpenVPN server with the primary NIC inside the same subnet / VLAN. No problems here. So I guess it really is just a routing problem. I just don't understand why this came with the upgrade.
Not working: vpnserver1 vtnet0 subnet 192.168.100.0 (no vlan) <=> firewall1 vtnet0 subnet 192.168.100.0 (no vlan) <=> firewall1 vtnet1 subnet 10.20.0.0 (vlan 4) <=> firewall2 vtnet1 subnet 10.20.0.0 (vlan 4) <=> server vm subnet 10.20.0.0 (vlan 4)
Working: vpnserver 2 vtnet0 subnet 10.20.0.0 (vlan 4) <=> server vm subnet 10.20.0.0 (vlan 4)
I went back to a 2.3.2 snapshot and - tadaa - working again out of the box with the same config.
Anything I need to know that changed in 2.3.4?!?
Sorry for all the spamming. I found the culprit: pfBlockerNg.
It was installed, but disabled, on firewall2. I enabled it, selected all interfaces (because if would nag me otherwise), then disabled it again. Then uninstalled it completely and reinstalled it. Voila, no more problems.