Cannot copy files via static route

silverado

I have a very strange problem. I cannot copy files via static routes between different subnets.

My main network is 192.168.1.0/24 with a pfsense 2.1 router (internet acces via PPPoE on the WAN-port, LAN-Address 192.168.1.1).
Everything is working fine within this network.

Now I have added a second router to this network. It is connected to the existing net via it's WAN-port using the ip 192.168.1.254.
To the LAN-port of the second router (ip 192.168.2.1) I have connected a second network 192.168.2.0/24.

Router 1 Router 2
INTERNET –-------- WAN LAN --------------------------WAN LAN
PPPoE 192.168.1.1 192.168.1.254 192.168.2.1

On Router 1

I have setup a gateway 192.168.2.254 and attached it to a route to 192.168.2.0/24.
I also activated the option "System: Advanced: Firewall and NAT: Static route filtering: Bypass firewall rules for traffic on the same interface"

Everythings seems to work fine.

All PCs within the 192.168.1.0/24 network I can reach all PCs within network 192.168.2.0/24 and vice versa. (Ping, SSH, RDP etc.)
All PCs within 192.168.2.0/24 can download or send files from/to PCs within 192.168.1.0/24.

BUT: Any PC within 192.168.1.0/24 cannot send or download files from any PC within 192.168.2.0/24. I can create or delete folders, edit small files etc. but when I try to copy any bigger file the connection times out / dies after a few bytes. Every method of file transfer is affected (Samba/SMB, SSH, FTP...) When I use Total Commander it seems to work sometimes. Files seem to be copied incredibly fast, but the resulting files (although they have the correct size) are corrupt/empty. The logfiles of my pfsense box do not show any errors.

For troubleshooting I have created a completely fresh test environment (different cables, PCs, Routers) and it behaves exactly the same way as my regular network.

What is going wrong here? What am I missing? Who can help me?

============================

Additional Information:

In my troubleshooting environment I have now replaced pfsense by monowall. Using exactly the same hardware and network settings file transfer is now working flawlessly into both directions.

So my settings seem to be okay and the file transfer problems seems to be caused by a bug in pfsense.

But I can't believe that this bug is not showing up for other users. I always thought that static routes are very basic and common. What is so exotic about my network that I'm running into problems nobody else seems to have? Or are there any special "hidden" settings that monowall does not require but pfsense does?

phil.davis

This looks like an asymmetric route issue - the returning packets from LAN 2 will be delivered by router 2 directly back to the LAN 1 client, without needing to go through pfSense. pfSense will drop the state it establishes when it does not see the other half of the flow after a short time, which is why you can do small things but not things that take a long time.
There are a few ways around it I can think of:
a) Add a pass rule on LAN with destination LAN2net and advanced option state type = sloppy state - pfSense should stop worrying about not seeing the reply packets coming back (maybe monowall allows something like this by default?); or
b) Make a manual NAT rule for traffic from LAN1net to LAN2net that NATs to LANaddress - then the systems in LAN2 will see all connections from LAN1net as coming from LAN1address and thus reply packets will go back to pfSense to be unNATted and delivered to their real destination. Of course that means LAN2 systems will not be able to know which LAN1 address a connection is coming from - it depends if you care.
c) If you have an extra NIC (or have a VLAN switch) then make an extra interface and separate subnet on pfSense to connect the other router, then everything really does route through pfSense and there is no asymmetric routing. (If you only have LAN2 to connect, then eliminate router2 altogether. Put LAN2 directly on pfSense.)

silverado

Thank you very much! Your answer was very helpful. Now I begin to understand what was going on in my network and was really starting to drive me crazy.

The firewall rule with the option "sloppy state" (a) didn't help but the manual NAT rule (b) seems to have solved the problem. Now I can transfer files without any issues. :)
That all traffic now seems to come from one IP is no problem for me. You just made my day!

Maybe your proposal "c" would be even better but my router is an Alix box with three build in NICs (which are already in use for WAN, LAN and DMZ) and no expansion slots. So I will have to stay with the NAT solution.

Many thanks again!