So, I do have some answers for you. Yes, MTU is affecting ping and tracepath, but that may not be the problem here…though, pmtu may be to blame. The best thing I can suggest is to try and run a packet capture on each end and on one firewall or the other to see if the packets are needing to be fragmented and are not getting fragmented. If the DF flag is set, then it could be dropping packets once it hits the firewall if the mtu is not properly set.
Your ss shows that mtu of 1198 works to ping over the tunnel. What I recommend you do to verify this is to use something like this:
ping -c 2 -M do -s <size><ip address="">...the above is a linux command, but it should show that the message is too long until you drop it down to 1198.
The second thing to do is check that pmtud is running/enabled. The next step is to make sure that ICMP type 11 can make it through each end of the VPN tunnel as that will show "destination unreachable" and is what is necessary for PMTUD to work.
Of course, the quickest and easiest way to see if MTU is the problem is to drop the MTU on your Synology interfaces down to 1198 on the interface itself (generally can be achieved with ifconfig <devicename>mtu 1198 run as root/sudo).
If you drop it down to 1198 and can ping without using the -s flag to drop the message length and rsync STILL isn't working...there may be other factors at play.
Also, something to consider is the methodology that rsync is using to connect to the remote host...this generally is done over the rsync port IF there are no credentials for ssh access...rsync is PREFERRED to be done over ssh, however, because it is more secure than using the unsecured rsync port (I do realize that you're doing this over VPN, but it's still more secure to do rsync over ssh).
In another note that I remembered based on your statement "Logging into the GUI of the first Synology, and mounting the second Synology to the local filesystem and consequently copying a large movie (5 gig) through the VPN tunnel works fine. The other way around works fine too."
...suggests that general connectivity is fine between the two devices...this likely means that there is a firewall in the way or something to that effect for rsync specifically.
I would also like to address why you were getting a "too many hops message"...that was due to the devices in between each point not responding to the ICMP type 11 time exceeded that should be received on traceroute and tracepath messages...you should look at how those work to get a better understanding of the tool. The basic function is that a packet is sent out with ttl=1. Once it gets to the next hop, it decrements the ttl and checks to see if the ttl has expired, if expired, it sends back ICMP type 11, which lets the client know the next hop's information. It repeats this until it hits the final destination and checks for "Destination port unreachable" ICMP message to be sent back. It does this (by default) over UDP in Linux/Unix.
So, the reason you got the messages COULD be for a few reasons, but it looks like none of the devices responded to the ICMP messages. This very well could be due to MTU, but tracepath will drop the MTU automatically...the more likely scenario is that these devices are not responding to ICMP type 11 messages.
...hope some of this is helpful.</devicename></ip></size>
