If this is really happening on L3 and not on L2 somewhere, a rule that blocks all traffic from that interface to 10.0.4.10/24 should help, I believe. Should be placed before the "allow to *" rule you probably have for internet access. I think that's what you also thinking of, too?
I do not share your concerns, because 1. everything that happens between your clients and the Netgate isn't affected by any firewall rule on the Netgate and 2. this doesn't disturb routing (you do not "access" the Netgate for this), because neither the Netgear nor the Netgate is the destination on IP level. I usually have a rule that just blocks traffic to all private network ranges (including the own range, because I don't whitelist it) on all interfaces of the Netgate and only allow specific traffic, e.g. DNS to the Netgate. Just make sure that you don't lock yourself out of the Netgate (allow HTTPS from your "management network", but usually there is the "Anti-Lockout Rule" that does that for you). ;)