As long as nothing else uses that port, you can tie destion IP or Range/Netblock with that port.
Kind of problematic if they use say 443 or 80 or any other port that some other site/service will use. So you need to use enough variables to only route that specific traffic and not traffic you don't want to go out that gateway.
Why source IP is used - is it simple that if that changes its completely under your control. Problem with destination IP is most stuff is served via some sort of CDN these days, and IPs used could be in the 1000's or 10's of thousands - and they can change all the time.
Same with port, they are not always unique to whatever site/service you would want to route out a specific gateway.
While you know your source IP is what you set it to be, and won't be changing unless you change it.
But any combination you can come up with that makes the traffic unique enough to identify can be used.