It's all typical subnet math, nothing special about it. The code for checking that is in util.inc, like check_subnets_overlap() but you might also look at things like gen_subnet() and get_subnet_mask() and the v4/v6 specific ones.
The question is very vague, though. We need more context to know what you're attempting to do that the current checks aren't allowing to know if it's valid.
While nothing stops something from using the network/broadcast addresses of a subnet (e.g. if it's routed to the firewall, all can be used for NAT so long as they aren't defined on an interface), you still can't force things to work that are not valid at a subnet level when used directly on an interface.
When using larger subnets, there is indeed nothing special about those addresses either. Such as when using a /23, it's valid to have clients use the .254 and .0 addresses in the middle of the /23.