OpenVPN Oauth2 restrict vpn access

[Getbys]

Hello guys in our organization we use OAuth2 for authentication to our office via OpenVPN. People use google accounts to generate a token which than is used to authenticate and get access to the office. The OpenVPN client config is stored on a web server and people can access it and download if they know the URL. My question is how can I restrict people that have company google accounts to access the vpn ? Even if they have the client and the token still to have a restriction until granted access. I though with CSO but its kinda overkill to add every email and remove the routing table or etc. Maybe a way to stop token generation ? P.S We don't have AD in our env.