Assuming the "server" is located in your network, accessing the router by the public address requires hairpinning: a client's packet is first source NATed, then destination NATed by port forwarding/reverse NAT. That can be slow on some routers and it's generally inefficient.
When both client and server are located on the same private network, NAT is only a burden. A better solution is to access the server by its private address. The easiest way to do this is to setup split-brain DNS - on your DNS server, you set up an A record with the server FQDN, pointing to the private address. That way, clients in your local network use the private IP address while anyone outside uses the public IP.