Assuming the "server" is located in your network, accessing the router by the public address requires hairpinning: a client's packet is first source NATed, then destination NATed by port forwarding/reverse NAT. That can be slow on some routers and it's generally inefficient.

When both client and server are located on the same private network, NAT is only a burden. A better solution is to access the server by its private address. The easiest way to do this is to setup split-brain DNS - on your DNS server, you set up an A record with the server FQDN, pointing to the private address. That way, clients in your local network use the private IP address while anyone outside uses the public IP.

Assuming the "server" is located in your network, accessing the router by the public address requires hairpinning: a client's packet is first source NATed, then destination NATed by port forwarding/reverse NAT. That can be slow on some routers, doesn't work at all on others, and it's generally very inefficient.

When both client and server are located on the same private network, NAT is only a burden. A better solution is to access the server by its private address directly. 

The easiest way to do that is to set up split-brain DNS: on your DNS server, you set up an A record with the server FQDN, pointing to the private address (instead of the public DNS which points to the public = the router's address). That way, clients in your local network use the private IP address while anyone outside uses the public IP.