Setting an IP address as a default gateway provides two things:
- allows the host to "find" (via ARP) the appropriate MAC address for L2 encapsulation
- informs the host which interface to use to find the gateway (via Routing table)
Imagine a host has three network connections, each with unique IP address space. Typically, only one of those network connections will have the intended default gateway.
If you specify the gateway's MAC address, then the host doesn't know which interface to use to reach the gateway. That MAC address could exist on all three of the interfaces.
If you specify the gateway's IP address, then the host can use it's local routing table to determine where the gateway is. And then can use ARP to find the actual MAC address.
Technically, if you could specify both the Interface and the MAC address while setting the default gateway, you then (in theory) would not need an IP address on your default gateway.
But the key being, the MAC address by itself does not provide any indication of which interface to use.
While not strictly related to your question, it's worth noting: This is why when you set a default gateway using an IPv6 Link-Local address, you must also indicate which interface to use. Because the link-local scope exists on each interface, the host can not use a routing table to determine exactly which interface the intended gateway exists within.