You need to understand the network layers. They are independent of each other. Ethernet and IPv4 are currently the dominant protocols, but that was not always the case, and IPv4 is being replaced with IPv6, which has very different addressing. Layer-2 protocols can carry many different layer-3 protocols, and layer-3 protocols don't care which layer-2 protocol is used. You could ask the same thing of layer-4. Some layer-4 protocols have addresses. In TCP and UDP, these are called ports.
LANs are layer-2 broadcast domains. There are multiple layer-2 protocols, some use 48- bit MAC addresses, some use 64-bit MAC addresses, and some do something else altogether.
LANs can carry a variety of layer-3 protocols, IPv4 is only one of those. If the layer-2 devices need to be specific to a particular layer-3 protocol, it becomes a problem. If you had switches which used IPv4, what happens when you need to add IPv6 devices?
In the past, IPX was the layer-3 protocol of choice for LANs. If switches were designed for IPX, IPv4 may not have come to the fore for the layer-3 protocol. That would have caused a complete overhaul of LAN devices. Since a layer-2 protocol can carry any number of layer-3 protocols, even at the same time, IPv4 could use the same equipment. That is happening now as IPv6 is being rolled out. You can run both IPv4 and IPv6 with the same LAN equipment.