Why can't we say router works both at Layer 2 and Layer 3?
This is implied. Basically, when we say that a device works on layer N, we actually means that the device implements all layers up to including layer N of the model, and its main functionality is in layer N.
The logic is as follows. The role of a device, that interacts with a network, is to send "data" outside of the device (i.e., "data" will eventually ends on the cable/air) and receive "data" from the network (i.e., receive data from the cable/air).
Since the device is "obviously" communicating with "the outside", it is implied that the device needs whole stack, starting from physical layer, then data link layer, and so on. For "textbook" routers this ends in layer 3 (at least for the data plane). For "textbook" hosts it ends with layer 4 inside OS and in application layer outside.
To put it in the opposite direction. Layer N cannot send (or receive) data without Layer N-1. For router as an example, layer 3 cannot send data without layer 2, layer 2 cannot send data without layer 1. And since it is clear that the data needs to be eventually sent, it is obvious that the presence of layer 3 implies presence of layers 2 and 1.