I am new to networking so this probably a naive question. I've been reading up on the spanning tree protocol and understand why it's necessary in a network (it prevents loops) and I understand that this is not the most optimal way to route traffic in some cases. But the question is: why do we not find and use the optimal route from network to network? It seems fairly easy to do, even algorithmically, and it would decrease the number of bridges that traffic needs to go through. Is it too much computation per packet? Is it because we don't store state of the previous bridge - and why don't we?
You are confusing layer-2 and layer-3. Routing happens at layer-3, and bridging happens at layer-2.
Routing routes packets between networks, and it usually finds the best path between networks. Layer-3 packets also have a TTL that gets decremented as a router processes the packets, and the packet is discarded when the TTL hits
Bridging bridges frames on the same network, and the path for the frames may be suboptimal because of possible loops. Remember that there will be broadcasts at layer-2, and the broadcasts are sent to every interface (there is no optimal path for a broadcast as there is no single endpoint), and that is where you can get loops. If every interfaces forwards broadcast frames, then there is no stopping the broadcasts, and they will loop forever (a broadcast storm) crashing the network. Frames do not have a TTL, and bridges are transparent devices so they do not process the frames to even be able to decrement a TTL.