In addition to what Ron and Ricky have mentioned (the recipient needing to know who to respond to and nodes along the way needing to know where to send any ICMP messages,) TCP connections are identified by the unique 4-tuple of source IP address, destination IP address, source port number, and destination port number. Without the source IP address, the receiver's TCP stack wouldn't know which stream a particular incoming packet belonged to. The same is true, of course, for UDP or any other transport-layer protocol that supports having multiple remote hosts sending packets to the same transport-layer protocol and transport-layer protocol stream identifier (e.g. port number.)
Consider, for example, a web server. It listens for incoming packets on the server's TCP port 80. However, all of the clients connected to that server will be sending their packets to port 80 on the IP address of the web server. Additionally, there's no guarantee that the source port number will be different between clients on different computers. As such, without the source IP address being in the packet, the TCP stack doesn't know which of the connections to port 80 any given incoming packet belongs to and, thus, doesn't know which stream (socket) its payload should be delivered to. Additionally, it wouldn't know which receive window, etc. to use, since it wouldn't be able to uniquely identify which connection the packet belongs to.
Also, while this is not the reason for the design, an additional place where this becomes useful is in NATs. NATs use a 5-tuple to uniquely identify which connection incoming packets belong to, source IP address, source port, destination IP address, destination port, and transport-layer protocol (e.g. TCP or UDP.) Again, without all of that information, the NAT would be unable to uniquely identify the stream and, thus, wouldn't know which internal address/port it should forward the packet to.