Let's say an attacker gets the sequence number between me and my client.
Sequence numbers should be initialized with good randomization.
Now he continues the conversation with me and has changed the IP address.
That should not be possible and if it is, the host is broken. Vulnerable hosts should never be connected/exposed to the Internet and should be replaced asap.
The sequence number must be interpreted specific to the socket in question (tuple of source IP/destination IP/source port/destination port). Different sockets must be able to use the sequence numbers independently.
Random sequence number initialization is still important since an attacker being able to guess port numbers and sequence numbers could spoof the source IP address and then insert data into an unencrypted connection or break it for a DoS attack.
How do I prevent packages from being sent to addresses that do not match the IP of the initial request/tcp message.
A stateful firewall is the usual measure to prevent things like that.