I will ask the question based on the following scenario. Assume I have to send 10 packets using TCP using a window size of 5. I send 1, 2 and receive ACK for them, I send 3 and the ACK is lost, I send 4, 5 and receive their ACK. My question is: having SACK negotiated, does TCP continue transmitting outside the window? I.e. It starts transmitting 6 before receiving 3? That in the case that the window doesn't shift
no, SACK will not move the beginning of the sender window (also called SND.UNA)
Let's consider that the receiver has receiver buffer of size 5. If the receiver has received packets 4 and 5 but not 3, his receiver buffer will look like: [space for 3, packet 4, packet 5, space for 6 and 7]. He can't deliver 4 and 5 to application and can't move its window. Thus there is no point for the sender to move its window and send anything past 6 and 7 (see note 1 below).
when you receive an ACK for one, the window will shift, so 6 can be sent; the same applies to ACK for two and 7.
if you send the packet three, it gets delivered but the ACK is lost, then when you send the packet 4, TCP will ACK 4 (normal cumulative ACK) indicating that all packets up to 4 are delivered. You don't need SACK for this.
having a TCP with constant window is a very simplified situation that never happens. TCP will employ congestion control, which will change the size of window. In particular TCP Reno and NewReno will utilize fast recovery which can change the size of the window past 5. It can use the SACK fields to determine how to change the size of the window. (of course provided that the receiver window is not exceeded)