I was reading about TCP and came across the topic of congestion control wherein the TCP sender is responsible for obeying flow control and congestion control messages to prevent congestion in the network. What will happen if the sender decides not to listen to the congestion control messages and continues sending packets at a high rate anyway? Is there some way to protect the network from such devices as it seems like a security threat? I could not find the answer to this question in the text book and google wasn't of much help either so decided to ask here.
Congestion control enforced by TCP clients is not there to protect the network, it's simply to try determine the maximum network resource that is reliably available for that specific client at that point in time.
As TCP is a reliable protocol (acknowledging each packet) any packet loss (for example caused by congestion) will result in the TCP connection slowing down to perform re-transmissions, congestion control tries to limit these retransmissions to speed up the connection.
Congestion will always happen on your network, as a networking engineer you should always assume that clients are not trusted and therefore you need tools within the (trusted) network that will protect you from their traffic.
Look into tools like Quality of Service, Scheduling and Queues. Simply put you should be enforcing polices on your network so high priority traffic like network control OSPF traffic should not have the same level of service as someone downloading a video from YouTube, that way client congestion will not affect other services on your network.
What will happen if the sender decides not to listen to the congestion control messages and continues sending packets at a high rate anyway?
I would argue that a sender refusing to abide by congestion control algorithms is attempting a DoS attack. To me, it's akin to saying "I don't care if you're getting too much data, out of order, and with missing data. I'm just going to send it anyways, even if you don't accept."
If you get the time, RFC 5681 isn't that bad of a read. Especially if you need something to help you sleep.
The good thing is that a sender that falls into this category is going to be sending more than it's receiving, meaning you can drop traffic before it reaches your border without much effort. This is also easy to detect if you're monitoring your hosts bandwidth usage. If you were the receiver, however, you might have a little more of a difficult time. This is something webmasters constantly battle with.
Is there some way to protect the network from such devices as it seems like a security threat?
I'll reiterate what I said above, a host attempting something like that should be considered compromised. That is abnormal host behavior. QoS will help to minimize the internal impact, but having control of your hosts is even more vital (not to mention CYA when your ISP comes complaining).
To add to Ryan Foley's and iTom's great answers, I just wanted to mention a quick note from a non malicious or compromised host perspective.
In TCP, any packet that didn't make it through must be re-transmitted. If I, as the sender, decide to ignore the Congestion indication from the Receiver, and continue to send traffic as fast as I can, I am only shooting myself in the foot because sooner or later the Receiver is going to request re-transmission and I'm going to have to do twice the work to send the same number of packets.
This is why you don't typically see senders ignoring congestion indications, because its beneficial to both parties (sender and receiver) to obey the congestion control rules and slow down their transmission. (Again, this is only relevant in a non-malicious scenario -- see the other answers for the proper response to this happening under dubious circumstances)
Another way of looking at it: If you're trying to fill a teacup with water using a fire-hose, its going to take considerably more effort and time then if you simply used a regular faucet.