In TCP header, what happens when both SYN and FIN flags are set to 1? Or, can both even be simultaneously set to 1 ?
In normal TCP behavior, they should never both be set to 1 (on) in the same packet. There are many tools that exist that let you craft TCP packets, and the typical response to a packet with SYN and FIN bits set to one is a RST, since you are violating the rules of TCP.
The response depends on the type of Operating System.
The combination of SYN and FIN flag being set in TCP header is illegal and it belongs to the category of illegal/abnormal flag combination because it calls for both establishment of connection(via SYN) and termination of connection(via FIN).
The method to handle such illegal/abnormal flag combinations is not conveyed in the RFC of TCP. So, such illegal/abnormal flag combinations are handled differently in various operating systems. Different operating system also generate different kind of responses for such packets.
This is a very big concern for security community because attackers shall exploit these response packets to determine the type of OS on the target system to craft his attack. So, such flag combinations are always treated as malicious and modern intrusion detection systems detect such combinations to avoid attacks.