Client send TCP packet to server to a closed port， ie 80. In which layer will the sync packet be discarded? I guess for Client, the request will timeout, for server it should discard the request above tcp layer.
TCP, itself, will send a RST.
This is explained in RFC 793, Transmission Control Protocol:
If the connection does not exist (CLOSED) then a reset is sent in response to any incoming segment except another reset. In particular, SYNs addressed to a non-existent connection are rejected by this means.
If the incoming segment has an ACK field, the reset takes its sequence number from the ACK field of the segment, otherwise the reset has sequence number zero and the ACK field is set to the sum of the sequence number and segment length of the incoming segment. The connection remains in the CLOSED state.
By the way, TCP does not know anything about clients or servers. TCP creates connections between peers. The client/server concept is for applications, which are off-topic here.
If the port is closed since no listener is on this port at the server the OS kernel will not be able to find a listening socket for the arriving packet and will discard it. Given that the port is involved to determine if something is listening but no application data are transferred yet (initial SYN from client) the rejection is done at the transport layer (TCP).
If the initial SYN gets discarded or rejected by the server the initial TCP handshake will fail on the client side, i.e. no TCP connection will be established. Therefore no application data will be sent and the failure is propagated at the transport layer too, i.e. the connect will fail.