This is my own understanding regarding Ports:
The Ports are defined in the 5th layer of the OSI model which establishes and controls the dialogue between the source and destination. Port definition is done in the diaogue establishment process.
In the TCP/IP model, Layers 5,6 and 7 of the OSI model are included into one Layer (Layer 4 of the TCP/IP model).
So to answer your question, it all depends on the model you are considering. if you are using the OSI model, "Port" is a layer 5 definition, otherwise if you are using the TCP/IP model it will be considered as a layer 4 entity.
P.S.: for a more specific and maybe more correct answer you should take a look at the OSI model basic definitions in the ISO/IEC 7498-1 (Link Here)
Cheers,