Good questions.
1)The 1st and 2nd packets are transfer of SA proposals and cookies.
Question:Arent the cookies generated by hashing the senders IP, port ,
protocol and timestamp? If thats the case, does the receiver need the
algorithm used to calculate the hash so that he can confirm the data
or is it just a random value which the receiver does not have to be
bothered by.
The cookie's are arbitrarily generated (often randomly). They are 64 bits from the Initiator and 64 bits from the Responder, and collectively (the 128 bits) serve as the identification of a particular ISAKMP "tunnel" (or ISAKMP "SA") -- i.e., a particular set of the 3 ISAKMP keys (encryption, authentication, derivative).
(much like SPI's will identify a particular IPsec SA / tunnel)
The values are also used within the calculation of the 3 ISAKMP keys, but that doesn't happen until after message 3 and 4, so the Initiator and Responder will have already agreed upon the necessary hashing algorithms.
2)The 3rd and 4th packets exchange the Diffie Hellman public key and
nonces and a shared secret is generated to again calculate additional
3 keys.
Question: The Nonces are combined with psk to generate a seed value .
So how and when does PSK gets exchanged. Is it communicated outside of
these packet exchanges or are part of these packets.
The PSK is "exchanged" between the peers before the tunnel has ever tried to build. Theoretically via phone call or manually configuring the PSK on both sides.
Both parties are verified to have the proper PSK in Messages 5 and 6. What goes into creating the E/A/D ISAKMP keys is the PSK, so if both parties didn't start with the correct PSK, they could never create the same E/A/D keys, and would not be able to "understand" each other's message 5 and 6.
3)After the seed value is calculated, it is combined with the shared
secret to generate additional 3 keys to authenticate, encrypt and use
as a keying material to generate keys for Phase 2.
Question: These keys are used to authenticate and encrypt the packets
which are used in phase 2 by the algorithms(authentication and
encryption) negotiated in Phase 1(packets 1 and 2)right? And also to
encypt and authenticate the 5th and 6th packets of Phase 1.
Yes. After Messages 1-4 of Main Mode, three ISAKMP Keys exist: Encryption Key, Authentication Key, Derivative Key.
The Encryption Key and Authentication Key will secure Messages 5 and 6 of Main Mode, and any ensuing Quick Mode negotiation messages.
The Derivative Key is not used by ISAKMP, it is instead given to IPsec so IPsec can generate its own Encryption and Authentication Keys.
So what exactly are we transferring in packets 5th and 6th to authenticate and encrypt them?
What is exchanged is somewhat arbitrary, the why is what is important. The purpose of Messages 5/6 are to validate the other party is who they claim to be (aka, the other party has the correct pre-shared-key).
Here is (some of) what goes into calculating the value that is exchanged in message 5 and 6, and more importantly the purpose it serves:
- Pre-Shared-Key -- a value which can only be known by the intended recipients in the conversation
- Diffie-Hellman Shared Secret -- a value which can only be known by the two parties who exchanged messages 3/4
- Arbitrary Data -- values exchanged in Messages 1-4 which tie this particular authentication attempt to this particular conversation (preventing replying a previous connection's authentication attempts)
There are two parts to the Arbitrary Data, one set is from the Initiator and the other set is from the Responder. So in the end, you have two collective values exchanged in Messages 5 and 6:
- PSK / DH Shared Secret / Arbitrary Initiator Data
- PSK / DH Shared Secret / Arbitrary Responder Data
Both the Initiator and Responder have the information they need to put together both values above. Therefore, both Initiator and Responder have a value they can use to prove they are who they say they are, and a value they can use to verify the other is who they say they are.
The specific details are outlined on Page 10 of RFC 2409. I also explain some of the details of the process in this answer.
And are all the 3 session keys claculated by both Initiator and responder?
And are those 3 keys(Authentication, encryption and derivative keys) symmetric?
Yes and Yes.
And in Main Mode they protect the Exchange of the identity values described above in Message 5 and 6.
