When I first started learning IPSec, I often got confused with PSK (pre-shared key set by administrators) and key generated by Diffie-Hellman calculation. Many administrators simply configure phase 1 and phase 2 settings without actually understanding what is going behind the scenes.

Message 1 and 2

Initiator sends its proposals + Cookies known as CKYi. Remote end responds with matching proposals and its cookies known as CKYr.

Message 3 and 4

Diffie- Hellman exchange starts. Peers sends their own public keys + Nonce. Nonce is a random generated number that is used only once. Once DH exchange is completed, shared secret key is created.


Seed value is then calculated with following formula:

SKEYID = prf(pre-shared-key, Ni_b | Nr_b)

Session keys

Seed value + DH shared secret are combined to create 3 keys:

  • Derivative key (sometimes called ‘Phase 2 key’, used for final KEYMAT in phase 2)
 SKEYID_d = prf(SKEYID, g^xy | CKY-I | CKY-R | 0)
  • Authentication key
 SKEYID_a = prf(SKEYID, SKEYID_d | g^xy | CKY-I | CKY-R | 1)
  • Encryption key
SKEYID_e = prf(SKEYID, SKEYID_a | g^xy | CKY-I | CKY-R | 2)

Message 5 and 6

The ID Method is then combined (using a PRF) with the Seed value (SKEYID), and a few other values, to create the Identity Hash. The ID Method and ID Hash are then sent across the wire, and the other party attempts to re-create the ID Hash using the same formula. If the receiver is able to re-create the same ID Hash, it proves to the receiver that the sender must have had the correct pre-shared-key.

HASH_I = prf(SKEYID, g^xi | g^xr | CKY-I | CKY-R | SAi_b | IDii_b )
HASH_R = prf(SKEYID, g^xr | g^xi | CKY-R | CKY-I | SAi_b | IDir_b )

Phase 1 is now completed.

IPSec Phase 2 Quick mode (non PFS)

Phase 2 is made out of 3 messages:

Phase 2 Message 1 and 2

Initiator sends proposal with transform sets + Nonce + hash of the message. It is encrypted with SKEYID_e generated in phase 1. Note ESP SPI is different than IKE SPI. Proxy IDs are sent at the time too (what network will be part of VPN tunnel).

HASH = prf(SKEYID_e, M-ID(entire message - padding) SA | Ni)

Responder decrypts and validates message and answers with Message 2 with agreed proposal and its own calculated Hash.

Message 3

KEYMAT is calculated for each direction.

KEYMAT=prf(SKEYID_d, protocol | SPI |Ni_b | | Nr_b)

IPSec tunnel is now complete!

Leave a Comment

Your email address will not be published. Required fields are marked *