IPsec is commonly used to secure communication between networks, especially when connecting data centers, cloud environments, or two organizations over the internet. It provides encryption, integrity protection, and authentication so that data can travel across an untrusted network without exposing its contents.
But configuring an IPsec tunnel is not only about choosing an encryption algorithm. One area that deserves attention is how encryption keys are generated, exchanged, renewed, and eventually replaced.
A VPN tunnel may use strong encryption, but if its keys are poorly managed, the overall security can still be compromised. This is why the National Institute of Standards and Technology (NIST) provides guidance on cryptographic key management and IPsec VPN configuration.
Understanding Key Management in IPsec
Before discussing the recommendations, it helps to understand how IPsec manages encryption keys.
IPsec operates at the network layer and commonly uses the Encapsulating Security Payload (ESP) protocol to protect traffic. ESP can provide confidentiality, integrity, and data origin authentication, depending on the configured algorithms and mode.
IPsec also uses Security Associations (SAs). An SA defines the security parameters used to protect traffic, including the algorithms, keys, and other settings. For ESP, each SA protects traffic in one direction, so a typical bidirectional tunnel requires separate SAs.
Manually configuring these keys is technically possible, but it becomes difficult to manage when multiple VPN connections are involved. Keys must be securely distributed, updated, and removed when they are no longer needed.
This is why most modern IPsec deployments use Internet Key Exchange version 2 (IKEv2).
IKEv2 automates the negotiation of cryptographic parameters and the establishment of security associations. It also handles the generation of fresh key material when security associations are rekeyed.
There are two important types of SAs involved:
- IKE SA: Protects the IKE communication used to negotiate and manage the connection.
- Child SA: Carries the parameters and keys used by ESP to protect the actual network traffic.
These have separate lifecycles. Rekeying a Child SA does not necessarily mean that the IKE SA must also be replaced.
Why Key Rotation Matters
Encryption keys should not remain in use indefinitely.
The longer a key is used, the more traffic it protects. Depending on the cryptographic algorithm and operating conditions, excessive use of the same key can increase security risks.
Key rotation limits how much data depends on a particular key. If a key is compromised, the amount of traffic potentially exposed may be reduced, although rotating a key cannot undo an earlier compromise.
IKEv2 supports rekeying, allowing new cryptographic material to be established before an existing SA expires. This helps maintain secure communication without requiring the entire VPN connection to be manually reconfigured.
For example, consider a site-to-site VPN connecting a company’s primary data center to a disaster recovery environment. The connection may remain established for months, even when there is little traffic.
Using the same encryption keys throughout that period would be an unnecessary risk. A better approach is to configure appropriate SA lifetimes and allow IKEv2 to renew the keys automatically.
The exact lifetime should depend on the organization’s security policy, the algorithms being used, and the VPN equipment’s capabilities. There is no single rekey interval that is correct for every IPsec deployment.
What NIST Recommends
NIST provides guidance through several publications. Two particularly relevant documents are SP 800-57 Part 1 Revision 5, which covers general cryptographic key management, and SP 800-77 Revision 1, which focuses on IPsec VPN implementations. General+1
1. Use Strong Cryptographic Algorithms
The security of an IPsec tunnel depends partly on the algorithms negotiated between its endpoints.
AES-GCM is a practical choice for modern ESP deployments because it provides encryption and authentication together. AES-128 and AES-256 are common options, with different security strengths.
For key exchange, use an appropriately strong Diffie-Hellman group, such as a supported 256-bit elliptic-curve group. Both VPN endpoints must support compatible cryptographic proposals.
Avoid keeping weak legacy algorithms enabled only for compatibility. NIST SP 800-131A provides guidance for transitioning away from algorithms and key lengths that no longer provide adequate protection. NIST Computer Security Resource Center+1
2. Protect Authentication Credentials
IKEv2 supports authentication using pre-shared keys (PSKs) or digital certificates.
Pre-shared keys are relatively easy to configure, particularly when connecting two devices. However, they must be generated securely, stored carefully, and never shared through unprotected communication channels.
Certificate-based authentication can be easier to manage across larger environments because each endpoint can have its own private key and certificate. Certificates also introduce operational responsibilities, including renewal, trust management, and revocation.
For enterprise deployments, I would generally prefer certificate-based authentication when a proper public key infrastructure is available.
3. Use Perfect Forward Secrecy
Perfect Forward Secrecy (PFS) is an important consideration when establishing IPsec security associations.
With PFS, fresh Diffie-Hellman key agreement is performed when generating new session keys. This helps prevent the compromise of long-term key material from exposing previously recorded traffic.
In IKEv2, fresh Diffie-Hellman exchanges during Child SA rekeying provide this protection for the new Child SA. Simply deriving replacement keys from existing IKE key material does not provide the same independent forward secrecy.
For tunnels carrying sensitive information, enabling fresh Diffie-Hellman exchanges during rekeying is a sensible security choice.
4. Define Appropriate Key Lifetimes
NIST SP 800-57 introduces the concept of a cryptoperiod, which is the period during which a key is authorized for a particular use.
Cryptoperiods should consider the algorithm, amount of protected information, sensitivity of the data, and potential consequences of compromise. General+1
For IPsec, administrators should configure appropriate IKE SA and Child SA lifetimes, and verify that automatic rekeying works correctly.
A practical consideration is to initiate rekeying before an SA expires. Otherwise, failed negotiations or mismatched configurations could interrupt traffic.
Common Configuration Mistakes
One mistake is assuming that an established VPN tunnel is automatically secure. A tunnel can remain connected while using outdated algorithms or credentials that have not been reviewed for years.
Another is configuring both endpoints independently without checking compatibility. Differences in supported algorithms, authentication methods, or rekeying behavior can cause intermittent connection failures.
Monitoring is also important. Administrators should be able to identify failed negotiations, repeated authentication failures, and unexpected SA expiration. Logs should contain enough information for troubleshooting without exposing private keys or sensitive credentials.
For business-critical connections, I would also test rekeying and tunnel recovery during maintenance activities. A successful initial connection does not guarantee that the tunnel will recover correctly after a network interruption or device restart.
Final Thoughts
IPsec provides strong mechanisms for protecting network communication, but the security of a VPN depends on how those mechanisms are configured and maintained.
NIST’s guidance is useful because it treats key management as an ongoing process rather than a one-time configuration. Strong algorithms, secure authentication, appropriate cryptoperiods, and reliable rekeying all contribute to a secure deployment.
For developers and infrastructure engineers, the main takeaway is to look beyond whether the VPN tunnel is connected. We also need to understand how its keys are managed throughout the connection’s lifetime.
A working tunnel is only the starting point. Keeping it secure requires proper configuration, monitoring, and regular review.
