What Is an IPsec VPN? IKEv2, ESP, and Site-to-Site Tunnels

Key takeaways

• IPsec protects selected IP traffic between configured endpoints.
• IKEv2 authenticates peers and manages security associations, while ESP protects data packets.
• Tunnel mode carries the original IP packet inside an outer packet routed between VPN endpoints.
• A connected tunnel still needs correct traffic selectors, routes, firewall rules, and application permissions.

An IPsec VPN uses Internet Protocol Security to protect IP traffic between devices or networks. A typical deployment combines IKEv2 for authentication and key management with ESP for protecting the packets that carry application data.

Suppose a company needs its branch office to access an inventory service at headquarters. The branch uses 10.10.0.0/16, and headquarters uses 10.20.0.0/16. Gateways at the two locations can protect traffic between those networks as it crosses the public internet.

The employees’ applications do not need to implement IPsec themselves. The gateways handle the protected path.

What IPsec Adds to a VPN

IPsec operates at the network layer. It can protect traffic from many applications according to policy, rather than requiring each application to use a special forwarding protocol.

NIST’s Guide to IPsec VPNs describes this framework and its deployment considerations.

In the branch example, the policy might cover traffic between the two private subnets. Other internet traffic could leave directly through the branch’s ordinary connection.

An IPsec VPN is therefore a particular VPN implementation, not a guarantee that every packet from every connected device receives the same protection.

IKEv2 Establishes the Relationship

Internet Key Exchange version 2, or IKEv2, lets the gateways authenticate and negotiate the state needed to protect traffic.

That state is organized into security associations. An IKE security association protects management exchanges; Child security associations support the protected data traffic. The peers also negotiate traffic selectors describing the traffic covered.

Authentication can use pre-shared secrets or certificates, among other deployment-supported mechanisms. IKEv2 also manages rekeying so a long-running connection does not depend indefinitely on one set of traffic keys.

If the branch and headquarters disagree about identities, credentials, or supported parameters, this setup can fail before application traffic crosses the link.

ESP Protects the Packets

Encapsulating Security Payload, or ESP, provides packet protection according to the negotiated configuration. Encrypted VPN deployments use it for confidentiality together with integrity and authentication protections.

In tunnel mode, the original IP packet is encapsulated. An outer IP header allows the internet to route the result between the gateways. The receiving gateway verifies and processes the protected packet before forwarding the original traffic.

ESP can also detect replayed packets through sequence numbers and receiver state. That is separate from an application’s handling of repeated business requests.

The ESP specification defines these services. Authentication Header, another IPsec protocol, does not provide encryption and is less common in typical encrypted VPN deployments.

Tunnel Mode and Transport Mode

Mode	What happens to the original packet	Typical use
Tunnel	The original IP packet is carried within a new outer packet	Gateway-to-gateway network connections
Transport	Protection applies primarily to the payload after the original IP header	Suitable host-to-host configurations

The company’s office connection is naturally described using tunnel mode. The outer addresses identify the gateways, while the inner addresses identify the communicating private hosts.

Remote access changes one endpoint to an individual device. It also introduces user authentication, device policy, and per-user access requirements.

Why NAT and Packet Size Matter

Many endpoints sit behind Network Address Translation. NAT traversal commonly carries ESP in UDP so translated traffic can pass through the intervening network.

Firewall policy must allow the deployment’s IKE and data traffic. Successful initial negotiation does not prove that every packet type or later exchange will pass.

Encapsulation also adds bytes. A path that handles small requests may struggle with larger packets if MTU discovery or fragmentation is mishandled. The symptom can look like an application that connects but stalls during transfers.

These are data-path problems, not necessarily weak encryption or incorrect passwords.

Troubleshoot Setup and Delivery Separately

For the inventory service, first determine whether the required security associations exist. Then trace the traffic they are supposed to carry.

Observation	Useful next check
Peer authentication fails	Identities, credentials, certificate validation, and negotiation logs
Tunnel established, no outgoing protected packets	Local route and traffic selectors
Outgoing traffic without a response	Remote firewall, destination service, and return route
Only larger transfers fail	MTU behavior and path filtering
Failures after a period of normal operation	Rekeying, expired credentials, and state changes

Overlapping private address ranges can also make routing ambiguous. A green tunnel indicator does not resolve an addressing conflict.

Keep the Protection Boundary Clear

The gateway tunnel protects traffic between the branch and headquarters gateways. Traffic before and after those endpoints needs its own appropriate protections, including HTTPS where applicable.

The inventory service must still authorize users. A branch network connection should not grant unrestricted access to every headquarters system. Zero trust access principles remain relevant even when the transport is encrypted.

Likewise, malware on an authorized device can send harmful requests through a functioning tunnel.

Define the permitted traffic, confirm both directions of routing, and monitor the application outcome. The goal is a controlled, protected connection to the inventory service, not merely a successful VPN status.

A tunnel can be established successfully while the application traffic it was built for still goes nowhere.