ESAE series part 5 - Windows IPSec - TEAL Technology Consulting GmbH
post-template-default,single,single-post,postid-1814,single-format-standard,bridge-core-3.0.7,,no_animation_on_touch,qode-title-hidden,qode-child-theme-ver-1.0.0,qode-theme-ver-29.4,qode-theme-bridge,disabled_footer_top,qode_header_in_grid,qode-wpml-enabled,wpb-js-composer js-comp-ver-6.10.0,vc_responsive

ESAE series part 5 – Windows IPSec

Welcome to the fifth part of our ESAE series. In this part, we would like to focus on an area that is not covered so comprehensively on the Internet – Windows IPSec.

What is IPSec?

“IPsec (short for Internet Protocol Security) is a protocol suite designed to enable secure communication over potentially insecure IP networks such as the Internet.

IPsec works directly on the Internet Layer of the DoD model and is a further development of the IP protocols. The goal is to provide encryption-based security at the network level. IPsec offers this possibility by the connectionless integrity as well as the access control and authentication of the data. In addition, IPsec ensures the confidentiality and authenticity of the packet sequence through encryption.”

(Source Wikipedia)

Why IPSec in the current customer context?

As described in part 2 of the ESAE series, the Global Authentication Service environment was set up at a co-location service provider. In addition, the WAN connection between the customer site and the co-locator was also rented. Last but not least, all network components within the environment are also operated by a service provider. This means that there are a lot of network devices outside the control of the customer…

In principle, there are two ways to ensure the confidentiality of communication:

1. encryption at application level

2. network level encryption

Both are valid approaches. In our case, we opted for Option 2 with Windows IPSec in order not to have to analyze the complete communication of all components in detail and search for ways to encrypt them. The probability that we could not encrypt all relevant protocols (with simple means) was too high with option 1.

Schematic representation of the network

For a better overview here again the schematic representation of the network:

ADFS communication takes place via SSL and is therefore already encrypted. At the time GAS was set up, there were no global services. These have to be considered later.

The following communication lines remain to be encrypted:

   1.Domain controller to domain controller
   2.Server to domain controller
   3.Server to server
   4.PAW to Domain Controller
   5.PAW to server


In principle, this corresponds to the domain isolation model.

How does Windows IPSec work?

Microsoft has implemented IPSec as part of the Windows Firewall with Advanced Security. The IPSec configuration is thus defined in firewall rules. These rules can be created and managed either manually per server or via GPO.

For each rule 5 areas can be configured:

1. Endpoints

Here you define the IPs for which this rule is to apply.

2. Authentication

This defines whether authentication for incoming and/or outgoing network traffic is attempted or mandatory.

3. Authentication method

If authentication is attempted or enforced, this is where you specify which method should be used. The advanced settings and the IPSec default settings will be discussed later.

4. Protocols and ports

Here, as the name suggests, the protocols and ports to which the rule applies can be restricted.


On the last page before naming, you can define the firewall profiles for which the rule is to be applied.

At first glance, this seems quite simple, so why a separate blog post on the subject? A little anecdote in the next section to make clear why.

No PKI? No problem, authentication is also possible via Kerberos! Or?

At this point the IPSec defaults come into play. If you don’t want to redefine which authentication should be used for each rule, you can configure it in the IPSec defaults by clicking through this slim menu (right click on Windows Defender Firewall with Advanced Security -> Properties):

We’ll come back to the Main and Quick Mode settings later.

Ok, where is the anecdote? All right: Since we didn’t set up a PKI in the project due to time constraints, certificates were omitted (for now). Pre-shared key is certainly not a secure option. No problem – we still have Kerberos. We are in a Windows network and all our servers are domain-joined. So, we had configured IPSec for the above mentioned network routes in our lab and everything went great. Shut down VMs – closing time. The next morning the nasty surprise – nothing works anymore. After many hours of troubleshooting and swearing, it fell like scales from our eyes (well, a Microsoft Support Engineer was not completely uninvolved 😉):

To establish a Kerberos authenticated IPSec connection, a valid Kerberos token is needed. This is created by the domain controller. Once all the rules for the above network routes are configured, the domain controller responds only if IPSec is used for the connection… A vicious circle.

Certificates after all…! Here I spare you the story (= our suffering). IPSec certificates have to contain the OID (IP Security IKE Intermediate) as “Key usage”. Unfortunately, none of our customer’s providers issues such certificates. If you still want to use certificates, ALL certificates must be issued by exactly the same CA.

All certificates there, all from the same root, all rules configured – all good?

Of course not. The communication between all parties was running and in the monitoring tab of the firewall console we could see that the Main Mode and the Quick Mode were negotiated but that the traffic was not encrypted:

Indeed, we were quite a bit astonished. After some research we came across the crucial setting in the Advanced Quick Mode configuration menu. You have to check the box “Require encryption for all connection security rules that use these settings”! Only if this check mark is set, encryption will be performed. 😊

One last hint from the support engineer was that name resolution should be better excluded from IPSec.

Summary – Which rules and settings have been implemented?

General settings:

Key Exchange (Main Mode):

Integrity algorithms: SHA-384

Encryption algorithms: AES-CBC 256

Key exchange protocols: Elliptic Curve Diffie-Hellman P-384

Data Protection (Quick Mode):

Require encryption for all connection security rules that use these settings: checked

Encryption algorithms: AES-GCM 256-bit

Integrity algorithms: AES-GCM 256-bit

Firewall rules:

All Server /PAW Require IPSec:

Endpoint 1: <PAW VPN subnet> and <Server subnet>

Endpoint 2: <PAW VPN subnet> and <Server subnet>

Authentication Mode: Require inbound and outbound authentication

Authentication methods: Custom (Certificate)

Endpoint 1 port: Any

Endpoint 2 port: Any

Protocol: Any

Server/PAW to DC DNS exception:

Endpoint 1: <PAW VPN subnet> and <Server subnet>

Endpoint 2: <all domain controller IPs>

Authentication Mode: Do not authenticate

Authentication methods: No authentication

Endpoint 1 port: 53

Endpoint 2 port: Any

Protocol: UDP

Emergency PAW Exception :

Endpoint 1: <all domain controller IPs + Well defined PAW IP>

Endpoint 2: <all domain controller IPs + Well defined PAW IP>

Authentication Mode: Do not authenticate

Authentication methods: No authentication

Endpoint 1 port: Any

Endpoint 2 port: Any

Protocol: Any

Many may now wonder why there is the last rule. Well, as long as everything goes well, the rule is unnecessary. However, if something goes wrong, it is indispensable in the concrete context. We recall that the environment was built by a co-locater. This is located several hundred kilometres away from the nearest customer location. One can easily imagine that in the event of an error, longer downtimes would occur if one of the few trustworthy Tier 0 admins had to drive several hundred kilometres to work on the problem with local access only. So, we built in a two-stage fallback:

In the event of an error, the network provider can use VPN configuration to assign a specific IP to a PAW that is located outside the normal PAW VPN subnet. This IP is excluded from IPSec by the third rule from above. This enables remote administration in the event of an error, but no individual person can communicate maliciously or accidentally with the GAS environment unencrypted.

Before we conclude with the topic troubleshooting, a paragraph on another topic:


Commissioning of IPSec and installation of new systems

The above rules represent the final configuration. To achieve this, however, a step-by-step procedure is necessary. If you implement the rules the way they are written there, the GPOs are applied to the domain controllers first (normally), because the Group Policy Refresh Interval for domain controllers is by default 5 minutes (without offset). For all other AD members, however, the default interval is 90 minutes with a random offset of 0 – 30 minutes.

So how did we proceed?

Since the environment was completely rebuilt and the number of servers was manageable, we first added each IP individually to the GPO, started the GPO update locally and when the client had the new configuration, updated the domain controllers. As soon as we had done this for all systems and all communication was using IPSec, we created the rules for the complete subnet, waited for the refresh interval and deleted the rules with the individual IPs.

So far – so good! But what about clients that need to be reinstalled and added to the domain? This is quite simple. You have to create a local IPSec rule for the communication with the domain controller with the appropriate certificate, then the client can perform the domain join and gets the above mentioned rules via GPO.


The above explanations show that Windows IPSec is not a trivial thing, especially if you don’t have local access to the systems. Below are some tips that can help with troubleshooting.

1.Disable IPSec locally

To deactivate IPSec locally for a server, you have to deactivate the firewall service with the following registry key:

REG add “HKLM\SYSTEM\CurrentControlSet\services\MpsSvc” /v Start /t REG_DWORD /d 4 /f

Restart the server.

Since the firewall service is stopped, however, no changes can be made to the firewall (and therefore IPSec) settings. If these have to be adjusted, both the target system and a domain controller must be configured for unencrypted communication so that the target server can receive the new settings via Group Policy.


2. Activate logging

If you want to better understand what is happening with IPSec, you have to activate the corresponding Event Viewer Logs:

auditpol /set /subcategory:”IPsec Driver” /success:enable /failure:enable

auditpol /set /subcategory:”IPsec Main Mode” /success:enable /failure:enable

auditpol /set /subcategory:”IPsec Quick Mode” /success:enable /failure:enable

auditpol /set /subcategory:”IPsec Extended Mode” /success:enable /failure:enable

auditpol /set /subcategory:”Filtering Platform Packet Drop” /success:enable /failure:enable

auditpol /set /subcategory:”Filtering Platform Connection” /success:enable /failure:enable

There are more categories, but those were the ones we used.

Once the problem is solved, you can disable the logs accordingly:

auditpol /set /subcategory:”IPsec Driver” /success:disable /failure:disable

auditpol /set /subcategory:”IPsec Main Mode” /success:disable /failure:disable

auditpol /set /subcategory:”IPsec Quick Mode” /success:disable /failure:disable

auditpol /set /subcategory:”IPsec Extended Mode” /success:disable /failure:disable

auditpol /set /subcategory:”Filtering Platform Packet Drop” /success:disable /failure:disable

auditpol /set /subcategory:”Filtering Platform Connection” /success: disable /failure:disable

3. Firewall rules and Powershell

It was often helpful to show firewall rules on the command line:

show-netipsecrule –policystore activestore

get-netipsecrule –policystore activestore

4. Network trace on Server Core

If you need a network trace on a server core system, you can start the trace with:

netsh trace start capture=yes tracefile=.\mytrace.etl maxsize=300

And stop it with:

Net trace stop

The trace file can be opened with Netmon.

This concludes the article on IPSec. I hope it can help some of you who dare to tackle the subject. If you have any questions, please feel free to contact us. We are happy to help 😊.