The ISA Corner section of my blog contains all the essential information any ISA administrator needs in order to keep ISA Server at peak performance.

Get the buzz on new Microsoft ISA KB articles, learn how to properly publish different services without exposing the firewall to vulnerabilities; deploy and manage ISA Server; make the most of ISA’s caching; and keep the company network protected against hackers and malware threats.

I don’t know if many ISA server guys and girls also read the Routing and Remote Access Blog but, because ISA makes extensively use of the Windows RAS features for the VPN stuff, it might be useful to follow that one too.

Lately there was a two part blog about the RAS Administration DLL:

• RAS Administration DLL - Your customized solution to manage the server!
• Steps to develop a RAS administration DLL using Visual Studio

In my humble opinion, one of the applications of the RAS Administration DLL could be to implement a control on the ISA server such that only one VPN connection could be made to an ISA server using a single user name. This is basically to prevent users from distributing their usernames and hence resulting in lot of connections with the same username. I think this is a much asked feature.

Maybe some ISA folks with some software development skills can build and test such a control for the ISA server community.

HTH,
Stefaan

Let this be a lesson to all of us, just because someone complains about an application level problem doesn’t mean it’s always above layer-5 … (Jim Harrison).

Some time ago, I wanted to catch up on some TechEd 2007 videos published on the TechNet Spotlight site. However I was unable to play any Silverlight enabled video through an up-to-date ISA 2006 server as long as my IE 7 browser was configured as a Web Proxy client. Each time the Silverlight ActiveX control showed the following error message:

This clearly indicated a connection problem. So, my best bet was to take a look at the ISA logging to find out what was going on. I found quickly that each time that error message appeared, I saw the following entry in the ISA logging:

The HTTP Status Code 13 The Data is invalid suggested that the ISA server complained about the HTTP request, although I couldn’t see anything wrong in those ISA log entries. So, I decided to take a network monitor trace on the ISA internal and external interface. For a detailed scenario layout, check out my ISA lab. Here is an excerpt of that trace:

In frame #27 you see the request for a particular TechEd session given by Steve Riley, sent by the client to the ISA server. In frame #32 that request is forwarded by the ISA server to the web site. In frame #73 the last packet of the response is received by the ISA server from the web site and in frame #83 it is forwarded by the ISA server to the client. Now, in frame #444 we see the request for the actual video belonging to that TechEd session, sent by the client to the ISA server. In frame #447 you see the response from the ISA server to the client with the HTTP Status Code 400 Bad Request ( The Data is invalid. ). Here are the details of that HTTP request in frame #444:

Because I couldn’t find anything wrong, I mailed all the info I have to Jim Harrison to get a second opinion. At first sight, Jim couldn’t find anything wrong either in those traces. After some further investigation and testing, Jim saw the light. In fact, the answer was there right in front of us, even in the ISA logging! To clearly show the cause of this problem, let reformat a little bit the exact same excerpt of the network monitor trace:

As you can see, we just added the column source and destination port number. That means they are very important in this case. In frame #27 you see the request is sent to TCP port 8080 on the ISA server internal interface. That’s normal behavior because the Web Proxy service is listening on that specific port number. In frame #32 that request is forwarded by the ISA server to the web site on TCP port 80. That’s also normal behavior. Now, in frame #444 the request is sent to TCP port 80 on the ISA server internal interface. Oops… that’s definitely wrong because the Web Proxy service is listening on TCP port 8080! Since only auto discovery requests are allowed to the TCP port 80 listener (/wpad.dat, /wspad.dat, /array.dll?Get.Routing.Script, /array.dll?Get.Info.v[1|2|3]), ISA server rejects it with the HTTP Status Code 400 Bad Request ( The Data is invalid. ).

In other words, the Silverlight ActiveX control does only partial honor the proxy settings obtained (only the host name, not the TCP port number). This is clearly a bug in the Silverlight ActiveX control and is resolved from version 1.0.21115.00 onwards (at least as far as I can determine). Also, if we would have thoroughly inspected the ISA log entry, we would have seen that it said that that request was originating from 192.168.22.100 (the client) and was destined for 192.168.22.1:80 (the ISA internal interface on TCP port 80). Again, the answer was right in front of us, but we didn’t realize it at first sight.

 
HTH,
Stefaan

An ever recurring topic on the message boards is the inability to connect to a VPN server with multiple VPN clients from behind a NAT device. We can assure you that if you run an up-to-date ISA 2004/2006 server, that means one with all the latest ISA and Windows service packs, the culprit is *not* the ISA server but definitely the NAT device not handling properly multiple VPN clients. In this blog we will analyze this scenario for L2TP/IPsec based VPN clients. Take note that because the L2TP protocol is protected by the IPsec protocol, that means that IPsec is the outer protocol seen by the NAT device, we will focus on the IPsec NAT Traversal problem only.

As always, we will use my ISA lab running in VPC 2007 to setup this scenario. The ISA 2006 Server used is listening on the Local External Network (192.168.1.0/24) and the L2TP/IPsec VPN clients are sitting on the Remote External Network (192.168.11.0/24). Both networks are interconnected with a Windows 2003 RRAS box acting as a NAT device (N:1). As we will see, the design of the IPsec NAT Traversal protocol (NAT-T) does *not* require that the NAT device have a special NAT editor or “helper” for the IPsec protocol. Ironically, a NAT box with an IPsec “helper” functionality might create further incompatibilities, making an already difficult problem harder or even impossible to solve. For more information, check out the IETF document RFC 3715 - IPsec-NAT Compatibility Requirements.

Before diving into the details, it’s necessary to have a good understanding of how the IPsec protocol works and especially how the IPsec NAT Traversal is done. Also, a good understanding of how to troubleshoot such a scenario is very handy. To refresh your knowledge on how the IKE Negotiation happens, check out my blog Basic Troubleshooting for IPsec based VPN’s and related links. In short:

• In the main mode messages 1 and 2 the peers announce their IPsec NAT-T capability (VendorID or VID).
• If both peers do support it then the NAT-T discovery phase is done in the main mode messages 3 and 4 (NAT-D payloads).
• Assuming at least one NAT device is detected along the path then, from the main mode mode messages 5 and 6 onwards, the IKE traffic will move to a new UDP port number, the IKE Header will change to a Floated IKE Header format and the peer behind the NAT device will start sending NAT-keepalive packets.

Now that we know what to look for, we can finally analyze the N:1 NAT scenario. To accomplish that we placed two VPN clients behind a NAT device (in this case an RRAS server) and simultaneously traced with a network monitor the IPsec IKE negotiation on both sides of the NAT device for the first and the second VPN client. When analyzing the traces, we carefully looked into the IKE main mode messages, including their IP and UDP header.

For both VPN clients we can summarize the result in the following figure:

The IKE main mode negotiation start as usual at UDP port 500. The NAT device changes as expected the IP address and UDP port number of the initiator (192.168.11.22:500 and 192.168.11.20:500). The translated packet header fields are marked with a yellow color in the above figure. After the IKE main mode message 4, both peers have detected the presence of the NAT device and therefore all further communication (main mode, quick mode and ESP) will happen at UDP port 4500 (NAT-T). Again, the NAT device will change the IP address and UDP port number of the initiator (192.168.11.22:4500 and 192.168.11.20:4500). Those translated packet header fields are also marked with a yellow color. Take note that the NAT device should not change anything else. In other words the UDP payload should not be touched at all. Moreover, the normal NAT housekeeping in the NAT device should be enough to distinguish the different IPsec sessions from each other.

For those interested what a network monitor trace looks like in the above scenario, the following figure shows what the second VPN client and the ISA server see on the wire during the L2TP/IPsec VPN setup:

Another way to prove that both VPN clients could connect to the ISA server, is looking on all the IPsec peers which IPsec security associations (SAs) are established. In the following figure we show all the main and quick mode SAs and how they are related to each other on both the second VPN client (one of the initiators) and the ISA server (the responder):  

On the left-hand side the main and quick mode SAs of the VPN client (initiator) are listed.  On the right-hand side the main and quick mode SAs of the ISA server (responder) are shown. Take note we marked some communication related info:

• When looking up the corresponding main mode SAs on the initiator and the responder, make sure that the cookies matches (1).
• Once the main mode SAs are found it is easy to find the used IP addresses and UDP port numbers (2). In this example you can see that the initiators socket (IP address and port number) is translated from 192.168.11.20:4500 to 192.168.1.30:62636.
• To find the corresponding quick mode SAs can be a little bit harder, especially on the ISA server (responder). The reason for this is that there might be a large number of them in a production environment. The trick is to use the IP addresses and UDP port numbers found in the main mode SA (3).
• Once the quick mode SAs  are found you will note they do not reveal much additional communication related info (4 and 5). From the point of view of the IPsec NAT Traversal problem, the fact there is a quick mode SA is far more important.

It should be obvious by now that in order to pass multiple L2TP/IPsec VPN clients through a NAT device, the NAT device must *not* have a special NAT editor or “helper” for the IPsec protocol. In fact a NAT box with an IPsec “helper” functionality might create further incompatibilities. So, if you have problems with multiple L2TP/IPsec VPN clients behind a NAT device, don’t blame the ISA server but get out your favorite network monitor tool to determine if the NAT device is behaving well.

HTH,
Stefaan

I want to share the following information provided by Jim Harrison about the random authentication prompts from Windows Media Player (WMP).

Problem:

WMP sometimes displays authentication prompts even though the logged-on user account is resolvable by the ISA server and has permissions to access the content through the ISA server policies.

Scenario:

The ISA web proxy is configured for Windows Integrated authentication. The ISA server policies enforces authentication for HTTP traffic. WMP is configured to use the ISA server as a web proxy server for the HTTP protocol (this includes “autodetect” or “browser”).  

Discussion:

When WMP is acting as a web proxy client (CERN) and the web proxy server requires Windows Integrated authentication, WMP will not auto-authenticate to the web proxy server if the web proxy server is specified as either an FQDN or an IP address. If the web proxy server is specified as a NetBIOS (unqualified) name, WMP will auto-authenticate using the interactive account credentials. If the web proxy server requires Basic or Digest authentication, an authentication prompt is expected, regardless of how the web proxy server is specified. This behavior is the same if the web proxy server is obtained via an automatic configuration (WPAD) script.

By default, ISA 2004 and higher lists the web proxy servers using their IP addresses in the WPAD script. This default was chosen to prevent name resolution errors from impeding normal client-to-web proxy communications. While this works well enough for browsers, WMP has issues when the web proxy server is specified using anything other than the NetBIOS name.

Solution (two-part):

1. Disable the proxy server settings for HTTP (pick one).

• Using WMP:
  under Tools, Options, Network, Protocols, HTTP, set to None
  
• Using Regedit:
  Key: HKCU\Software\Microsoft\MediaPlayer\Preferences\ProxySettings\HTTP
  Name: ProxyStyle
  Type: DWORD
  Value: 0
  
• Using GPO:
  under User Configuration\Administrative Templates\Windows Components\Windows Media Player\Networking, set the Configure HTTP Proxy option to Disabled.

2. Install the Firewall client from Microsoft downloads http://www.microsoft.com/downloads/details.aspx?Fa...43da89

After making this two changes, the Firewall client will handle all HTTP requests from WMP and ISA server authentication will now be satisfied through the Firewall client control channel instead of the HTTP protocol mechanisms. This will stop the random authentication prompts from WMP.

HTH,
Stefaan

When ISA server delivers the WPAD script, it sets a Cache-Control header with the value “max-age=3000″, that means 50 minutes. Internet Explorer rounds this up to 1 hour. So, if you are testing WPAD changes, i.e. you are playing with “Auto Discovery” or “Direct Access”, then you will need to be sure to delete the previously acquired WPAD script and resolved WPAD name.

The easiest way to accomplish this is to execute the following steps:

1. Clear the Internet Explorer cache completely: ActiveX Controls, Cookies, History, etc..
   
2. Close all instances of Internet Explorer.
   
3. Delete all WPAD script instances. Open a command window as administrator and type the following command:
   del \wpad*.dat /s
   
4. Clear the DNS and Netbios name caches. Open a command window as administrator and type the following commands:
   ipconfig /flushdns
   nbtstat -R

You should now have a clean starting point for testing WPAD script changes.

HTH,
Stefaan

When you request a certificate using the Certificate MMC snap-in or the certificate auto-enrollment mechanism, the request fails. Of course we assume here that the CA is started and you have sufficient permissions to request a certificate. This issue occurs because DCOM is required to acquire a certificate. However, for access rules a default “Enable strict RPC compliance” setting is configured on each RPC rule. With this setting in place, DCOM traffic is blocked. 

To workaround this issue without using an access rule that allows all outbound protocols between the host requesting the certificate (the source) and the CA (the destination), you can apply the following steps:

• On the CA, configure the RPC application or DCOM endpoint to use a custom TCP protocol port as a static port.
• On the ISA, turn off the “Enable strict RPC compliance” setting on the RPC access rule.
• On the ISA, create the custom protocol for outbound use.
• On the ISA, create an access rule to allow the custom protocol between the required source and destination.

On the CA, configure the RPC application or DCOM endpoint to use a custom protocol port as a static port.

The CA is primarily implemented as a DCOM application. By default, DCOM uses high-numbered TCP ports to respond to client requests. With the DCOM Configuration MMC Snap-In, Certificate Services can be enforced to use a custom TCP port.

1. While logged on to the issuing CA, from a command-line prompt, run the following command:
   dcomcnfg.exe
2. In the left pane of the Component Services MMC Snap-In, expand Component Services, Computers, My Computer, and then DCOM Config.
3. In the right pane, select CertSrv Request.
4. On the Action menu, click Properties.
5. On the Endpoints tab, click Add.
6. Select Use static endpoint, enter an unused TCP port number, for example 789 (my choice), and then click OK twice.
7. Close the MMC Snap-In.
8. Restart the certification authority service through the MMC Snap-In or by the following commands:
   net stop certsvc & net start certsvc

To undo this change at a later stage, remove the added entry from the list of endpoints.

On the ISA, turn off the “Enable strict RPC compliance” setting on the RPC access rule.

If ISA Server is requesting the certificate, disable the “Enforce strict RPC compliance setting” on the system policy rule.

1. To do this, on the Firewall Policy tab of ISA Server Management, click Edit System Policy on the Tasks tab.
2. Select the Active Directory group in the Configuration Groups list.
3. On the General tab, clear the Enforce strict RPC compliance checkbox.

If a host is requesting the certificate from another network through ISA Server, do the following: in the Firewall Policy tab of ISA Server Management, right-click the access rule allowing the traffic, and then click Configure RPC protocol. On the Protocol tab, clear the Enforce strict RPC compliance checkbox.

On the ISA, create the custom protocol for outbound use.

I assume everybody is able to accomplish this task   

My choice was to use TCP port 789 outbound for the CertSrv Request protocol definition.

On the ISA, create an access rule to allow the custom protocol between the required source and destination.

Again, I don’t think that creating an access rule requires much explanation. The important thing to note is that if ISA Server is requesting the certificate, the rule must apply to all users. Obviously the source is Local Host in this case and the destination can be further restricted to only the computer object representing the CA.

For publishing rules ISA Server blocks DCOM traffic, and as far as I know this setting cannot be modified. Therefore, if a host is requesting the certificate from another network through ISA Server, you must have a route relationship between the required source and destination, and create the appropriate access rule.

For more information, check out the following resources:

• RPC Filter and “Enable strict RPC compliance”
• Configuring and Troubleshooting Windows 2000 and Windows Server 2003 Certificate Services Web Enrollment

HTH,
Stefaan

The ISA Server product team blog ISA Server and Windows Server 2003 Service Pack 2 states that it might be necessary to disable receive-side scaling (RSS) when running an ISA server on Windows 2003 SP2. In fact, according to Jim Harrison it is absolutely recommended to disable receive-side scaling *and* offload support in the TCP/IP stack. How to accomplish that is explained in the Microsoft KB article You may experience network-related problems after you install Windows Server 2003 SP2 or the Scalable Networking Pack.

However, depending on the NIC driver it might be necessary to disable also offloading of tasks to the NIC (DisableTaskOffload) as well as the TCP Chimney offload in the stack (EnableTCPChimney). This registry entries are found under the registry subkey HKLM\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters :

• DisableTaskOffload
  Type: REG_DWORD
  Values: 0, 1
  
• EnableTCPChimney
  Type: REG_DWORD
  Values: 0, 1

Update March 11, 2008: check out http://support.microsoft.com/kb/948496. This disables the “Scalable Network Pack” that has caused so much hate & discontent for ISA administrators the world over. So, add this to your “must include” for any ISA or RRAS deployment that includes Windows Server 2003 SP2.

HTH,
Stefaan

You can read Jim Harrison’s definitive guide at:

• Testing RPC over HTTP through ISA Server 2006 Part 1; Protocols, Authentication and Processing
• Testing RPC over HTTP through ISA Server 2006 Part 2; Test Tools and Strategies
• Testing RPC over HTTP through ISA Server 2006 Part 3; Common Failures and Resolutions

HTH,
Stefaan

Troubleshooting IPsec based VPN’s can be very challenging. The reason for this is that you need to have a thorough knowledge of the IPsec protocol. Moreover, how to enable and interpret the more advanced IPsec diagnostic logging, more precisely the Oakley log, is not very well documented. In this short article we will summarise some troubleshooting steps you can apply to the IPsec part of the VPN. However, it is outside the scope of this article to explain how to read the Oakley log.

Before diving into the details, it’s necessary to have a good understanding of how the IPsec protocol works. Of course the IETF RFC documents are the authorative resource but they are probably too boring to read for most people. Instead, we can recommend the excellent Technet article series IPSec Technical Reference. This is definitely a must read! When you only want to refresh your knowledge on how the IKE Negotiation happens, check out the Cable Guy articles IKE Negotiation for IPsec Security Associations and IPsec NAT Traversal Overview, as well as section 2. IPSec NAT Traversal in my article How to pass IPSec traffic through ISA Server.

1. Enable IKE Auditing

The Windows Security Log is the recommended starting point when trying to determine the reason for an IKE negotiation failure. To view these IKE events, enable success and failure auditing for the Audit logon events audit policy for your domain or local computer. The IKE event category is also used for auditing user logon events in services other than IPsec.

When you enable success or failure auditing for the Audit logon events audit policy, IPsec records the success or failure of each Main mode and Quick mode negotiation and the establishment and termination of each negotiation as separate events. Keep in mind that enabling this type of auditing can cause the security log to fill with IKE events.

In Windows XP SP2 and Windows Server 2003, all IKE audits can be disabled with a DisableIKEAudits registry key. Be sure to check this value on computers that are being investigated. To disable auditing of IKE events in the security log, even if the Audit logon events audit policy is enabled, do the following:

set the HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\Lsa\Audit\DisableIKEAudits registry key to a value of 1.

Deleting the DisableIKEAudits registry key or setting the value to 0 restores the default setting. After making a change to this registry key, you must either restart the computer or stop and then restart the IPSec service by running the net stop policyagent and net start policyagent commands at the command prompt. Note that stopping and restarting the service may disconnect all computers using IPsec from the computer on which the service is stopped.

In Windows Vista, the IKE audits can granulary be enabled or disabled with the auditpol.exe commandline tool. Moreover, no IPsec service must be restarted before the new settings become active. To see all possible Audit logon events subcategories, use the following command at the command prompt:

auditpol /get /category:”Logon/Logoff”

 To change the audit settings for one of the IKE subcategories, use the following command at the command prompt:

auditpol /set /subcategory:<name> /success:<enable>|<disable> /failure:<enable>|<disable>

Now that we know how to enable and fine tune the IKE auditing, how does a normal IKE audit looks like in Windows XP SP2, Windows Server 2003 and Windows Vista?

In Windows XP SP2 and Windows Server 2003, an IKE audit for a successful L2TP/IPsec VPN connection shows the following sequence of events:

• ID 541: IKE security association established, Mode: Key Exchange (Main mode).
• ID 541: IKE security association established, Mode: Data Protection (Quick mode).
• ID 542: IKE security association ended, Mode: Data Protection (Quick mode).
• ID 543: IKE security association ended, Mode: Key Exchange (Main mode).

When something goes wrong during the IKE negotiation, one or more of the above event ID’s will be missing. Instead you will see some IKE failure audit events such as:

• ID 544: IKE security association establishment failed because peer could not authenticate. The certificate trust could not be established.
• ID 545: IKE peer authentication failed.
• ID 546: IKE security association establishment failed because peer sent invalid proposal.
• ID 547: IKE security association negotiation failed.

In Windows Vista, an IKE audit for a successful L2TP/IPsec VPN connection shows the following sequence of events:

• ID 4650: An IPsec Main Mode security association was established. Extended Mode was not enabled. Certificate authentication was not used.
     or
  ID 4651: An IPsec Main Mode security association was established. Extended Mode was not enabled. A certificate was used for authentication.
• ID 5451: An IPsec Quick Mode security association was established.
• ID 5452: An IPsec Quick Mode security association ended.
• ID 4655: An IPsec Main Mode security association ended.

When something goes wrong during the IKE negotiation, one or more of the above event ID’s will be missing. Instead you will see some IKE failure audit events. However, we didn’t find a list of all possible IKE failure audit events. Therefore we can only list those we have seen during our limited testing with Vista L2TP/IPsec VPN connections:

• ID 4652: An IPsec Main Mode negotiation failed. Failure Reason: IKE authentication credentials are unacceptable.
• ID 4653: An IPsec Main Mode negotiation failed. Failure Reason: New policy invalidated SAs formed with old policy.

As you can see, Windows Vista include new IPsec audit-specific events and the text of existing events has been updated with more useful information. One of the goals of these improvements is to help you troubleshoot failed IPsec negotiations without having to enable the advanced Oakley logging capability.

2. View IKE/IPsec Security Associations and Statistics

In Windows XP SP2, Windows Server 2003 and Windows Vista, IP Security Monitor is implemented as a Microsoft Management Console (MMC) snap-in. IP Security Monitor allows you to view details about an active IPsec policy that is applied by the domain or locally, and to view quick mode and main mode statistics, as well as IPsec security associations (SAs). IP Security Monitor also enables you to search for specific main mode or quick mode filters. 

Another way of viewing the same information is by using the commandline tools IPseccmd (Windows XP SP2) or Netsh (Windows Server 2003 and Windows Vista).

IPseccmd is a command-line alternative to the IP Security Policy MMC snap-in for Windows XP. In Windows XP SP2 the tool provides additional functionality and must be installed from the Support Tools folder on the Windows XP SP2 CD. Some useful commands are:

ipseccmd show sas

ipseccmd show stats

Netsh is a command-line scripting utility that allows you to display or modify the network configuration. In Windows Server 2003 and Windows Vista the tool is enhanced to provide IPSec diagnostic and management functionality and replaces therefore the IPseccmd in Windows XP. Some useful commands are:

netsh ipsec dynamic show mmsas all
netsh ipsec dynamic show qmsas all

netsh ipsec dynamic show stats 

3. Use Network Monitor and IKE Tracing

When you really want to know how the IPsec stuff works on the wire or when you have to analyse IKE negotiation failures, you should be handy with a network monitor tool and learn to explore the Oakley log. Both skills are required to be able to diagnose the hard to find problems. The most popular and free network monitor tools are Wireshark and Microsoft Network Monitor 3. Which one you use isn’t that important. Just make sure you know how to use the features available in the tool.

As an example, an excerpt of a ‘Network Monitor 3′ frame summary for a successful L2TP/IPsec VPN connection:

In the above figure you see clearly the important phases in the startup of an IPsec session. First you have the main mode negotiation shown in the frames 1 - 8. Second you find the quick mode negotiation in the frames 9 - 12. At last you have the data transfer (ESP packets) starting from frame 13 onwards. Take note that in the case of a L2TP/IPsec VPN connection the L2TP negotiation starts at the first ESP packet (frame 13 in this example).

Additional you can see that message 5 and 6 in the main mode negotiation are fragmented in two packets (frames 5-6 and 7-8). That’s normal behavior because the identification is generally based on certificates and they are usual too big to fit in one packet. Therefore, IP fragments must be supported along the path. Also, you can observe that from frame 5 onwards the flag “E” is set. This means that the payload is encrypted. In other words, you can’t inspect the content of those payloads. This shows clearly a major limitation in using a network monitor tool for debugging IPsec traffic. Nevertheless, it’s an excellent tool to check out the general packet flow up to and including layer-4 (e.g. NAT-T Encapsulation) as shown in the figure below:  

When troubleshooting IPsec and taking network traces between hosts, especially in a lab environment, it is considered best practice to lower the ESP encryption level (if it’s currently at DES/3DES) to ESP-Null. This will help in reading and understanding the network trace much better than going through the encrypted traffic capture. One method to accomplish that is described in my article How to enable ESP Null Encryption on ISA 2004 in a site-to-site VPN scenario.

As a last resort, you can enable IKE tracing or Oakley logging to analyse IKE negotiation failures. The Oakley log records all IKE (ISAKMP) main mode and quick mode negotiations in great detail. In Windows XP SP2 and Windows Server 2003, the Oakley log is stored in the systemroot\Debug folder. A new Oakley.log file is created each time the IPsec policy agent is started and the previous version of the Oakley.log file is saved as Oakley.log.sav.

To enable Oakley logging in Windows XP SP2, use the following command:

ipseccmd set logike|dontlogike  

To enable Oakley logging in Windows Server 2003, use the following command:

netsh ipsec dynamic set config ikelogging 1|0

As already mentioned, one of the goals of the IKE audit improvements in Windows Vista is to help you troubleshoot failed IPsec negotiations without having to enable the advanced Oakley logging capability. Apparently Microsoft decided to push that idea to such an extreme they removed the possibility to enable or disable the Oakley logging in the netsh command, though it is still listed in the usage of the netsh command.

To enable Oakley logging in Windows Vista, do the following (of course you need elevated privileges):

1. Create a local \Tools directory and copy in TRACEFMT.exe and TRACEPRT.dll from the Windows XP SP2 Support Tools as a free download.
2. Set the registry key HKLM\SYSTEM\CurrentControlSet\Services\Ikeext\Parameters\EnableLogging = DWORD 0×1.
3. Stop/Start the IKEEXT service. Do not use “restart” because that does not have the same effect. Alternatively, run “NET STOP IKEEXT” and “NET START IKEEXT” at a command prompt.
4. Repro the problem by initiating the IPsec based VPN connection that you wish to troubleshoot.
5. Verify the log file was created in the directory %windir%\system32\ikeext.etl and has flushed the memory buffers into the file (e.g. make sure the file is not a 0 byte value).
6. Run the following command from a command prompt with the working directory set to the \Tools directory:
   tracefmt.exe %systemroot%\system32\ikeext.etl -tmf %systemroot%\System32\wfp.tmf -o %TEMP%\WFPDiag.txt
7. Review the WFPDiag.txt file for Oakley-level data. Given the addition of AuthIP for transport mode IPsec policies from the Windows Firewall with Advanced Security, the file is a little busier than it used to be.
8. Set the registry key HKLM\SYSTEM\CurrentControlSet\Services\Ikeext\Parameters\EnableLogging = DWORD 0×0.
9. Stop/Start again the IKEEXT service. Do not use “restart” because that does not have the same effect. Alternatively, run “NET STOP IKEEXT” and “NET START IKEEXT” at a command prompt.
10. Optional: Delete the ‘%systemroot%\system32\ikeext.etl’ file.

To prove the above procedure really works, here is a small excerpt of an Oakley logging on a Windows Vista machine:  

For further troubleshooting information, check out the following articles: 

• IPSec Tools and Settings
• IPSec troubleshooting tools
• Troubleshooting VPN over IPsec
• Server and Domain Isolation Using IPsec and Group Policy - Chapter 7: Troubleshooting IPsec

HTH,
Stefaan