Asynchronous await gotcha with Mutex synchronization

I ran into an issue today that took me a while to figure out what was happening so I thought I'd share.

I wanted to take advantage of the asynchronous function: NamedPipeServerStream WaitForConnectionAsync to create a multi client Named Pipe server.  I also wanted to prevent other instances of the server running so I used a Mutex like this:

//Named pipe server run method

public async Task Run()
{
     using (var mutex = new Mutex(true, mutexName, out var createdNew))
            {
                if (createdNew)
                {
                    try
                    {
                        await RunInternal();
                    }
                    finally
                    {
                        mutex.ReleaseMutex();
                    }
                }
            }
}

Everything looked and ran normally until the RunInternal method completed.  The call to ReleaseMutex was throwing a synchronization exception:

ApplicationException: Object synchronization method was called from an unsynchronized block of code.

I didn't realize (or appreciate) that the await keyword returns control on a different thread.  Although method looks continuous, the Mutex was right to complain that a thread that didn't create the Mutex was trying to release it.

Once I understood what was going on, the fix was easy;  Use the synchronous Wait instead.  Note:  I also changed the method to be non-async.

public void Run()
{
       using (var mutex = new Mutex(true, mutexName, out var createdNew))
       {
         if (createdNew)
         {
           try
           {

             //Synchronously wait so that the finally block executes in current thread.
             //This is a requirement of the Mutex.ReleaseMutex method.
             RunInternal().Wait();
           }
           finally
           {
             mutex.ReleaseMutex();
           }
         }
       }
     }
}

Maybe there's better solution but this worked for me, much more to learn I'm sure.

Artifactory behind IIS reverse proxy on Windows Server with HTTPS

For those that aren't aware, Artifactory is a binary repository solution sold by JFrog.  Artifactory is primarily used for archiving and publishing, you guessed it, binaries as part of a continuous integration solution. The server software pretty much runs out of the box and supports multiple host systems.

My motivation behind this post is to document my adventures of enabling secure HTTPS access to Artifactory to increase security. 

My situation

• Artifactory Version 7.  Artifactory version 7 uses two ports, one for the API and one for the UI
• Windows Server as the host system
  
• Enterprise issued SSL certificates signed by corporate CA
• The desire to use the same secure port for both API and UI requests
  

My approach

• Use IIS as a reverse proxy 
• Requires the IIS add-on modules "Application Request Routing" & "URL Rewrite"
  
• Create two in bound rules to detect API and UI requests
• Hope for the best

Step one

Install IIS via the "Turn Windows Features On and Off".  There are lots of resources out there on how to do this so I won't repeat it here.

Step two

Download and install the two add-on modules:

https://www.iis.net/downloads/microsoft/url-rewrite
https://www.iis.net/downloads/microsoft/application-request-routing

Step three

Configure an IIS website to act as the reverse proxy to send all secure traffic to the locally hosted Artifactory endpoints.  For this, I used the default web site that wasn't used for anything else on the server. 

Application pool configuration

To help to ensure that no unexpected processing is done on the in coming requests, all managed code should be turned off.

 

Applying the certificates and enabling HTTPS

Import the HTTPS certificates into the server using the "Server Certificates" feature.

 Bind the site to HTTPS on the default 443 port with the specific certificate

 

Create inbound rules

Create an in-bound reverse proxy rule that matches API URLs by opening the URL Rewrite feature and adding a new Reverse Proxy rule.

 

Enter the location of the Artifactory API endpoint and check the "Enable SSL Offloading".   Ignore the outbound rules as they're not needed.

The result will look something like this:

 

Note:  The processing of downstream rules is disabled if the rule matches. 

Ensuring that URLs returned in the API are https

My first approach was to use the outbound rules to re-write any "http:" URLs in the produced content.  This required that compression be turned off in Artifactory so that the content could be modified. It did work but didn't feel right.  Thankfully I found a clue for a cleaner solution in the JFrog nginx reverse proxy documentation:  https://jfrog.com/help/r/artifactory-the-recommended-nginx-reverse-proxy-configuration-for-artifactory-7/artifactory-the-recommended-nginx-reverse-proxy-configuration-for-artifactory-7. 

The server variables dialog is used to set the X-JFrog-Override-Base-Url header.  The value is used by Artifactory when generating all links.  Note:  IIS uses some rules and conventions to convert the value entered in the dialog.  My first attempts failed because I naively thought you'd just use the header name directly.   https://learn.microsoft.com/en-us/iis/extensions/url-rewrite-module/url-rewrite-module-20-configuration-reference#note-about-request-headers

 

Server variables also need to be added so that the variable can be written or it just doesn't work

 

Adding a rule for the user interface endpoints

A second rule is required to map the UI endpoints to the Artifactory UI port using a similar process.  The end result looks like this:

 

Some final configurations

Note:  The following steps were taken from some documentation for putting Jenkins behind an IIS reverse proxy (which I also had to do): 

https://wiki.jenkins.io/JENKINS/Running-Jenkins-behind-IIS.html

 

As Artifactory is used to upload/download large files, the Maximum allowed content length needed to be increased as well.

Here's a template web.config that contains the above steps:

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <system.webServer>
        <rewrite>
            <rules useOriginalURLEncoding="false">
                <clear />
                <rule name="ReverseProxyInboundApi" stopProcessing="true">
                    <match url="(artifactory/.*)" />
                    <conditions logicalGrouping="MatchAll" trackAllCaptures="false">
                    </conditions>
                    <action type="Rewrite" url="http://myArtifactoryServer:8081{UNENCODED_URL}" appendQueryString="false" logRewrittenUrl="true" />
                    <serverVariables>
                        <set name="HTTP_X_JFrog_Override_Base_Url" value="https://myArtifactoryServer" />
                    </serverVariables>
                </rule>
                <rule name="ReverseProxyInboundUi" stopProcessing="true">
                    <match url="(.*)" />
                    <conditions logicalGrouping="MatchAll" trackAllCaptures="false" />
                    <action type="Rewrite" url="http://myArtifactoryServer:8082/{R:1}" />
                    <serverVariables>
                        <set name="HTTP_X_JFrog_Override_Base_Url" value="https://myArtifactoryServer" />
                    </serverVariables>
                </rule>
            </rules>
            <outboundRules>
                <clear />
                <preConditions>
                    <preCondition name="ResponseIsHtml1">
                        <add input="{RESPONSE_CONTENT_TYPE}" pattern="^text/html" />
                    </preCondition>
                </preConditions>
            </outboundRules>
        </rewrite>
        <httpErrors errorMode="Detailed" />
        <security>
            <requestFiltering allowDoubleEscaping="true">
                <requestLimits maxAllowedContentLength="4000000000" />
            </requestFiltering>
        </security>
    </system.webServer>
</configuration>
 

Trusting certificate authorities

There was also other fallout from applying corporate enterprise certificates; OpenSSL and Java don't automatically support trusted root certificates managed by Windows.  To solve this, I ended up having to manually add the certificates to the various Java keystores using the keytool utility: 

Note:  Run from an elevated command prompt

 set java_dir=C:\Program Files\Java\jdk-11.0.7
"%java_dir%\bin\keytool.exe" -importcert -alias MyCorpRoot -keystore "%java_dir%\lib\security\cacerts" -file "E:\MyCorpRoot.cer"
"%java_dir%\bin\keytool.exe" -importcert -alias MyEntCA -keystore "%java_dir%\lib\security\cacerts" -file "E:\MyEntCA.cer"

The source code revision tool, Git for Windows, also has this problem if the OpenSSL library is selected.  I'm sure there's a way to configure OpenSSL but it was easier for me to an un-install & re-installed with the option to use the native security channel.

I hope that helps someone trying to do the same or similar.