It’s always been annoying to pay for and manage the SSL certificates in the lab environments I manage.  That’s why I was a very early adopter of Let’s Encrypt.  It’s a fantastic resource for free, hands-off SSL certificates…as long as you’re on a platform that supports it.  Unfortunately NetScaler is, at this time, not one of those platforms.  Lucky for us, there are folks like Ryan Butler out there.  He created an awesome python script to automate the creation and renewal of Let’s Encrypt certificates on NetScaler.

His post goes into way more detail…  But, the short version is that the script uses a NetScaler Responder policy to intercept the Let’s Encrypt webroot validation requests and answer with the validated response.  This allows the NetScaler to handle the domain validation for the certificates without any modification to the backend web servers.  The script does need to run on a helper Linux VM, but that’s small overhead for free, perpetual certificates.

Ryan’s how-to is easy to follow and complete.  I did run into a couple issues with my specific NetScaler configuration.  Since the troublesome parts of my configuration are common, I wanted to show how to work around these issues.

HTTP to HTTPS Redirect

The first issue comes from my use of a HTTP to HTTPS redirect vServer.  This takes any port 80 HTTP traffic and sends it to the same address on HTTPS.  Note, that this is only applicable for the Responder based redirect method, not the old school “always down vServer” redirect.

Issue

This becomes an issue when you try to use the script to create a new LE certificate for a domain that the NetScaler doesn’t have an existing certificate for.  In this case, LE can’t validate the domain.  You’ll see the following error with some complaints about HTTP and HTTPS connections failing:

ERROR: Challenge is invalid! (returned: invalid)

This is because the LE Acme server first attempts to connect to the domain with HTTPS.  This fails because there is no valid certificate for the domain.  Next the Acme server attempts to connect with HTTP.  This fails because the redirect simply forwards it back to HTTPS.

Resolution

Fortunately, this is easy to fix.  When the python script was setup, it created a Responder policy to intercept anything sent to “well-known/acme-challenge”.  All we need to do is bind this same Responder policy to the HTTP redirect vServer.  Just do the following:

 

• Navigate to Traffic Management > Load Balancing > Virtual Servers.
• Click on your HTTP to HTTPS redirect vServer to edit it.
• Scroll down to Policies and click on “# Responder Policies”
• Click “Add Binding”
• Enter the following:
  • Select Policy: LE Responder policy (default name is le-responder-pol)
  • Priority: 90 (make sure this is the lowest number priority so it gets hit first)
• Click Bind

That should be it.  Now the validation will fail at HTTPS and successfully switch to HTTP.

Unified Gateway behind a Content Switch

The second issue I hit is probably a bit more obscure.  I have a single public IP that routes to a Content Switch.  This Content Switch routes connections to several websites as well as a Unified Gateway vServer based on the hostname.

Issue

The Responder policy for the Let’s Encrypt script is bound to the Content Switch, but it did not work for the Unified Gateway address.  The validation script was unable to reach the “well-known\acme-challenge” URL and fails with another ERROR: Challenge is invalid! (returned: invalid).

Resolution

To fix this, we have to add an additional Content Switch rule for LE validation requests to the Unified Gateway domain name:

• Navigate to Traffic Management > Content Switching > Policies
• Put a checkmark next to your existing Policy for your Unified Gateway address
• Click Add
• Change the name to something that makes sense (Unified Gateway – LetsEncrypt)
• Add the following to the END of the Expression: && HTTP.REQ.URL.CONTAINS(“/.well-known/acme-challenge”)
• Click OK
• Navigate to Traffic Management > Content Switching > Virtual Servers
• Click on your Content Switch vServer to edit it
• Click on “# Content Switching Policies”
• Note the Priority Number on your existing Unified Gateway policy
• Click Add Binding and enter the following:
  • Select Policy: Unified Gateway – LetsEncrypt (or whatever name you used)
  • Priority: Any number lower than the priority for the existing Unfied Gateway vServer
• Click Bind
• Optional: Click Regenerate Priorities to clean up the priority numbers

The Unified Gateway address should now validate.