Integrate HashiCorp Vault with WSO2 Identity Server

Disclaimer: Recently HashiCorp victim of a supply-chain attack. This however does not change our opinion about the use of the software and the viability of the vendor in general. In general, a security breach or hack can happen to any company. The likelihood of course depends on the way a company approaches security: its security strategy, mitigations and emergency plans. HashiCorp was not the only company attacked. The backdoor also impacted potentially 29.000 other companies who use the Codecov Bash Uploader. In the case of HashiCorp, hackers were able to steal the private key that is used to sign the hashes of download. In other words, with that key a hacker could have changed a download and calculate a new hash. HashiCorp has rotated the keys since and believes that there are (as far as known) no cases of misuse as stated in a publication. They handled it promptly and acted on the situation. The error has to do with the signage of the checksum files, not so much with the software itself. So the impact seems low. Go ahead, try out the Vault by HashiCorp as described in below blog.

Vault from HashiCorp is a product that stores secrets, and that we can integrate with Identity Server. It is a replacement for the Apache Secure Vault implementation that is currently in the WSO2 Identity Server. In this blog we are going to set up Vault and integrate it with WSO2 Identity Server version 5.11.0. This setup is going to be on Windows, but it can easily be done on Linux and MacOS as well.

If the name HashiCorp sounds familiar it might be because it is also the company that developed Vagrant and Packer. Two configuration and deployment tools often used in conjunction with WSO2. If you are unfamiliar with Vault, I would suggest that you look at the documentation that you can find on the site of HashiCorp, they do a great job explaining the concepts. In this blog I will explain some of the details as far as they’re pertaining to the combination of WSO2 Identity Server and Vault. 

Setting up Vault 

But first what we need to do is download and install Vault. As I said, I am going to do this on Windows. I’ll download Vault and install it in a directory, close to the root. Furthermore, what I am going to do is add the Vault installation directory to the path and make it accessible from anywhere. To test this, I am issuing the command vault -version.

This looks to be ok; I am getting a response from Vault. Now we are going to configure Vault. 

Let us start a special version that works locally, the so-called -dev server. This is an in-memory vault that does not use TLS and SHOULD NEVER be used for anything other than testing. Keep the Vault up and running when you walk through this blog.

Start Vault server using vault server -dev

We also need to set the Vault address with the command shown below.

set VAULT_ADDR=http://127.0.0.1:8200

Similarly, set the token to the value shown in the server log output. 
set VAULT_TOKEN=s.dx9VgiWBnb0h6KgqBWQKNBM

Keep in mind that when you stop and start the vault server, you empty it since it is an in-memory vault. A regular vault writes to disk. This is however a more complex setup.

We are starting Vault and leave it open in the command window. In a second window we are requesting the status.

Setup WSO2 Identity Server 

Download the version 5.11.0 of the Identity Server and install it on your computer. In this blog, I am not going to describe the whole process of setting up a WSO2 product. In case you need more information about this you can read our other blogs on how to install WSO2 products. 

Key/value engine 

We are going to start with a KV engine, this of course stands for key value. The KV sub command allows you to set and retrieve, among other things, all the key value pairs stored in Vault. 

We are going to work with a so-called Path, which allows us to store values in an orderly fashion. In other words, using defaults to store values pertaining to Identity Server but also for other WSO2 products or user cases. 

Let us start with adding the key value engine. Enter the following command in the command line: 

vault secrets enable -path=wso2is -version=2 kv

# Add new secret
vault kv put wso2is/keystore_password value=wso2carbon

# Get a secret (To check)
vault kv get -field=value wso2is/keystore_password

Now we need to make sure that the Identity Server will use HashiCorp Vault rather than its own secure vault. Extending the Identity Server with a special jar file that will be used to redirect to HashiCorp. 

To do this, we will download the source code of the HashiCorp Vault Integration OSGI bundle. In this case I am using the zip file. You can also clone it, by using Git or open it with GitHub desktop. 

Take the zip file and unzip it in a directory. I have chosen to put the files in a subdirectory of the vault installation itself. We will build the jar file from scratch using the following command:

mvn clean install 

This build will take a couple of minutes to finish. In the target directory you will find the OSGI bundle that was just created and that we are going to copy to the dropins directory of the Identity Server. 

Copy the target/org.wso2.carbon.securevault.hashicorp-1.0.jar file to [IS_HOME]/repository/components/dropins/ directory.

We also add the special vault-java-driver-5.1.0.jar to the [IS_HOME]/repository/components/lib/ directory.

Redirecting

We need to add the following elements to the [IS-HOME]/repository/conf/security/secret-conf.properties file and set following configurations. For the -dev server we use we changed the https://127.0.0.1:8200 to http://127.0.0.1:8200.

In a production environment, you should always use the vault address with HTTPS/TLS enabled.

keystore.identity.location=repository/resources/security/wso2carbon.jks
keystore.identity.type=JKS
keystore.identity.store.password=identity.store.password
keystore.identity.store.secretProvider=org.wso2.carbon.securevault.DefaultSecretCallbackHandler
keystore.identity.key.password=identity.key.password
keystore.identity.key.secretProvider=org.wso2.carbon.securevault.DefaultSecretCallbackHandler
carbon.secretProvider=org.wso2.securevault.secret.handler.SecretManagerSecretCallbackHandler

secVault.enabled=true
secretRepositories=vault
secretRepositories.vault.provider=org.wso2.carbon.securevault.hashicorp.repository.HashiCorpSecretRepositoryProvider
secretRepositories.vault.properties.address=http://127.0.0.1:8200
secretRepositories.vault.properties.namespace=ns1
secretRepositories.vault.properties.enginePath=wso2is
secretRepositories.vault.properties.engineVersion=2

Log4J2

When we want to debug the new vault addition, we can add the following lines to the [IS_HOME]/repository/conf/log4j2.properties file. Setting the level to debug will log debugging information when invoked. 

logger.org-wso2-carbon-securevault-hashicorp.name=org.wso2.carbon.securevault.hashicorp
logger.org-wso2-carbon-securevault-hashicorp.level=INFO
logger.org-wso2-carbon-securevault-hashicorp.additivity=false
logger.org-wso2-carbon-securevault-hashicorp.appenderRef.CARBON_CONSOLE.ref = CARBON_CONSOLE

Then append org-wso2-carbon-securevault-hashicorp to the loggers list elsewhere in the log4j2.properties file. Look for the loggers = line and add this logger-name at the end.

loggers = AUDIT_LOG, trace-messages, ... ,org-wso2-carbon-securevault-hashicorp

Adding values to the vault

When we want to use values stored in Vault, we’ll need to add them first. This whole blog is written around the values being stored in the path ‘wso2is’ key vault. So, you need to add them to the vault and make changes to the deployment.toml.

Lets first add two values to Vault.

vault kv put wso2is/admin_password value=wso2carbon
vault kv put wso2is/database_password value=wso2carbon

Now we need to change the deployment.toml of WSO2 Identity Server. This can be found in [IS_HOME]/repository/conf/ directory. Open this file with an editor and add these lines at the end of the file (if the [secrets] marker already exists then do not add the marker). 

[secrets]
admin_password = ""
keystore_password = ""
database_password = ""

Add the encrypted password alias to the relevant sections in the deployment.toml file by using a placeholder: $secret{alias}. For example (and we are not showing all lines):

[super_admin]
username="admin"
password="$secret{admin_password}"

ystore.primary]
file_name = "wso2carbon.jks"
password = "$secret{keystore_password}" 

[database.identity_db]
type = "h2"
...
password = "$secret{database_password}"

Starting Identity Server

We can now start Identity Server. You will be prompted for the keystore password before start. Enter the value wso2carbon (the default value) to start the Identity Server. You can place the password in plain text in a password-persist file (password-persist.txt for Windows), so it will read it from there. This is however less secure. If you want to read that from Vault as well, you should create custom Java code (a custom DefaultSecretCallbackHandler / Custom CallBack Handler) that will make a connection to the Vault API and read the value from there. This will make it more secure, since it also utilizes the Vault for the storage of this value.