How to secure and manage secrets using Google Cloud KMS

Image for post
Image for post
Secrets + Google KMS + Git

Let’s jump right in. We all know it’s a bad idea to store application secrets within our code. So why we are storing there it still? Let’s take an example.

We could store those secrets in a file and add it to the gitignore so it’s not added to version control. But there are a couple of hurdles:

  • How do we manage those secrets?
  • What happens when the local copy is deleted?
  • How do we share it with other developers?
  • How do we manage versioning of those secrets during changes and an audit log of who changed what?

A lot of questions! So we end up storing it within the code since it’s too much complexity to deal with.

For a big application or application which needs a higher level of security, we can use Production grade secret management services like Hashicorp Vault.

In this article, we will look at a decent approach in dealing with secrets while still achieving better security. We are going to achieve this using Google KMS + Git + IAM+ automation.

The idea is not new. This is what we are going to do:

  • We are going to store the encrypted version of plaintext in version control using Google KMS
  • We will use KMS IAM to allow appropriate users to manage secrets for each environment by granting encrypt/decrypt roles
  • We’ll deploy the application with encrypted secret files
  • We will allow permission for the server to decrypt secrets for each environment
  • At runtime, we’ll load encrypted files, decrypt using KMS APIs and use it.

Cloud KMS is a cloud-hosted key management service that lets you manage cryptographic keys for your cloud services. You can generate, use, rotate, and destroy cryptographic keys. Cloud KMS is integrated with Cloud IAM and Cloud Audit Logging so that you can manage permissions on individual keys and monitor how these are used.

So Cloud KMS will encrypt and decrypt our secrets so we don’t have to store the keys. Only an authorized user or a service account can perform encrypt or decrypt operations.

Let’s get started!

Step1: Preparing Secrets

Image for post
Image for post
credentials per each environment

Make sure this folder is not tracked under version control by adding the following line in the .gitignore file:

/credentials/

Here I am using a properties file, but it could be anything like JSON, YAML, etc. Now you can add any sensitive information in these files. I have added the following:

# dev credentials
oauth_client_id=1234
oauth_client_secret
=abcd
api_key=api_123
# ...

Okay, our secrets are ready for hiding.

Step2: Creating KMS Secret Keys

So go ahead and create a key for each environment using this link Creating Symmetric Keys(recommended). It has step by step instructions (different ways) to create those keys. We are creating those keys using the command line like below:

// create key-ring (think of this as grouping)
gcloud kms keyrings create [KEYRING_NAME] \
--location [LOCATION] \
--project live-project-id
// create the encryption key
gcloud kms keys create [KEY_NAME] \
--location [LOCATION] \
--keyring [KEYRING_NAME] \
--purpose encryption \
--project live-project-id

Here I am creating a key for production using the production project id. Repeat this process for each environment by replacing the Project ID for stag and other environments.

Note: You need to have four pieces of information for each key: location keyring cryptokey and project. This information is not sensitive so you can store it in your code or build scripts

Step3: Assigning Permission to use these keys

Check out Using IAM with Cloud KMS for more information. With this, we can achieve the following:

Production Environment:

cloudkms.cryptoKeyEncrypterDecrypter

So in this way, even though the encrypted credentials are stored in version control, other developers won't be able to use them. Note, even those developers can make live deployments without ever needing to know the secrets (more on this later).

Staging Environment:

// for read only
cloudkms.cryptoKeyDecrypter
// for managing
cloudkms.cryptoKeyEncrypterDecrypter

Likewise, you can grant key roles for different environments depending on the need. For the exact commands, refer to Granting Permission in the docs.

Step4: Encrypting Secrets

gcloud kms encrypt --location global \
--keyring secrets-key-ring --key quickstart \
--plaintext-file credentials/stag/credentials.properties \
--ciphertext-file credentials-encrypted/stag/credentials.properties.encrypted

Since it’s a shell gcloud command, you can easily integrate it with any build system to encrypt all files under the credentials folder. For example, I am using gradle for this:

Basically, there are two helper functions:

  • kmsEncryptSecrets takes the src folder to encrypt each file within it and write it to the target folder with .enc (encrypted) extension, and
  • kmsDecryptSecrets which does the reverse process.

So each time we modify secrets, you can call the encrypt helper method with a simple task:

Now the encrypted folder will look like below:

Image for post
Image for post
encrypted credential files

This folder can be added to version control so each time an authorized user changes secrets, a new encrypted file is generated and logs the history in version control.

Similarly, there is a Decrypt Task for the reverse process.

Step4: Using Encrypted Secrets in the deployment

1. Decrypting secrets and passing during deployment:

This approach is good when deployer needed to be very restrictive or process is automated using CD pipeline.

2. Passing encrypted secrets during deployment and decrypting at runtime:

Note: this works best within the Google Cloud Platform. Otherwise, you need to generate a service account so you can use this approach with external providers.

This approach is even more secure since we are not relying on any intermediate user action or a pipeline, but instead only on authorized servers that can decrypt content at runtime.

For example, we can allow the staging server (service account) the ability to decrypt staging secrets and not the ability to decrypt production secrets.

With this approach, even any developer who doesn’t have access to decrypt production secrets can able to perform production deployment and everything still works fine.

Step 5: Using secrets at runtime

For the demo, assume we are going to deploy to AppEngine since it has a default service account generated already. We will grant it the access to decrypt secrets like below:

gcloud kms keys add-iam-policy-binding secrets-enc-key \ 
--project kms-demo \
--location global \
--keyring secrets-key-ring \
--member
serviceAccount:kms-demo@appspot.gserviceaccount.com \
--project kms-demo \
--role
roles/cloudkms.cryptoKeyDecrypter

Thus when the server starts, we could simply load the encrypted file and use the KMS client libraries to decrypt its content.

Step6: KMS Audit Logs

You can enable the audit log using gcloud, but we have seen enough of the command-line way. Alternatively, we can enable this configuration using the Cloud Console UI. From the left menu, choose IAM & admin -> Audit Logs.

Click Cloud Key Management Service and enable Data Read and Data Write and hit Save.

Image for post
Image for post
Google IAM Audit Log Console

That's it! Now if any encrypt, decrypt or any other sorts of operations are performed, an audit log is generated and you can check those in the Logging section under Cloud KMS CryptoKey.

Image for post
Image for post
Audit Logs for IAM operations

As you can see, it has audit logs for all sorts of operations including failures like Invalid permissions, requests etc. It shows which user performed what operation using which key (or if it was done under a service account). That's a pretty neat solution. For more info, read Using Cloud Audit Logging with Cloud KMS.

Conclusion

For a list of kms commands, refer to KMS Commands. Also, check out the sample application for the complete code:

Here are some reference links:

Written by

Senior Software Engineer @payconiq | rameshl.com

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store