Secrets management in docker-compose

Delegate / Validator Candidates Node Operation
#1

Hi everyone,

Secrets being exposed is an issue we all need to deal with so I want to start a formal thread to discuss approaches.

There is a feature in docker for secrets that you can encrypt and mount in a container. Basically you disable your history temporarily (set +o history to turn it off then set -o history to re-enable) then encrypt the secret with a command and reference it in your docker-compose like this.

 echo "YOUR PASSWORD" | docker secret create <secret name> -

Then reference it like this,

version: '3.1'

services:
  prep:
    image: 'iconloop/prep-node:1910261021xc97f33'
    container_name: "prep"
    environment:
      NETWORK_ENV: "mainnet"    
      PASSWORD_ENCRYPTED: true
      PASSWORD_ENCRYPTED_NAME: <secret name>
....
    secrets:                    
     - <secret name>
secrets:                        # top level secrets block
  <secret name>:
    external: true

And it will mount the secret in /run/secrets as a file that can be read by the container.

The problem is we need it accessible from the entrypoint which we can modify to read from that location. So basically we’d need to modiy the line in the entrypoint that reads the password, ie export PRIVATE_PASSWORD=${PRIVATE_PASSWORD:-"test"} to have an option to read from /run/secrets.

Basically this,

if [[ ${PASSWORD_ENCRYPTED} ]];then
    export PRIVATE_PASSWORD=`cat /run/secrets/${PASSWORD_ENCRYPTED_NAME}`  # private cert key  or keystore file password
fi

Where PASSWORD_ENCRYPTED is a boolean env var that is set to trigger reading from /run/secrets.

Does this sound about right? I would test now by overriding the entrypoint but we don’t have TestNet up and I have also heard of Tom from ICON Viet and Thomas from Blockmove who said they were looking at it and figured we should all be talking about it. Tom found that this feature is only available to docker swarm which shouldn’t be an issue.

“Docker secrets are only available to swarm services, not to standalone containers. To use this feature, consider adapting your container to run as a service. Stateful containers can typically run with a scale of 1 without changing the container code.”

And is the repo that has the Dockerfile and entrypoint public so I can submit a PR? Any feedback welcome. These are the kinds of things as P-Reps we need to be talking about together as nothing is worse than working on a problem only to find that others did the same thing or came up with a different approach.

Ultimately we as P-Reps should be able to submit changes to the entrypoint that we feel will enhance the security and can be incorporated as a feature we can all share. For instance after we get a basic secret encyrption routine going for this, I would like to include a feature to read secrets from hashicorp’s vault and include that as an option in the entrypoint.

Let me know what you think.

References:

#2

I really thank for your efforts. Overall I think that it is a good solution to solve this problem. However, this idea still has the disadvantage of exposing passwords inside docker so if the server is stolen, the password will be exposed. So I think that we can also discuss the following items.

  1. Use KMS (https://www.vaultproject.io/)
  2. When starting Docker, it receives password internally as standard input
#3

Hi Bong,

Thank you for response.

I totally agree that Vault is the way to go and can spearhead this effort. I am running consul right now for services discovery and can help other users with this setup to build a backend for Vault. I am running 3 consul nodes in HA and going to spin up 3 Vault nodes soon as well. Code is ready. The cost should be about 30$ per month using 6 t2.micro instances.

Talking with Ben on your Wednesday and can do a little more research in the interim to see what needs to be done. Off the top of my head, users will also need to setup GPG keys to unseal the vault cluster.

Do you think there is still a case to put together an easier to run / cheaper solution with docker secrets? Almost all the node operators are running mutable infrastructure which would make running consul / vault difficult I think. I can put together templated IaC to help users run Vault but it will be a lot of coordination to get everyone on same page. But like I said, I can spearhead this effort if you think it is the absolute right play.

Definitely would be difficult to coordinate with sub-p-reps though and I fear that if they keep exposed passwords, they might be more inclined to transfer money out of their wallet and not stake.

References