How to SSH into a CircleCI build container

source: CircleCI docs

When a CircleCI job runs, it gets a fresh container or VM (Linux, macOS, Windows…) that acts as your machine for the length of the job. Your code is checked out, dependencies installed, tests run, and artifacts built and deployed, one step after another. Each step's output feeds the next, which is what makes it a pipeline.

When a step fails and the logs don't say why, the fastest fix is often to get inside that exact environment and look around. CircleCI lets you do that by rerunning the job with SSH.

Why SSH into a build

• Reproduce a failure in the exact environment it happened in, and try fixes live.

• Check what's installed: tool versions, packages, and whether a cache was restored.

• Find files: where something was written, or why a path doesn't exist.

• Inspect environment variables without printing them in your build logs.

• Check resources when a job is slow or runs out of memory:

  cat /proc/meminfo    # memory (Linux)
  nproc                # CPU cores
  df -h                # disk

A note on environment variables: you could add a step that runs printenv, but then every value ends up in the build logs, visible to anyone who can see the project.

Don't do this: it prints your secrets to the logs

Reading them over SSH keeps them out of the logs. But it also means anyone who can rerun a job with SSH can read your project's secrets, so treat that ability as sensitive.

Step 1: set up an SSH key

CircleCI authorises the SSH session with the keys on your GitHub (or Bitbucket) account. If you already push to GitHub over SSH, you can skip to step 2.

Check for an existing key:

ls ~/.ssh/id_ed25519.pub ~/.ssh/id_rsa.pub 2>/dev/null

If there's none, create one. Ed25519 keys are shorter and faster than RSA, and are what GitHub recommends:

ssh-keygen -t ed25519 -C "you@example.com"

Press Enter to accept the default file (~/.ssh/id_ed25519), and choose a passphrase.

Start the SSH agent and add the key. On macOS, --apple-use-keychain saves the passphrase in your Keychain (it replaces the old -K flag):

eval "$(ssh-agent -s)"
ssh-add --apple-use-keychain ~/.ssh/id_ed25519   # macOS
ssh-add ~/.ssh/id_ed25519                        # Linux

Copy the public key:

pbcopy < ~/.ssh/id_ed25519.pub                   # macOS
cat ~/.ssh/id_ed25519.pub                        # anywhere: copy the output

Add it to GitHub under Settings → SSH and GPG keys → New SSH key, and check it works:

ssh -T git@github.com

Step 2: rerun the job with SSH

Open the failed (or any) job in CircleCI, open the Rerun menu at the top right, and choose Rerun job with SSH.

Rerun job with SSH (older CircleCI interface; the option is in the same place today)

The job runs again with an extra Enable SSH step. Expand it to find the connection command:

The Enable SSH step shows the host and port

You can now SSH into this box if your SSH public key is added:
    $ ssh -p 54782 3.80.12.34

Step 3: connect

Run the command from your terminal:

ssh -p 54782 3.80.12.34

If you have several keys and the connection is refused, point SSH at the right one:

ssh -i ~/.ssh/id_ed25519 -p 54782 3.80.12.34

Step 4: debug

You're now in the same environment as the job, after its steps have run. Some useful starting points:

cd ~/project            # the default working directory for checked-out code
ls -la                  # did checkout and the build produce what you expect?
printenv | sort         # environment variables (stays in your terminal, not the logs)
node --version          # tool versions
cat "$BASH_ENV"         # variables exported by earlier steps

Then rerun the command from the failing step and try fixes. Anything you change only lives in this box, so copy working fixes back into your code or .circleci/config.yml.

Step 5: end the session

The SSH job keeps running until you disconnect and it times out, and it uses credits while it runs. When you're done:

exit

Then cancel the job in the CircleCI UI so it doesn't keep running. See CircleCI's debugging with SSH docs for the current time limits.

Troubleshooting