One of the first hurdles for me to start using Jenkins was coming up with a strategy for managing SSH key’s for connecting to version control (git in my case) as well as ssh connections within jobs for distributing artifacts. What made it difficult was just figuring out how to use certificates within Jenkins because it’s a little bit inconsistent since various plugins do things differently.

<UPDATE 6/19/2014

My mind is blown about how inconsistent the use of credentials is across plugins. Some want you to setup credentials for the individual plugins, others will use the Jenkins Credentials Plugin while others seem to utilize the SSH Agent Plugin – which is fine if the interface to plugins made it obvious how to interact with them!

A perfect example of this is Polling SCM, there is an option to select a credential from the Credentials Plugin but the update doesn’t even work without the SSH Agent Plugin running and then the polling fails anyway!

http://stackoverflow.com/questions/19171180/using-ssh-agent-with-jenkins-while-polling-scm

So… as long as you are ok with using certificates without passwords and not really knowing what in the world is going on within the plugins you might be able to get things working. I’m moving forward with using jenkins but I’m a little wary of this situation.

/UPDATE>

The Credentials plugin that comes standard with Jenkins is very useful however not all plugins will utilize it and the credentials aren’t exposed in all the contexts that you would like them to be. For example, if you are using the Publish Over SSH Plugin you are required to specify the user, private key and passphrase within the configuration of the job, it doesn’t appear to leverage the credentials that are setup in the credentials plugin. Also, if you attempt to use “Execute shell” as a build step and attempt to ssh to another server, it will fail. The trick in this case is to install the “SSH Agent Plugin” and enable it for any job that you would like to use the “Execute shell” as a build step. This is very slick, taking care of the private key and passphrase and is actually integrated with the Jenkins Credentials Plugin.

What I have decided to do is:

  • Generate ssh keys from the command line (on the system I am connecting from)
  • Use Jenkins Credentials Plugin to manage private keys (Jenkins > Credentials) where I can
  • Install the Jenkins SSH Agent Plugin to expose ssh credentials to a job
  • Publish Over SSH Plugin – surprisingly I haven’t been able to get this to work with the Credentials Plugin to manage the private key and passphrase, it requires configuration of the ssh credentials within the job configuration

Setup

  • BitBucket Repo
  • jenkins.nerdnuts.com – CentOS 6.5 Droplet running Jenkins
    • Jenkins 1.565
    • Publish Over SSH Plugin 1.11
    • SSH Agent Plugin
    • Credentials Plugin
  • target.nerdnuts.com – CentoOS 6.5 Droplet running Apache

Using Key-Based Authentication

Basic Concept

  • Generate a key pair (a public & a private key)
  • Distribute the public key to the systems you would like to use key based authentication
  • Use the private key and passphrase to when connecting to the system (in our case managed by the Jenkins Credentials, SSH Agent and Publish Over SSH plugins)

 

Generating SSH keys

Cent OS 6.5, Using a Key-Based Authentication Documentation

First we generate the key pair on the jenkins server:

[root@jenkins jenkins.nerdnuts.com]# sudo -u jenkins ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/var/lib/jenkins/.ssh/id_rsa): /var/lib/jenkins/.ssh/id_rsa_target.nerdnuts.com
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /var/lib/jenkins/.ssh/id_rsa_target.nerdnuts.com.
Your public key has been saved in /var/lib/jenkins/.ssh/id_rsa_target.nerdnuts.com.pub.
The key fingerprint is:
93:3c:e1:c4:69:37:21:ba:2b:e7:57:51:d3:56:6c:e2 jenkins@jenkins
The key's randomart image is:
+--[ RSA 2048]----+
|           . . o.|
|          o o + o|
|          . * +  |
|          * .. E |
|        . S .    |
|     . o.        |
| . o .           |
| + .             |
| ..              |
+-----------------+
[root@jenkins jenkins.nerdnuts.com]#

Distribute the public key

How you distribute the public key depends on what system you are distributing to, for this example I am creating an SSH connection between two Cent OS 6.5 machines to distribute artifacts.

SSH to the Apache server that we would like to connect to.

Create a jenkins user to use for pushing content:

[root@target.nerdnuts.com www]# useradd jenkins
[root@target.nerdnuts.com www]# passwd jenkins
Changing password for user jenkins.
New password:
Retype new password:
passwd: all authentication tokens updated successfully.
[root@target.nerdnuts.com www]#

Change to the jenkins user

[root@target.nerdnuts.com ~]# su jenkins
[jenkins@target.nerdnuts.com root]$ cd ~/.ssh/
bash: cd: /home/jenkins/.ssh/: No such file or directory
[jenkins@target.nerdnuts.com root]$
[jenkins@target.nerdnuts.com root]$ mkdir ~/.ssh/
[jenkins@target.nerdnuts.com root]$ chmod 700 ~/.ssh
[jenkins@target.nerdnuts.com root]$ vi ~/.ssh/authorized_keys

Since the file didn’t exist we can just paste the contents of the public key from the jenkins serverver (/var/lib/jenkins/.ssh/id_rsa_target.nerdnuts.com.pub), if it already exists we would have to append (add to the end of what is already there).

[jenkins@target.nerdnuts.com root]$ chmod 600 ~/.ssh/authorized_keys

Using Jenkins Credentials

Add domain

Jenkins > Credentials > Add domain

  • Domain Name : target.nerdnuts.com
  • Description : Development Server
  • Click “OK”

Add Credentials

Jenkins > Credentials > target.nerdnuts.com > Add Credentials

  • Kind : SSH Username with private key
  • Scope : Global
  • Username : jenkins
  • Description : jenkins SSH Credentials for target.nerdnuts.com
  • Private Key > From a file on Jenkins master > File : /var/lib/jenkins/.ssh/id_rsa_target.nerdnuts.com
  • Passphrase : The passphrase used when generating the key.

Now this credential is available within jobs in jenkins!

SSH Agent Plugin

If you need to use ssh commands within a build step you can use this plugin to provide the passphrase for the key when jenkins executes the shell script. To use the credential that we have setup we can look at the settings within a job.

Jenkins > {YourJob} > Configure

  • Scroll down to “Build Environment”
  • Check “SSH Agent”
  • Select the Credentials to use

Now those credentials are available for jenkins to connect with to our development server.

Publish Over SSH Plugin

Jenkins > {YourJob} > Configure

  • Add build step > Send files or execute commands over SSH
  • SSH Server
    • Name : target.nerdnuts.com
    • Click ‘Advanced’
    • Check ‘Credentials’
    • Username : jenkins
    • Passphrase : {PrivateKeyPassPhrase}
    • Path to key : /var/lib/jenkins/.ssh/id_rsa_target.nerdnuts.com
  • Transfer Set
    • Source Files : **/**

Troubleshooting

Authenticity of host

This didn’t seem to be an issue:

[root@jenkins jenkins.nerdnuts.com]# sudo -u jenkins ssh jenkins@target.nerdnuts.com
The authenticity of host 'target.nerdnuts.com (123.12.12.35)' can't be established.
RSA key fingerprint is ef:22:44:73:29:55:9a:46:61:a3:44:74:ed:41:a6:49.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'target.nerdnuts.com,123.12.12.35' (RSA) to the list of known hosts.
jenkins@target.nerdnuts.com's password:
[jenkins@target.nerdnuts.com ~]$

Failed to get hostname

SSH: Failed to get hostname [jenkins: jenkins]

Message [Auth fail]

ERROR: Exception when publishing, exception message [Failed to connect session for config [target.nerdnuts.com]. Message [Auth fail]]
Build step 'Send files or execute commands over SSH' changed build result to UNSTABLE
[ssh-agent] Stopped.

Attempt to connect from the command line using the same key and passphrase as we are configuring Jenkins with.

[root@jenkins jenkins.nerdnuts.com]# sudo -u jenkins ssh -i /var/lib/jenkins/.ssh/id_rsa_target.nerdnuts.com jenkins@target.nerdnuts.com
The authenticity of host 'target.nerdnuts.com (123.12.12.35)' can't be established.
RSA key fingerprint is ef:22:44:73:29:55:9a:46:61:a3:44:74:ed:41:a6:49.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'target.nerdnuts.com,123.12.12.35' (RSA) to the list of known hosts.
Enter passphrase for key '/var/lib/jenkins/.ssh/id_rsa_target.nerdnuts.com':
Last login: Sat Jun 7 19:33:20 2014 from 17.17.17.17
[jenkins@target.nerdnuts.com ~]$ exit
logout
Connection to target.nerdnuts.com closed.
[root@jenkins jenkins.nerdnuts.com]#
  • If this Fails, then there is an issue with the setup
    • Did you set the permissions correctly on the target ssh files?
    • Did you create the certs as the appropriate user?
  • If this is successful, then it’s not a problem with the cert or passphrase or configuration of destination being configured with our public key.
    • Are you expecting the Jenkins Credential Plugin to handle this? You need to configure the Publish Over SSH Plugin within the appropriate section of the job, Jenkins > {JobName} > configuration > “Send files or execute commands over SSH” build step
    • Perhaps the key (or path to key) was entered incorrectly?
Leave a reply

Your email address will not be published. Required fields are marked *