SSH Login Automation, Tunneling, Authprogs, and More

Bri Hatch Personal Work
Onsight, Inc
bri@ifokr.org
ExtraHop Networks
bri@extrahop.com


Copyright 2013-2014, Bri Hatch, Creative Commons BY-NC-SA License

SSH Pubkeys

What are they?

Generating a pubkey

$ ssh-keygen -b 2048 -C foo@example.com -f /tmp/key
Generating public/private rsa key pair.
Enter passphrase (empty for no passphrase): (enters passphrase)
Enter same passphrase again: (enters passphrase again)
Your identification has been saved in /tmp/key.
Your public key has been saved in /tmp/key.pub.
The key fingerprint is:
8b:07:14:c3:6e:f7:17:4c:78:0f:fb:94:71:ce:01:4b foo@example.com
The key's randomart image is:
+--[ RSA 2048]----+
|     .o    o E.  |
|     .o.  . * o..|
|     o.    o = *.|
|      ...   o + o|
|      ..S.   +   |
|                 |
+-----------------+

Generating a Pubkey (continued)

Default Public/Private key is in ~/.ssh/id_rsa:

$ cat ~/.ssh/id_rsa
-----BEGIN RSA PRIVATE KEY-----
Proc-Type: 4,ENCRYPTED
DEK-Info: AES-128-CBC,5543DF16C874A2B95AD8BCAFE383A8A9

N9K0VKkAROsq1siAPNgeD4enZiISszeP7LXj7tzvMkJ3gweGW7okvIhR/5iTIVih
CjGCXb727YiDX/dHm0B8mPYTUPVHQ+xu+0FpuhKNP70erhI3jsimDO9PR3xZc6lx
h2UaTKd7C7WwI9kyOdLrGrPIgAuoVpSpmJNOGjLZK54gooA7LfWZtK/o4CMDD1vf
hlQ064qhIabk/j7FZ8LpjLf5TbEATaPRmpLrZOKABKVz0VfXEsw7sa8jqutNwB5e
...

$ ls -l ~/.ssh/id_rsa
-rw------- 1 xlr xlr 1766 Jan 21 11:50 id_rsa
This file is nicely wrapped at 64 columns.

Generating a Pubkey (continued)

Public key by itself is also in ~/.ssh/id_rsa.pub:

$ cat ~/.ssh/id_rsa
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD3k5ng5Fn0089ed99z8Yr
elv4FqYbN4VNespZx4tiY/jPleNVxPVId8WnxYOn/C4cmeAFltmEV8xDRr7
bnnDfmRqGonrH2untJapB2p6cvv4u6VrG+UyfeZjJ2cPhCDdYzFqUCp+bPC
07ZbXpfX5a4MCHj+wI5bmIBy2TllxFPE8HESCEpY7cO2yFCMn6kn1cJJSw5
87BMgXr35TA+ZYj2Tur8NzQTSoSaT1lgqfCMoNLzsFf4V9wJAcvEI1WdvVP
bdNOZBnVM7wIfUqmj7Iy8H03fm4MFkO5c2zBKVW9EHl43F19Ij0yZHZq0yg
shSueK7PDQN07Gwak7NpYlzmBX foo@example.com

$ ls -l ~/.ssh/id_rsa.pub
-rw-r--r-- 1 xlr xlr 1766 Jan 21 11:51 id_rsa.pub
This file is one big long line, though it is wrapped here in the presentation.

Logging in with Pubkeys

Connecting with your pubkey:

$ ssh somemachine -v -i /tmp/key
OpenSSH_5.9p1 Debian-5ubuntu1, OpenSSL 1.0.1 14 Mar 2012
debug1: Reading configuration data /home/xlr/.ssh/config
...
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,password
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /tmp/key
debug1: Authentications that can continue: publickey,password
debug1: Next authentication method: password
xlr@somemachine's password: 

That wasn't an improvement...

Trusting pubkeys

You need to add your pubkey's public key to the ~/.ssh/authorized_keys file on the SSH server.

$ scp /tmp/key.pub somemachine:key.pub
$ ssh somemachine
xlr@somemachine's password: (types password)

somemachine$ mkdir ~/.ssh; chmod 700 ~/.ssh
somemachine$ touch ~/.ssh/authorized_keys
somemachine$ chmod 600 ~/.ssh/authorized_keys
somemachine$ cat key.pub >> ~/.ssh/authorized_keys
somemachine$ exit

Trusting pubkeys (continued)

Or, to be lazier, just use ssh-copy-id:

$ ssh-copy-id -i /tmp/key.pub somemachine
xlr@somemachine's password: (types password)
Now try logging into the machine, with ssh xlr@somemachine, and check ~/.ssh/authorized_keys to make sure we haven't added extra keys that you weren't expecting.

Logging in with Pubkeys (take two)

Connecting with your pubkey:

$ ssh somemachine -v -i /tmp/key
OpenSSH_5.9p1 Debian-5ubuntu1, OpenSSL 1.0.1 14 Mar 2012
debug1: Reading configuration data /home/xlr/.ssh/config
...
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,password
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /tmp/key
Enter passphrase for key '/tmp/key': (types pubkey password)
somemachine$ 

Default pubkey files

  • If no pubkey is specified on command line or config file, ssh tries these:
    • ~/.ssh/id_dsa
    • ~/.ssh/id_ecdsa
    • ~/.ssh/id_rsa
  • SSH only tries three keys before falling back to other authentication methods. Using IdentiesOnly yes can help.

SSH Agent

Trading a password for a pubkey password doesn't seem like much improvement.

Instead, stick them in an ssh-agent.

$ eval $(ssh-agent)
Agent pid 24028
$ echo $SSH_AUTH_SOCK
/tmp/ssh-qcqvgHa24027/agent.24027
$ echo $SSH_AGENT_PID
24028

$ ssh-add -l
The agent has no identities.

SSH Agent (continued)

SSH Agent notes:
  • Many distros and X11 servers start your agent automatically.
  • Some even have alternate daemons, such as gnome-keyring-daemon that may also support additional protocols, e.g. gpg.
  • Some daemons pretend keys are loaded before they are.
  • Some may ask for passphrases on the GUI, even when you're connected remotely

SSH Agent (continued)

Adding keys to the agent

$ ssh-add -l
The agent has no identities.

$ ssh-add /tmp/key
Enter passphrase for /tmp/key: (passphrase)
Identity added: /tmp/key (/tmp/key)

$ ssh-add -l
2048 8b:07:41:c3:e6:f7:17:c4:78:0f:fb:94:71:ce:01:4b /tmp/key (RSA)

SSH Agent (continued)

And now, log in!

$ ssh -v somemachine -i /tmp/key
...
debug1: Authentications that can continue: publickey,password
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /tmp/key
debug1: Server accepts key: pkalg ssh-rsa blen 279
debug1: Authentication succeeded (publickey).
somemachine$ 
Look ma, no password!

SSH Agent Security Ramifications

  • Root on your machine can use your agent as you.
  • Agent does not 'expire' passwords by default.
    • Select (ssh-add|ssh-agent) -t lifetime to automatically purge credentials.
    • ssh-add -x locks your agent with a password, ssh-add -X unlocks.
    • Delete all saved keys with ssh-add -D
    • Delete specific keys with ssh-add -d pubkeyfile

Agent Forwarding

You can forward your agent to the machines to which you connect.

$ ssh somemachine 'ssh-add -l'
Could not open a connection to your authentication agent.

$ ssh -A somemachine 'ssh-add -l'
2048 8b:07:41:c3:e6:f7:17:c4:78:0f:fb:94:71:ce:01:4b /tmp/key (RSA)

SSH Bouncing

Say you need to ssh to Epsilon3 before you can ssh to Greatmachine.

$ ssh -A -ttt epsilon3 ssh -ttt greatmachine
greatmachine$ 

Notes:

  • Assumes your pubkey is trusted on both machines
  • Your agent is still forwarded on epsilon3
  • A pain to type every time...

SSH Bouncing (continued)

Simplify using ProxyCommand and netcat.

$ cat ~/.ssh/config
Host greatmachine
ProxyCommand ssh -q epsilon3 'nc %h %p'

$ ssh greatmachine
greatmachine$
Notes:
  • Easier to type
  • No agent available on intermediate machine
  • %h and %p in the ProxyCommand are replaced with the actual host and port

Port Forwards

  • Bind a local port, tunnel through and out from destination
  • Bind a remote port, tunnel through and out from client
  • Usable locally only by default
    • Use GatewayPorts yes to allow non-local use.

Local Port Forwards

Tunnel something from our client out through the server.

$ ping -q -c 10 192.168.15.68
PING 192.168.15.68 (192.168.15.68) 56(84) bytes of data.
10 packets transmitted, 0 received, +1 errors, 100% packet loss

$ ssh -L 8080:192.168.15.68:80 -Nf ssh-server
$ curl http://localhost:8080/index.html
<head><title>Welcome to 192.168.15.68!</title>
<body> ...

  • You can point anything at port 8080, e.g. an actual web browser
  • -Nf prevents it from giving you a shell, and forks off in the background. For older ssh versions, you can use -f ssh-server sleep +99d to emulate.

Remote Port Forwards

Tunnel things from the server out through the client.

work$ ssh -R 8080:127.0.0.1:22 -N myhouse 

Later...

myhouse$ ssh -p 8080 policy_violator@localhost
policy_violator@localhost's password: (enters password)
work$

Don't do this at home^H^H^H^H work!

SSH SOCKS Proxy

SSH can even bind a SOCKS proxy, which dynamically forwards connections without hard coding a bunch of local ports.

$ ssh -Nf -D 9999 ssh-server
$ curl --socks5 localhost:9999 http://192.168.15.68/

Similarly, you could configure your browser to use a SOCKS proxy on localhost:9999

Authorized_keys restrictions

You can add restrictions to the entries in your ~/.ssh/authorized_keys file.

$ cat ~/.ssh/authorized_keys
ssh-rsa AAAANzaC1kc3MAAACBANJGoaoq+1gWhx781810vvf...EQQx7= key1@example.com
no-port-forwarding ssh-rsa AAAAIpSVdm3aqGSMOgQ7P1...Pm1X8= key2@example.com
no-agent-forwarding,no-X11-forwarding ssh-rsa AAA...xGGor= key3@example.com

Authorized_keys restrictions (continued)

Complete options are in the sshd man page. Some of the most common options:
  • no-X11-forwarding, no-agent-forwarding, no-port-forwarding
  • permitopen="host:port"
  • from="..."
    • Allows you to restrict which machines are allowed to use this pubkey
  • command="..."
    • Denies direct shell access and instead runs the command you specify

Command Restrictions

This command= restriction sounds great! Prevent your keys from being used for any willy-nilly command.

Unfortunately:

  • You only can specify one command, not multiple options like from
  • Putting security rules in authorized_keys is painful to the eyes

Enter authprogs.

Authprogs

Authprogs is an SSH command authenticator.
  • Decides if the specified command should run or be rejected
  • Has some minimal protocol support, e.g. scp today
  • Configuration is in ~/.ssh/authprogs.yaml and files in ~/.ssh/authprogs.d directory.
  • Can install keys into authorized_keys automagically
  • Running at ExtraHop since 2012
  • Learns from mistakes of original perl version from 2003
  • Released as Open Source (GPL Licensed) at SeaGL in 2013

Installation

$ sudo pip install authprogs

Working on getting it packaged into Debian.

Installing keys

$ authprogs --install_key /path/to/id_rsa.pub
$ tail -1 ~/.ssh/authorized_keys
command="authprogs --run",no-port-forwarding ssh-rsa AAAAOg...Pm1X8= user@example.com

$ authprogs --keyname push --install_key /path/to/push.pub
$ tail -1 ~/.ssh/authorized_keys
command="authprogs --keyname push --run",no-port-forwarding ssh-rsa AAAAGxNo22Hx...

Installation automatically disables port forwarding. Tweak manually if desired.

Configuration Examples

Simplest version - a bunch of allowed commands.

$ cat ~/.ssh/authprogs.yaml
allow:
    - command: /bin/tar czvf /backups/www.tgz /var/www/
    - command: /usr/bin/touch /var/www/.backups.complete

Configuration Examples (continued)

Keyname restrictions, scp support
$ cat ~/.ssh/authprogs.d/backups.yaml
keyname: backups
allow:
    - command: /bin/tar czvf /backups/www.tgz /var/www/
    - command: /usr/bin/touch /var/www/.backups.complete
--
keyname: backup_snagger
allow:
    - rule_type: scp
      allow_download: true
      allow_recursion: true
      files:
        - /backups/www.tgz
        - /etc/

Configuration Examples (continued)

Trailing args, IP restrictions, PCRE command matching.

$ cat ~/.ssh/authprogs.d/admins.yaml
from:
    - 192.168.1.1
    - 192.168.1.2
allow:
    # Allow unrestricted ls
    - command: /bin/ls
      allow_trailing_args: true
--
from: [192.168.0.10, 192.168.0.15, 172.16.3.3]
allow:
    - command: ^sudo\s+/etc/init.d/apache2\s+(reload|restart)$
      pcre_match: true

Limitations

  • Nothing that requires bash-like processing
    • No multiple-command support
    • No shell constructs like for or while
    • No glob expansion
  • Not whitespace aware

Future

Things that may get implemented:
  • Rsync support
  • Environment variables
  • Specify authorized_keys restrictions at pubkey install time
  • CIDR support
  • Shell regex command matching

It's Open Source... what do you want to add? ;-)

Verifying Host Keys

Ever seen this?

$ ssh somemachine
The authenticity of host 'somemachine' can't be established.
RSA key fingerprint is b8:5a:1e:47:87:48:58:14:6b:d2:52:3c:50:55:f6:f7
Are you sure you want to continue connecting (yes/no)? 

Verifing the Host Key after login

So you have no idea if that's really the server key, you're taking it on faith. It's too late to survive a MITM. But it's not to late to see if you were subject to one!

$ ssh somemachine
The authenticity of host 'somemachine' can't be established.
RSA key fingerprint is b8:5a:1e:47:87:48:58:14:6b:d2:52:3c:50:55:f6:f7
Are you sure you want to continue connecting (yes/no)? yes
xlr@somemachine's password: (enters password)

somemachine$ ssh-keygen -l -f /etc/ssh/ssh_host_rsa_key.pub
2048 b8:5a:1e:47:77:a8:f5:14:bc:d2:25:99:50:55:f6:f7 somehost (RSA)

If it's different, call the admininistrator!
(And change your passwords....)

Verifying Host Keys Visually

What follows is the exact same commands as before, but adding VisualHostKey for pretty pictures.

$ ssh -o VisualHostKey=yes somemachine
The authenticity of host 'somemachine' can't be established.
RSA key fingerprint is b8:5a:1e:47:87:48:58:14:6b:d2:52:3c:50:55:f6:f7
+--[ RSA 2048]----+
|    .=*o..o      |
|     +o+ . .     |
|    o =.o   . .  |
|      + .        |
|     o .         |
|    .            |
+-----------------+
Are you sure you want to continue connecting (yes/no)? yes
xlr@somemachine's password: (enters password)

Verifying Host Keys Visually (continued)

Check server's fingerprint.

somemachine$ ssh-keygen -v -l -f /etc/ssh/ssh_host_rsa_key.pub
2048 b8:5a:1e:47:77:a8:f5:14:bc:d2:25:99:50:55:f6:f7 somehost (RSA)
+--[ RSA 2048]----+
|         .   .   |
|        S   . .  |
|         = o o   |
|        o * = . .|
|         + * +E +|
|          . ++=B+|
+-----------------+
Yes, that art is different. As you'll see on the next slide, the fingerprints were different this whole time. It was just hard to notice it.

Verifying host keys visually (continued)

  • Here are the fingerprints that were presented on previous slides.

    b8:5a:1e:47:87:48:58:14:6b:d2:52:3c:50:55:f6:f7
    b8:5a:1e:47:77:a8:f5:14:bc:d2:25:99:50:55:f6:f7
    
  • There are tools that can generate keys with similar-looking fingerprints.
  • Takeaway: Visual mode is helpful!

Showing art on all connections...

$ head -3 ~/.ssh/config
# Global configuration options
VisualHostKey yes
NumberOfPasswordPrompts 1

$ ssh somemachine
Host key fingerprint is b8:5a:1e:47:87:48:58:14:6b:d2:52:3c:50:55:f6:f7
+--[ RSA 2048]----+
|    .=*o..o      |
|     +o+ . .     |
|    o =.o   . .  |
|      + .        |
|     o .         |
|    .            |
+-----------------+
xlr@somemachine's password:

This allows you to abort if the art is different - even if someone has modified your ~/.ssh/known_hosts file!

Uh oh, this is bad...

Seen this before?
$ ssh -o VisualHostKey=yes somemachine
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
b8:5a:1e:47:77:a8:f5:14:bc:d2:25:99:50:55:f6:f7
Please contact your system administrator.
Add correct host key in /home/xlr/.ssh/known_hosts to get rid of this message.
Offending RSA key in /home/xlr/.ssh/known_hosts:14
  remove with: ssh-keygen -f "/home/xlr/.ssh/known_hosts" -R somemachine
RSA host key for localhost has changed and you have requested strict checking.
Host key verification failed.

Uh oh, this is bad. (continued)

Various things can cause this warning:
  • Non-security problems
    • You're hitting the wrong server, e.g. a DNS/DHCP change.
    • The machine has been reinstalled
    • The machine had new keys generated intentionally
  • Security issues
    • The machine has been compromised
    • Someone is doing a MITM against you
    • Someone mucked with your ~/.ssh/known_hosts
  • Actions
    • Talk to the server administrator
    • Pretend everything's OK and delete the old known_hosts key
    • Run away, RUN AWAY!

Verifying Host Keys with SSHFP

Thus far our host key verification choices are
  • Get the host key fingerprint out-of-band ahead of time
  • Verify it matches after it's too late
  • Blind faith

Verifying Host Keys with SSHFP (continued)

One Solution: Store host keys in DNS.
  • New RR type, SSHFP, stores fingerprints.
  • Generate the DNS data with ssh-keygen, and stick into DNS.
    $ ssh-keygen -r host.example.com \
                 -f /etc/ssh/ssh_host_rsa_key.pub 
    somemachine IN SSHFP 1 1 8a181c1f137559eefc7b0c35fceb357af3651683
    
  • Use VerifyHostKeyDNS to query DNS for keys
    • yes: automatically trust DNS (if secure)
    • ask: show DNS results and still ask to trust
    • no: Don't look up SSHFP records at all

Verifying Host Keys with SSHFP (continued)

If you're in ask mode, or don't have DNSSEC enabled, you'll still need to answer the question, but you'll get an indication that the key is likely valid.

$ ssh -o VisualHostKey=yes somemachine
The authenticity of host 'somemachine' can't be established.
RSA key fingerprint is b8:5a:1e:47:87:48:58:14:6b:d2:52:3c:50:55:f6:f7
Matching host key fingerprint found in DNS.
Are you sure you want to continue connecting (yes/no)? 

Other options

  • Push out authoritative /etc/ssh/ssh_known_hosts files to ssh client machines.
    • Perhaps build it via ssh-keyscan
    • Push via your configuration management tool of choice, or scp, etc.
    • Maybe restrict the push using authprogs ;-)

Questions?

Q&A

Thanks!

Git repository: http://github.com/daethnir/authprogs
Python PyPI package https://pypi.python.org/pypi/authprogs/
Presentation: http://www.ifokr.org/bri/presentations/lfnw-2014-ssh/

PersonalWork

Bri Hatch
Onsight, Inc
bri@ifokr.org

Bri Hatch
ExtraHop Networks
bri@extrahop.com

Copyright 2013-2014, Bri Hatch, Creative Commons BY-NC-SA License