Task Automation with DoJobber and Python

Bri Hatch Personal Work

Onsight, Inc
bri@ifokr.org ExtraHop Networks
bri@extrahop.com

Copyright 2018, Bri Hatch, Creative Commons BY-NC-SA License

Bri Hatch	Personal	Work
Onsight, Inc bri@ifokr.org	ExtraHop Networks bri@extrahop.com

Audience

Some Linux/Unix experience
Some Python experience
Folks with real problems to solve

Automation Tools

Orchestration/Configuration Management Tools

Our (initial) Problem to solve

ExtraHop's initial DoJobber use case: offboarding

Our (initial) Problem to solve (cont)

And it grew

Our (initial) Problem to solve (cont)

Still growing...

Idempotency

Most critical feature of any CM tool!
Running the tool multiple times will not change the result

Idempotency (cont)

Idempotency Examples:

adding 0
multiplying by 1
pressing the elevator floor button

Idempotency (cont)

Idempotency Examples:

adding 0
multiplying by 1
pressing the elevator floor button
pressing the elevator door close button

Idempotency (cont)

Idempotency Examples:

adding 0
multiplying by 1
pressing the elevator floor button
pressing the elevator door close button
pressing the walk button

Ansible Example

- name: install packages
  apt: pkg={{ item }} state=present install_recommends=no
  with_items:
    - python-authprogs
    - python-crypto
    - virtualenvwrapper
  tags:
    - apt
- name: create service user
  user:
      name: wendell_bagg
      comment: "Foo Service User"
      shell: /bin/tcsh
      system: yes

Pros: easy, terse, ansible galaxy.
Cons: magic purple smoke

Ansible Example

- name: install packages
  apt: pkg={{ item }} state=present install_recommends=no
  with_items:
    - python-authprogs
    - python-crypto
    - virtualenvwrapper
  tags:
    - apt
- name: create service user
  user:
      name: wendell_bagg
      comment: "Foo Service User"
      shell: /bin/tcsh
      system: yes

Pros: easy, terse, ansible galaxy.
Cons: magic purple smoke

(Sunday 2018-04-29 15:00, CC-236, Mark Foster)

Idempotency (redux)

How does this work?

- name: create service user
  user:
      name: wendell_bagg
      comment: "Foo Service User"
      shell: /bin/tcsh
      system: yes

Does it literally create an account each time? Add entry to /etc/passwd? No, it checks first to see if the account exists. If not, only then does it create it.

DoJobber Pattern: Jobs

The DoJobber unit of work: a Job that does the minimum amount of work.

Jobs are python classs with two methods, Check and Run, called as follows:

Check
Run (if necessary)
(re)Check

Check/Recheck are considered successful if they do not throw an Exception.

DoJobber Add User Example

class CreateUser(Job):
  """Create our user's account."""

  def Check(self, *args, **kwargs):
    """Verify the user exists."""
    import pwd
    pwd.getpwnam(kwargs['username'])

  def Run(self, *args, **kwargs):
    """Create user given the commandline username/gecos arguments."""
    import subprocess
    subprocess.call([
      'sudo', '/usr/sbin/adduser',
      '--shell', '/bin/tcsh',
      '--gecos', kwargs['gecos'],
      kwargs['username'])

Job CreateUser's Check and Run methods

DoJobber Add User Example

class CreateUser(Job):
  """Create our user's account."""

  def Check(self, *args, **kwargs):
    """Verify the user exists."""
    import pwd
    pwd.getpwnam(kwargs['username'])

  def Run(self, *args, **kwargs):
    """Create user given the commandline username/gecos arguments."""
    import subprocess
    subprocess.call([
      'sudo', '/usr/sbin/adduser',
      '--shell', '/bin/tcsh',
      '--gecos', kwargs['gecos'],
      kwargs['username'])

Variables come in via args and kwargs.

Dependency Management

$ cat deploy_script
#!/bin/bash
set -u
set -e
id wendell_bagg >/dev/null 2>&1 || /usr/sbin/adduser \
   --shell /bin/tcsh \
   --gecos "Foo Service User" wendell_bagg
mkdir -p /srv/service_directory || /bin/true
rsync -C --chmod=Dg+s,ug+w,Fo-w,+X -tlrvCO \
   /src/git/service/ \
   /srv/service_directory
chown -R wendell_bagg /srv/service_directory
apt-get install python-authprogs
...

$ wc -l deploy_script.sh
41900 730927 2004121    deploy_script.sh

The problem with the monolithic shell script is that it requires lots of tests and stops as soon as it hits the first error.

However in this example we actually only have the following dependencies: chown is dependent on user, chown is dependent on rsync, rsync is dependent on directory creation.

If we stop at the first error we don't do other non-blocked actions.

DoJobber Dependencies

class CreateUser(Job):
    ...

class CreateDir(Job):
    ...

class Rsync(Job):
    DEPS = (CreateDir,)
    ...

class ChownFiles(Job):
    DEPS = (CreateUser, Rsync)

class PackageInstall(Job):
    ...

DoJobber Graph Example

Interactive Time

Let's investigate dojobber_example.py:

Run with --help (note movie)
Run with --display_prerungraph
View ValidateMovie
Run with --movie Zardoz
Run with --battery_state charged --couch_space
Fix dependency bug

Setup

Configure, set arguments, and run it!

# Set your arguments manually:
   dojob = dojobber.DoJobber()
   dojob.configure(RootJob)
   dojob.set_args('arg1', 'arg2', foo='foo', bar='bar', ...)
   dojob.checknrun()


# Or via argparse / optparse automagically:
   myparser = argparse.ArgumentParser()
   myparser.add_argument('--movie', dest='movie',
                         help='Movie to watch.')
   args = myparser.parse_args()
   ...
   dojob = dojobber.DoJobber()
   dojob.configure(RootJob)
   dojob.set_args(**args.__dict__)
   dojob.checknrun()

Setup

Configure, set arguments, and run it!

# Set your arguments manually:
   dojob = dojobber.DoJobber()
   dojob.configure(RootJob, ...)
   dojob.set_args('arg1', 'arg2', foo='foo', bar='bar', ...)
   dojob.checknrun()


# Or via argparse / optparse automagically:
   myparser = argparse.ArgumentParser()
   myparser.add_argument('--movie', dest='movie',
                         help='Movie to watch.')
   args = myparser.parse_args()
   ...
   dojob = dojobber.DoJobber()
   dojob.configure(RootJob, ...)
   dojob.set_args(**args.__dict__)
   dojob.checknrun()

Setup

Configure, set arguments, and run it!

# Set your arguments manually:
   dojob = dojobber.DoJobber()
   dojob.configure(RootJob, ...)
   dojob.set_args('arg1', 'arg2', foo='foo', bar='bar', ...)
   dojob.checknrun()


# Or via argparse / optparse automagically:
   myparser = argparse.ArgumentParser()
   myparser.add_argument('--movie', dest='movie',
                         help='Movie to watch.')
   args = myparser.parse_args()
   ...
   dojob = dojobber.DoJobber()
   dojob.configure(RootJob, ...)
   dojob.set_args(**args.__dict__)
   dojob.checknrun()

Checknrun

DoJobber configure arguments

configure takes some keyword arguments:

verbose: output a line per Job showing Check/Run status
debug: same as verbose but with full stacktrace on exception
no_act: only run Check methods
cleanup: run Cleanup upon checknrun completion (default)

Interactive Time (redux)

More passes at dojobber_example.py:

-v --no-x11
-d --no-x11

Special Jobs

DummyJobs have neither Check nor Run. Useful for holding dependencies.

    class PlaceHolder(DummyJob):
        DEPS = (Dependency1, Dependency2, ...)

RunonlyJobs have no Check. If the Run does not raise an exception, it is considered successful.

    class RemoveDangerously(RunonlyJob):
        DEPS = (UserAcceptedTheConsequences,)

        def Run(...):
            os.system('rm -rf /')

Local Storage

Setting values in the self.storage dictionary allows you to store state between Check and Run phases of a single Job.

class UselessExample(Job):

  def Check(self, *dummy_args, **dummy_kwargs):
    if not self.storage.get('sql_username'):
      self.storage['sql_username'] = (some expensive API call)
    (check something)

  def Run(self, *dummy_args, **kwargs):
    subprocess.call(COMMAND + [self.storage['sql_username']])

This storage is not available to any other Jobs!

Usually employed when you have some expensive setup you wish to cache.

Global Storage

Setting values in the self.global_storage dictionary allows you to store state between all subsequently executed Jobs.

  # Store the number of CPUs on this machine for later
  # Jobs to use for nefarious purposes.

  class CountCPUs(Job):
    def Check(self, *args, **kwargs):
      self.global_storage['num_cpus'] = len(
          [x
           for x in open('/proc/cpuinfo').readlines()
           if 'vendor_id' in x])

  class FixFanSpeed(Job):
     DEPS = (CountCPUs,)
     def Check(self, *args, **kwargs):
       for cpu in range(self.global_storage['num_cpus']):
          ....

Cleanup method

A Job's Cleanup method, if supplied, is run at the end of the checknrun in LIFO order.

class GitRepo(Job):
   ...

   def Run(self, *args, **kwargs):
     self.global_storage['gitrepodir'] = tempfile.mkdtemp(
        prefix='tmp-git-%s-' % kwargs['reponame'])
     os.chdir(self._repodir)
     subprocess.coll(GIT_CLONE_CMD)

   def Cleanup(self, *args, **kwargs):
     shutil.rmtree(self.global_storage['gitrepodir'])

Dynamically creating Jobs

You can modify the DEPS to include new jobs at will.

# Base Job
class SendInvite(Job):
    EMAIL = None
    NAME = None

    def Check(self, *args, **kwargs):
      (do something with self.EMAIL and self.NAME)

# Create a Job per person as an InviteFriends dependency
for person in people:
  job = type('Invite {}'.format(person['name']),
             (SendInvite,), {})
  job.EMAIL = person['email']
  job.NAME = person['name']
  InviteFriends.DEPS.append(job)

Last Slide

Thanks!

Presentation: https://www.ifokr.org/bri/presentations/lfnw-2018-dojobber/

Github: https://github.com/ExtraHop/DoJobber

Installation: pip install dojobber

Personal Work

Bri Hatch
Onsight, Inc
bri@ifokr.org
Bri Hatch
ExtraHop Networks
bri@extrahop.com

LFNW, 2018-04-28