I have recently been spending more time thinking about how Otto should handle “steps” in a CI/CD pipeline. As I mentioned in my previous post on the step libraries concept, one of the big unanswered questions with the prototype has been managing flow-control of the pipeline from a step. To recap, a “step” is currently being defined as an artifact (.tar.gz) which self-describes its parameters, an entrypoint, and contains all the code/assets necessary to execute the step. The execution flow is fairly linear in this concept: an agent iterates through a sequence of steps, executing each along the way, end. In order for a step to change the state of the pipeline, this direction of flow control must be reversed. Allowing steps to communicate changes to the agent which spawned them requires a control socket.

The agent control socket will allow steps to send a fixed number of message types back to the agent during their execution. My current thinking is that the control socket should speak Nanomsg, which puts a little bit more system level requirements on the steps to be able to communicate with that protocol. My first thought was just lines of JSON encoded over the wire, but there are a number of practical problems with trying to send JSON over a unix socket, for example.

Update 2020-10-30: I have since decided to shy away from Nanomsg for the control socket and instead have opted for a small HTTP server listening on a unix socket. I chose this approach to make the debugging and client interactions much easier, even a totally bash-based shell step would be able to interact with it!

For the first implementation, I am planning to have a single long-lived socket for the duration of a pipeline’s execution by the agent. By adding the ipc field to the invocation file (below), I should have the flexibility to allow an agent to create a single IPC socket for each step to avoid any accidental overlap.

---
configuration:
  ipc: 'ipc:///tmp/agent-5171.ipc'
parameters:
  script: 'ls -lah'

The types of messages that come to mind are:

  • Terminate the pipeline
  • Change the pipeline’s running status (e.g. unstable)
  • Capture a variable

The last item really struck me as necessary but I have been struggling with it quite a bit. In a declarative Jenkinsfile there is no provision for setting variables. It wouldn’t otherwise be very declarative! This restriction leads to some confusing hacks in real-world pipelines. The most common hack is to use the script {} block as an escape hatch, such as:

stage('Build') {
    steps {
        sh 'make'

        script {
            def output = readYaml file: 'output.yml'
            sh "./deploy.sh ${output.stage}"
        }
    }
}

There are numerous legitimate reasons to capture and utilize variables inside of a CI/CD pipeline. I want to support variables in some fashion without building a full-on interpreter or sacrificing clarity in the pipeline modeling language.

As I wrestled with the concept, I noticed that my pseudo-code I was writing for how variables might be used looked familiar:

prompt msg: 'What is the best color for a bike shed?', into: 'color'

To me, this looks a lot like Smalltalk. Mmmm Smalltalk. If you have some spare time, and haven’t yet experienced Smalltalk, you should go download Pharo and explore! It’s a wonderful language and development environment, and well-worth experimenting with in your career.

Anyways, back to Otto.

The syntax above would be the prompt step saving some user-provided string (hand-waves right now on how that would manifest in a GUI) and storing it in the color variable.

With variables, storing is one part of the problem, but using is the other much more interesting part. I knew I didn’t want if color == 'red' { } type blocks littering the code, lest a user think that this pipeline language is a programming language for them to build application in! (This is a very real problem with Scripted Jenkins Pipelines).

A related problem I had set aside the day prior was how to handle “block-scoped steps”, such as the following in Jenkins:

stage('Build') {
    steps {
        sh 'make'
        dir('deploy') {
            echo 'Deploying from the deploy/ directory'
            sh './shipit.sh'
        }
    }
}

All steps executed within the dir block are executed with a current working directory of deploy/.

Variable use and block-scoped steps both led me to a very Smalltalk syntax, which honestly has me quite excited to explore further! In Smalltalk there is no control structures in the traditional sense. No if, no for, etc. Instead one can send the ifTrue/ifFalse message to a Boolean:

color = 'red'
    ifTrue: [
        "Great choice!"
    ]
    ifFalse: [
        "Why did you chose wrong?!"
    ]

Fully embracing this Smalltalk-inspired concept would also be convenient to implement. Anything that isn’t a defined step can be looked at like a variable, using an approach similar to #method_missing in Ruby (which is actually just Smalltalk striking again! It’s called the doesNotUnderstand message in Smalltalk).

Exploring what this would look like in a more concrete pipeline snippet:

sh 'ls -lah'
prompt msg: 'Which file should I dump?', into: 'filename'

filename equals: '/etc/password',
    then: [
        echo 'Stop trying to pwn me!'
    ],
    else: [
        # Not sure on this yet, I _think_ I want to avoid raw string interpolation syntax
        format pattern: 'cat {}', with: [filename], into: 'dumpcmd'
        sh script: dumpcmd
    ]

dir 'deploy' [
    echo 'Deploying from the deploy/ directory'
    sh './shipit.sh'
]

# Intentionally drop the `filename` variable, which would go out of scope
# at the end of the stage anyways
drop 'filename'

A couple notes on the above pseudo-code:

  • I’m not yet sold on the syntax. The benefit of this approach rather than copying Smalltalk directly is that this syntax will make it easier support more robust string operations in the future. The other benefit of this syntax is that it makes everything behave step-like, insofar as a stringvariable internal/hidden step could use the parameters, including the two blocks, and just execute the block scoped steps like any other step.
  • The block syntax is intentionally different from the directive syntax (to use Jenkins terminology) of curly braces I think will help make the code more readable.
  • I don’t want to actually implement a full Smalltalk interpreter here, but I am liking that the syntax does keep things (subjectively) simple.

In order to implement block-scope steps, I am planning to refactor some of the step execution code into an agent crate which will allow steps to re-use the logic for executing steps. From a data structure standpoint the invocation file for the dir in the example might look like:

---
configuration:
  ipc: 'ipc:///tmp/foo.ipc'
parameters:
  directory: 'deploy'
  block:
    - symbol: echo
      parameters:
        msg: 'Deploying from the deploy/ directory'
    - symbol: sh
      parameters:
        script: './shipit.sh'

At runtime the process tree on the agent machine would look something like:

.
└── agent
    └── dir
        └── echo

Despite the state of these ideas right now I haven’t actually implemented them! I typically like to sketch out syntax and run through use-cases before I go running into Rust code.

Part of why I am sharing these early thoughts is because I want to make sure my love of Smalltalk is not blinding me to usability issues with this approach. I think this pattern will allow some non-declarative functionality in the pipeline without requiring an actual interpreted language to be used, but these thoughts are still fresh. If you’ve got some thoughts on what could be improved, or pitfalls to be aware of, feel free to join #otto on Freenode, or email me (about)!