<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom" ><generator uri="https://jekyllrb.com/" version="3.10.0">Jekyll</generator><link href="https://brokenco.de//feed/by_tag/evergreen.xml" rel="self" type="application/atom+xml" /><link href="https://brokenco.de//" rel="alternate" type="text/html" /><updated>2026-10-31T17:58:32+00:00</updated><id>https://brokenco.de//feed/by_tag/evergreen.xml</id><title type="html">rtyler</title><subtitle>a moderately technical blog</subtitle><author><name>R. Tyler Croy</name></author><entry><title type="html">Choosing the right plugin dependencies</title><link href="https://brokenco.de//2018/09/27/choosing-dependencies-with-the-right-stuff.html" rel="alternate" type="text/html" title="Choosing the right plugin dependencies" /><published>2018-09-27T00:00:00+00:00</published><updated>2018-09-27T00:00:00+00:00</updated><id>https://brokenco.de//2018/09/27/choosing-dependencies-with-the-right-stuff</id><content type="html" xml:base="https://brokenco.de//2018/09/27/choosing-dependencies-with-the-right-stuff.html"><![CDATA[<p>For <a href="https://jenkins.io/">Jenkins</a>, the plugin ecosystem is one of its key
advantages over other tools offering some similar functionality. That power and
flexibility does not come without its own set of problems for the project
itself. From an outsiders perspective, the challenges around dependency and
update management between Jenkins plugins is a substantial topic, worthy of at
least a couple of doctoral theses in computer science and sociology
respectively. For insiders within the Jenkins developer community, the
relations between plugins in the ecosystem makes a bizarre kind of sense. Like
the tax code, it’s something you figure out how to work within, but never dare
dig in too deeply, for fear of your head exploding. In this blog post, I’d like
to share my philosophy on how we, the Jenkins project, <em>should</em> think about
plugin dependencies and how that contrasts to the status quo.</p>

<hr />

<p><strong>Note</strong>: This may be a bit of “<a href="https://en.wikipedia.org/wiki/Inside_baseball_%28metaphor%29">inside
baseball</a>” related to the <a href="/2018/08/30/cd-for-the-jenkins-project.html">Jenkins
Evergreen</a> effort which I have blogged
about previously.</p>

<hr />

<h3 id="how-jenkins-plugin-dependencies-work">How Jenkins plugin dependencies work</h3>

<p>Jenkins plugins are primarily written in Java and use
<a href="https://maven.apache.org">Apache Maven</a> to define their dependencies and build
process. By way of an example, imagine you were building a Taco Cat plugin which
depended on the <code class="language-plaintext highlighter-rouge">git-client</code> plugin’s APIs and functionality. In order to even
compile your plugin referencing APIs provided by <code class="language-plaintext highlighter-rouge">git-client</code>, the <code class="language-plaintext highlighter-rouge">pom.xml</code>
for your plugin would need to specify the <code class="language-plaintext highlighter-rouge">git-client</code> plugin as a dependency.</p>

<p>Of course this build-time dependency implies a run-time dependency.
When the Taco Cat plugin you are developing is actually running in a Jenkins
master’s JVM, it needs to invoke functionality provided by the <code class="language-plaintext highlighter-rouge">git-client</code>
plugin. Since the <code class="language-plaintext highlighter-rouge">git-client</code> dependency is not automatically “shaded”, or statically
added into the <code class="language-plaintext highlighter-rouge">tacocat.hpi</code> artifact when it is built, Jenkins must be
instructed to fetch and install <code class="language-plaintext highlighter-rouge">git-client.hpi</code> when <code class="language-plaintext highlighter-rouge">tacocat.hpi</code> is
installed.</p>

<p>The Jenkins project has tooling to add special plugin dependency information to
the <code class="language-plaintext highlighter-rouge">tacocat.hpi</code> file, which is really a <code class="language-plaintext highlighter-rouge">.jar</code>, in the
<code class="language-plaintext highlighter-rouge">META-INF/MANIFEST.MF</code>. Additional tooling behind the scenes processes that
information from the released <code class="language-plaintext highlighter-rouge">tacocat.hpi</code> artifact, and generates the “Update
Center”, which is itself a periodically updated <code class="language-plaintext highlighter-rouge">.json</code> file listing all the
latest released versions of plugins. Among other information, the Update Center references the dependencies by the plugin’s name and version.
For example, the entry for <code class="language-plaintext highlighter-rouge">tacocat</code> might look something
like the following:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>"tacocat": {
  "buildDate": "Sep 27, 2018",
  "version": "0.2",
  "dependencies": [
    {
      "name": "git-client",
      "optional": false,
      "version": "1.0"
    },
  ]
}
</code></pre></div></div>

<p>The <code class="language-plaintext highlighter-rouge">version</code> field is the version of the dependency specified way
back at the beginning in the <code class="language-plaintext highlighter-rouge">pom.xml</code> as a build-time dependency.
Unfortunately, the Update Center only includes the <em>latest</em> versions of
all released plugins, not the history of plugins. This means that this same
<code class="language-plaintext highlighter-rouge">.json</code> file will contain the <code class="language-plaintext highlighter-rouge">git-client</code> entry, but only for the latest
(<code class="language-plaintext highlighter-rouge">2.7.3</code>) version.</p>

<p>While this has clearly worked for the Jenkins project for a long time, there
are a number of technological and cultural issues with this approach.</p>

<h3 id="the-problems">The problems</h3>

<h4 id="untested-combinations">Untested combinations</h4>

<p>Perhaps the most notable problem is the “take the latest dependency” approach
Jenkins utilizes in the Update Center. In the <code class="language-plaintext highlighter-rouge">tacocat</code> example above, I
intentionally referenced <code class="language-plaintext highlighter-rouge">git-client</code> at version <code class="language-plaintext highlighter-rouge">1.0</code>. This means that when I
build and test <code class="language-plaintext highlighter-rouge">tacocat</code> locally, I will be grabbing the <code class="language-plaintext highlighter-rouge">1.0</code> version of
<code class="language-plaintext highlighter-rouge">git-client</code>.</p>

<p>The latest <code class="language-plaintext highlighter-rouge">git-client</code> in the Update Center is version <code class="language-plaintext highlighter-rouge">2.7.3</code>. This means
that when a new Jenkins user installs the Taco Cat plugin, they will be effectively
running an untested version combination of <code class="language-plaintext highlighter-rouge">git-client 2.7.3</code> and <code class="language-plaintext highlighter-rouge">tacocat 0.2</code></p>

<p>If an existing Jenkins user, with <code class="language-plaintext highlighter-rouge">git-client 2.0.1</code> installs the Taco Cat
plugin, they too will be running an untested version combination with
<code class="language-plaintext highlighter-rouge">git-client 2.0.1</code> and <code class="language-plaintext highlighter-rouge">tacocat 0.2</code>. And so it goes for every possible version
of <code class="language-plaintext highlighter-rouge">git-client</code> which is used in the wild from version <code class="language-plaintext highlighter-rouge">1.0</code> to <code class="language-plaintext highlighter-rouge">2.7.3</code>.</p>

<h4 id="least-possible-dependency">Least possible dependency</h4>

<p>Culturally the Jenkins project trends very conservative with dependency
management and upgrades. Many developers view plugin dependencies as a simply
<strong>build-time</strong> concern and apply a principle of taking the least possible
dependency. In practice this means that if the <code class="language-plaintext highlighter-rouge">git-client</code> plugin introduced
the API needed for the Taco Cat plugin in <code class="language-plaintext highlighter-rouge">1.0</code>, then the <code class="language-plaintext highlighter-rouge">pom.xml</code> for
<code class="language-plaintext highlighter-rouge">tacocat</code> should specify that version, regardless of what improvements, bug
fixes, or security issue have been addressed in the <code class="language-plaintext highlighter-rouge">git-client</code> plugin in the
interim.</p>

<p>Since build-time dependencies translate to <strong>run-time</strong> dependencies, this
means a user is <em>allowed</em> to have <code class="language-plaintext highlighter-rouge">git-client 1.0</code> and install the newer Taco
Cat plugin, regardless of what critical fixes have been made in the subsequent
releases of the <code class="language-plaintext highlighter-rouge">git-client</code> plugin. Fortunately for that user however, they
would actually be running a tested version combination.</p>

<p>While I understand the build-time perspective regarding bumping dependency
versions, in my opinion this behavior betrays a serious lack of appreciation
for how Jenkins plugins are consumed as part of a greater system.</p>

<p>In a cursory examination of plugin dependency relationships, I found numerous
plugins which specified dependencies with versions containing <a href="https://jenkins.io/security/advisories/">known security
issues</a>. As much as I would hope that
all installations of Jenkins are applying the appropriate security updates, the
behavior of <em>allowing</em> users to receive features and updates to some areas of
Jenkins such as Blue Ocean or Pipeline, without requiring the upgrade of
insecure dependencies, is not satisfactory.</p>

<h4 id="opportunity-for-destabilization">Opportunity for destabilization</h4>

<p>The most commonly encountered problem with the current approach to dependency
management is the sibling of “Untested combinations” wherein two plugins in the
environment depend on two incompatible versions of the same dependency. Jenkins
plugins are loaded in a manner similar to shared libraries such that only one
version of <code class="language-plaintext highlighter-rouge">git-client</code> will be loaded into the JVM at a time. Any plugins
which depend on <code class="language-plaintext highlighter-rouge">git-client</code>, must all be happy with whatever version is
loaded, regardless of what their <code class="language-plaintext highlighter-rouge">META-INF/MANIFEST.MF</code> declares.</p>

<p>For some users, this can lead to upgrades proving hazardous. There is no
guarantee that plugins follow Semantic Versioning, or any other sane versioning
scheme for that matter, it is entirely possible to see API incompatible changes
made in point-releases of some plugins. (Users of Docker Pipeline may recall
some unhappy changes to <code class="language-plaintext highlighter-rouge">ENTRYPOINT</code> behavior in 1.15).</p>

<p>Problems with incompatible plugins typically manifest in one of three
behaviors:</p>

<ol>
  <li>Jenkins is bricked and won’t start up.</li>
  <li>Jenkins logs an error that it cannot load one of the plugins, and continues
to operate without that plugin’s functionality present.</li>
  <li>Some functionality at the seams between plugins, which previously worked, no longer works properly.</li>
</ol>

<p>None of these is particularly desirable, and the burden of diagnosing what the
heck went wrong is left solely on the user’s shoulders.</p>

<hr />

<p>Jenkins is presently <em>developed</em> as a series of components, some of which are
loosely coupled, others less so.</p>

<p>Jenkins is <em>used</em> as a comprehensive system.</p>

<p>I believe that the current approach to plugin dependency management has led to
a subpar user experience, and though it has been part of why Jenkins
flourished organically over the past decade, in recent years the technological
and cultural infrastructure behind it is no longer sufficient to keep Jenkins
competitive.</p>

<h3 id="how-jenkins-evergreen-works">How Jenkins Evergreen works</h3>

<p><a href="https://jenkins.io/projects/evergreen">Jenkins Evergreen</a> is a curated and
automatically updating distribution of Jenkins core and plugins. It is
important to highlight that Evergreen is <em>not</em> something different from what
Jenkins is today, but rather new packaging with some nice auto-update
functionality to boot.</p>

<p>In its current implementation, Jenkins Evergreen is defined by a “<a href="https://github.com/jenkins-infra/evergreen/blob/master/services/essentials.yaml">Bill of
Materials</a>”
which describes the exact version combination of core and plugins will be
installed in the Jenkins environment.</p>

<p>The Bill of Materials is described in a file called <code class="language-plaintext highlighter-rouge">essentials.yaml</code> which
contains two relevant sections <code class="language-plaintext highlighter-rouge">spec</code> and <code class="language-plaintext highlighter-rouge">status</code>.  In the <code class="language-plaintext highlighter-rouge">spec</code> block, we
describe the plugins which contain the features deemed necessary for Jenkins
Evergreen. The <code class="language-plaintext highlighter-rouge">status</code> block is then populated by our tooling based on the
plugin dependencies outlined in the <code class="language-plaintext highlighter-rouge">META-INF/MANIFEST.MF</code> file described
earlier. The tooling will resolve dependencies using the same “least possible
dependency” approach, meaning that if a plugin declares a dependency on
<code class="language-plaintext highlighter-rouge">git-client 1.0</code>, and no other plugin requires a later version, Evergreen will
ship <code class="language-plaintext highlighter-rouge">git-client 1.0</code> unless otherwise specified.</p>

<p>To further reduce potential incompatibilities or destabilization, Evergreen
currently <strong>restricts the ability of a user to install additional plugins</strong>.
This may strike some as a controversial approach, but in our early alpha
testing we found it far too easy for testers to destabilize their Jenkins
environment and are preferring a more curated approach for the moment.</p>

<p>One suggestion in the development of Evergreen’s tooling was to unilaterally
pin our plugins to the latest released in the Update Center. This is similar to
an approach I used for a project in 2017 called “Code Valet.” This sort of
bleeding-edge approach is in fact quite bloody, and I ended up identifying a
large number of low-severity bugs by being the first person to ever encounter
the combination of plugin versions Code Valet was shipping. Great to find the
bugs, but shifts a significant testing overhead to the integrator (me).</p>

<p>For better or worse, Evergreen is trusting the versions which plugin developers
are declaring to have been tested and built against.</p>

<h3 id="what-would-be-better">What would be better</h3>

<p>Whether via Evergreen, or a traditional installation, Jenkins is used as a
coherent system, regardless of how it may be developed or delivered. To better
serve this end, I believe changes should be made in how this system is
developed.</p>

<p>Forgetting for a moment the technical complexity of any of these suggestions, I
think Jenkins would be much better with:</p>

<ol>
  <li>
    <p><strong>Plugin isolation:</strong> if my Taco Cat plugin really only ever says “give me
<code class="language-plaintext highlighter-rouge">git-client 1.0</code>, then at runtime, that’s exactly what should be provided.
This would certainly bloat the memory footprint of Jenkins, but I can imagine a
number of optimizations to lower that memory usage.</p>
  </li>
  <li>
    <p><strong>Automatic upgrade of <code class="language-plaintext highlighter-rouge">pom.xml</code> for security fixes:</strong> if the Taco Cat
plugin depends on a known-vulnerable-version of a specific plugin, we should
have GitHub-based automation to propose, and perhaps even merge, a change to
the <code class="language-plaintext highlighter-rouge">pom.xml</code> bumping that dependency to a known-safe-version.</p>
  </li>
  <li>
    <p><strong>Entire-system testing:</strong> There has already been some work done by Oliver
Gondza and others with the
<a href="https://github.com/jenkinsci/acceptance-test-harness">acceptance-test-harness</a>
work, which should be broadened to be used for <em>plugins</em> and their pull
requests. A plugin like <code class="language-plaintext highlighter-rouge">git-client</code> is so fundamentally important to the
usage of any Jenkins installation these days, that pull requests should be
blocked if changes to <code class="language-plaintext highlighter-rouge">git-client</code> break the system. (<em>Note:</em> <code class="language-plaintext highlighter-rouge">git-client</code> is
typically very well tested and maintained, I’m only picking on it as an
example).</p>
  </li>
  <li>
    <p><strong>Shed the culture of least possible dependency:</strong> Rather than thinking
about dependencies in <code class="language-plaintext highlighter-rouge">pom.xml</code> files as build time dependencies, thinking
about them as run-time dependencies, and of the user who will be running them.
If the <code class="language-plaintext highlighter-rouge">git-client</code> plugin provides very useful features in version <code class="language-plaintext highlighter-rouge">2.0</code> or
later, each maintainer of a plugin which depends on it, should strongly
consider upgrading their <code class="language-plaintext highlighter-rouge">pom.xml</code> declarations. Not because their plugin
necessarily requires newer APIs, but because users should be encouraged to
adopt the latest bug fixes and features.</p>
  </li>
  <li>
    <p><strong>Encourage plugin squashing:</strong> A large amount of complexity stems from a culture of
putting everything into its own plugin, as if Jenkins plugins must follow
Unix-style idioms. Jenkins Pipeline is particularly troublesome in this
regard. For many plugin suites, or sets of related plugins, there’s not any
user benefit from the fragmentation of functionality across multiple
plugins where they may end up with untested combinations of related plugins.
In many cases, a single plugin in the suite is functionally unusabe without the
others also present.  These suites of plugins should be squashed together
wherever possible.</p>
  </li>
  <li>
    <p><strong>Report and receive plugin health metrics:</strong> The Update Center is
fundamentally static today, but users would be much better served if the
updates interaction between a Jenkins instance and the project’s services were
more “live.” By reporting plugin health, or installation success, associated
with the versions installed in the Jenkins instance, we would be able to
surface useful information to users. Showing users whether a certain plugin
version is known to break in their version of Jenkins, or perhaps conflicts
with existing versions of other plugins, would help avoid numerous users
falling into the same broken-plugin tar pits.</p>
  </li>
</ol>

<p>I am by no means zealous about fixing the plugin issues in the Jenkins
ecosystem specifically. It is not the area of the project which I love to hack
on. I am zealous however about fixing the experience users encounter with
Jenkins overall. As of today, I see the plugin ecosystem as the biggest
double-edged sword we have to contend with in the project. Through our plugin
ecosystem users can extend Jenkins to meet all sorts of disparate needs.
Through our plugin ecosystem users can also easily find themselves with broken
environments.</p>

<p>That is not a “user error”, that is a “Jenkins error.” And one we <strong>must</strong> fix.</p>]]></content><author><name>R. Tyler Croy</name></author><category term="jenkins" /><category term="evergreen" /><category term="opinion" /><summary type="html"><![CDATA[For Jenkins, the plugin ecosystem is one of its key advantages over other tools offering some similar functionality. That power and flexibility does not come without its own set of problems for the project itself. From an outsiders perspective, the challenges around dependency and update management between Jenkins plugins is a substantial topic, worthy of at least a couple of doctoral theses in computer science and sociology respectively. For insiders within the Jenkins developer community, the relations between plugins in the ecosystem makes a bizarre kind of sense. Like the tax code, it’s something you figure out how to work within, but never dare dig in too deeply, for fear of your head exploding. In this blog post, I’d like to share my philosophy on how we, the Jenkins project, should think about plugin dependencies and how that contrasts to the status quo.]]></summary></entry></feed>