<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom" ><generator uri="https://jekyllrb.com/" version="3.10.0">Jekyll</generator><link href="https://brokenco.de//feed/by_tag/security.xml" rel="self" type="application/atom+xml" /><link href="https://brokenco.de//" rel="alternate" type="text/html" /><updated>2026-10-31T17:58:32+00:00</updated><id>https://brokenco.de//feed/by_tag/security.xml</id><title type="html">rtyler</title><subtitle>a moderately technical blog</subtitle><author><name>R. Tyler Croy</name></author><entry><title type="html">Keeping up with the Joneses</title><link href="https://brokenco.de//2026/10/31/keeping-up-with-the-joneses.html" rel="alternate" type="text/html" title="Keeping up with the Joneses" /><published>2026-10-31T00:00:00+00:00</published><updated>2026-10-31T00:00:00+00:00</updated><id>https://brokenco.de//2026/10/31/keeping-up-with-the-joneses</id><content type="html" xml:base="https://brokenco.de//2026/10/31/keeping-up-with-the-joneses.html"><![CDATA[<p>Every non-trivial application these days is a layer of business logic on
top of an enormous pile of open source dependencies. Keeping up with <em>their</em>
releases, especially the security-relevant ones, is a job in itself. I am not
able to directly follow pull requests for some of our key dependencies in
<a href="https://github.com/delta-io/delta-rs">delta-io/delta-rs</a>, but I <strong>do</strong> try to
keep an eye out for releases of
<a href="https://github.com/apache/arrow-rs">apache/arrow-rs</a> and
<a href="https://github.com/apache/datafusion">apache/datafusion</a>.</p>

<p>Before GitHub became Copilot AI Hub they built a lot of hidden features that
make integration with GitHub very useful like <a href="https://docs.github.com/en/rest/activity/feeds">Atom feeds for repo
tags</a> at a predictable URL. For
DataFusion, that is
<a href="https://github.com/apache/datafusion/tags.atom">github.com/apache/datafusion/tags.atom</a>.
Drop that URL into any feed reader and new releases show up like any other
news.</p>

<p>I prefer <code class="language-plaintext highlighter-rouge">tags.atom</code> over the <code class="language-plaintext highlighter-rouge">releases.atom</code> variant you will also see since
lots of projects don’t bother publishing GitHub Releases, like yours truly.
virtually every Rust crate however pushes git tags.</p>

<h2 id="generating-the-list">Generating the list</h2>

<p>Every published crate records its repository on crates.io, so the pipeline is:
parse the manifest, ask the <a href="https://crates.io/data-access">crates.io API</a>
where the code lives, and append <code class="language-plaintext highlighter-rouge">/tags.atom</code> to anything on GitHub:</p>

<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">#!/usr/bin/env python3
</span><span class="s">"""Generate GitHub tag Atom feed URLs for a Cargo.toml's dependencies."""</span>
<span class="kn">import</span> <span class="nn">json</span>
<span class="kn">import</span> <span class="nn">sys</span>
<span class="kn">import</span> <span class="nn">tomllib</span>
<span class="kn">import</span> <span class="nn">urllib.request</span>


<span class="k">def</span> <span class="nf">crate_repository</span><span class="p">(</span><span class="n">name</span><span class="p">):</span>
    <span class="s">"""Look up the repository URL for a crate on crates.io."""</span>
    <span class="n">req</span> <span class="o">=</span> <span class="n">urllib</span><span class="p">.</span><span class="n">request</span><span class="p">.</span><span class="n">Request</span><span class="p">(</span>
        <span class="sa">f</span><span class="s">"https://crates.io/api/v1/crates/</span><span class="si">{</span><span class="n">name</span><span class="si">}</span><span class="s">"</span><span class="p">,</span>
        <span class="n">headers</span><span class="o">=</span><span class="p">{</span><span class="s">"User-Agent"</span><span class="p">:</span> <span class="s">"cargo-atom-feeds (https://brokenco.de)"</span><span class="p">},</span>
    <span class="p">)</span>
    <span class="k">with</span> <span class="n">urllib</span><span class="p">.</span><span class="n">request</span><span class="p">.</span><span class="n">urlopen</span><span class="p">(</span><span class="n">req</span><span class="p">)</span> <span class="k">as</span> <span class="n">resp</span><span class="p">:</span>
        <span class="k">return</span> <span class="n">json</span><span class="p">.</span><span class="n">load</span><span class="p">(</span><span class="n">resp</span><span class="p">)[</span><span class="s">"crate"</span><span class="p">][</span><span class="s">"repository"</span><span class="p">]</span>


<span class="k">def</span> <span class="nf">atom_url</span><span class="p">(</span><span class="n">repository</span><span class="p">):</span>
    <span class="s">"""Turn a GitHub repository URL into its tags.atom feed URL."""</span>
    <span class="k">if</span> <span class="ow">not</span> <span class="n">repository</span> <span class="ow">or</span> <span class="s">"github.com"</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">repository</span><span class="p">:</span>
        <span class="k">return</span> <span class="bp">None</span>
    <span class="n">path</span> <span class="o">=</span> <span class="n">repository</span><span class="p">.</span><span class="n">removesuffix</span><span class="p">(</span><span class="s">".git"</span><span class="p">).</span><span class="n">rstrip</span><span class="p">(</span><span class="s">"/"</span><span class="p">)</span>
    <span class="n">parts</span> <span class="o">=</span> <span class="n">path</span><span class="p">.</span><span class="n">split</span><span class="p">(</span><span class="s">"github.com/"</span><span class="p">)[</span><span class="o">-</span><span class="mi">1</span><span class="p">].</span><span class="n">split</span><span class="p">(</span><span class="s">"/"</span><span class="p">)</span>
    <span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">parts</span><span class="p">)</span> <span class="o">&lt;</span> <span class="mi">2</span><span class="p">:</span>
        <span class="k">return</span> <span class="bp">None</span>
    <span class="k">return</span> <span class="sa">f</span><span class="s">"https://github.com/</span><span class="si">{</span><span class="n">parts</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="si">}</span><span class="s">/</span><span class="si">{</span><span class="n">parts</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span><span class="si">}</span><span class="s">/tags.atom"</span>


<span class="k">def</span> <span class="nf">dependencies</span><span class="p">(</span><span class="n">cargo_toml</span><span class="p">):</span>
    <span class="s">"""Yield crate names from a parsed Cargo.toml."""</span>
    <span class="n">tables</span> <span class="o">=</span> <span class="p">[</span><span class="n">cargo_toml</span><span class="p">.</span><span class="n">get</span><span class="p">(</span><span class="n">section</span><span class="p">,</span> <span class="p">{})</span> <span class="k">for</span> <span class="n">section</span> <span class="ow">in</span>
              <span class="p">(</span><span class="s">"dependencies"</span><span class="p">,</span> <span class="s">"dev-dependencies"</span><span class="p">,</span> <span class="s">"build-dependencies"</span><span class="p">)]</span>
    <span class="n">tables</span><span class="p">.</span><span class="n">append</span><span class="p">(</span><span class="n">cargo_toml</span><span class="p">.</span><span class="n">get</span><span class="p">(</span><span class="s">"workspace"</span><span class="p">,</span> <span class="p">{}).</span><span class="n">get</span><span class="p">(</span><span class="s">"dependencies"</span><span class="p">,</span> <span class="p">{}))</span>
    <span class="k">for</span> <span class="n">table</span> <span class="ow">in</span> <span class="n">tables</span><span class="p">:</span>
        <span class="k">for</span> <span class="n">name</span><span class="p">,</span> <span class="n">spec</span> <span class="ow">in</span> <span class="n">table</span><span class="p">.</span><span class="n">items</span><span class="p">():</span>
            <span class="c1"># renamed crates: serde = { package = "serde_v8", ... }
</span>            <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">spec</span><span class="p">,</span> <span class="nb">dict</span><span class="p">)</span> <span class="ow">and</span> <span class="s">"package"</span> <span class="ow">in</span> <span class="n">spec</span><span class="p">:</span>
                <span class="k">yield</span> <span class="n">spec</span><span class="p">[</span><span class="s">"package"</span><span class="p">]</span>
            <span class="k">else</span><span class="p">:</span>
                <span class="k">yield</span> <span class="n">name</span>


<span class="k">def</span> <span class="nf">main</span><span class="p">():</span>
    <span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">sys</span><span class="p">.</span><span class="n">argv</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span> <span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">sys</span><span class="p">.</span><span class="n">argv</span><span class="p">)</span> <span class="o">&gt;</span> <span class="mi">1</span> <span class="k">else</span> <span class="s">"Cargo.toml"</span><span class="p">,</span> <span class="s">"rb"</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
        <span class="n">cargo</span> <span class="o">=</span> <span class="n">tomllib</span><span class="p">.</span><span class="n">load</span><span class="p">(</span><span class="n">f</span><span class="p">)</span>

    <span class="k">for</span> <span class="n">crate</span> <span class="ow">in</span> <span class="nb">sorted</span><span class="p">(</span><span class="nb">set</span><span class="p">(</span><span class="n">dependencies</span><span class="p">(</span><span class="n">cargo</span><span class="p">))):</span>
        <span class="n">url</span> <span class="o">=</span> <span class="n">atom_url</span><span class="p">(</span><span class="n">crate_repository</span><span class="p">(</span><span class="n">crate</span><span class="p">))</span>
        <span class="k">if</span> <span class="n">url</span><span class="p">:</span>
            <span class="k">print</span><span class="p">(</span><span class="sa">f</span><span class="s">"</span><span class="si">{</span><span class="n">crate</span><span class="si">:</span><span class="mi">30</span><span class="n">s</span><span class="si">}</span><span class="s"> </span><span class="si">{</span><span class="n">url</span><span class="si">}</span><span class="s">"</span><span class="p">)</span>
        <span class="k">else</span><span class="p">:</span>
            <span class="k">print</span><span class="p">(</span><span class="sa">f</span><span class="s">"</span><span class="si">{</span><span class="n">crate</span><span class="si">:</span><span class="mi">30</span><span class="n">s</span><span class="si">}</span><span class="s"> (no GitHub repository listed)"</span><span class="p">)</span>


<span class="k">if</span> <span class="n">__name__</span> <span class="o">==</span> <span class="s">"__main__"</span><span class="p">:</span>
    <span class="n">main</span><span class="p">()</span>
</code></pre></div></div>

<p><strong>NOTE</strong>: <em>The <code class="language-plaintext highlighter-rouge">User-Agent</code> header is not decoration; crates.io’s
<a href="https://crates.io/data-access">data access policy</a> asks for one, so be a mensch and use a custom one for your code should you run it :)</em></p>

<p>Paste the unique URLs into your reader of choice and preso-bango I’ve got a
little bit better understanding of when my key dependencies get updated.</p>

<hr />

<p>The feed entry is only a notification, I don’t automate further than that. I
still read the release notes before upgjading anything because a lot of
dependencies have … <em>errata</em> and other considerations that bar their adoption.</p>

<p>Typically when I see a new tag show up it translates to a ticket or todo list
item, but at least then I know. <a href="https://www.youtube.com/watch?v=6I-MK9mF9nE">And knowing is half the
battle</a></p>]]></content><author><name>R. Tyler Croy</name></author><category term="rust" /><category term="deltalake" /><category term="security" /><category term="software" /><summary type="html"><![CDATA[Every non-trivial application these days is a layer of business logic on top of an enormous pile of open source dependencies. Keeping up with their releases, especially the security-relevant ones, is a job in itself. I am not able to directly follow pull requests for some of our key dependencies in delta-io/delta-rs, but I do try to keep an eye out for releases of apache/arrow-rs and apache/datafusion.]]></summary></entry><entry><title type="html">How sudo gets root</title><link href="https://brokenco.de//2021/07/17/how-sudo-works.html" rel="alternate" type="text/html" title="How sudo gets root" /><published>2021-07-17T00:00:00+00:00</published><updated>2021-07-17T00:00:00+00:00</updated><id>https://brokenco.de//2021/07/17/how-sudo-works</id><content type="html" xml:base="https://brokenco.de//2021/07/17/how-sudo-works.html"><![CDATA[<p>Today I fell into a rabbit hole around user and process
permissions and had to learn how <code class="language-plaintext highlighter-rouge">sudo</code> actually worked. For the program I was
working on I set out to figure out how to perform something akin to a “user
swap” when launching subprocesses. On its face it seems simple enough, my
program runs with user id <code class="language-plaintext highlighter-rouge">1000</code> and I wish to shunt child processes over to
run as another user id. <code class="language-plaintext highlighter-rouge">sudo</code> can do it, so why can’t I? “For reasons” is the answer.</p>

<p>The alternative for this post could be “unix permissions are insane”.</p>

<p>Regardless of whether spawning a child or doing something within the process
for modifying “who” is running a program is effectively the same. The program
runs with the user id (<code class="language-plaintext highlighter-rouge">uid</code>) of the caller, so in the case of your shell which
is running as you (often <code class="language-plaintext highlighter-rouge">uid</code> of <code class="language-plaintext highlighter-rouge">1000</code>) and spawned process will inherit the
same <code class="language-plaintext highlighter-rouge">uid</code> .</p>

<p>In a C program, I can however reassign my process’ <code class="language-plaintext highlighter-rouge">uid</code> with
the <code class="language-plaintext highlighter-rouge">setuid(2)</code> system call:</p>

<blockquote>
  <p><strong><code class="language-plaintext highlighter-rouge">setuid()</code></strong> sets  the  effective  user  ID of the calling process.  If the
calling process is privileged (more precisely: if the process has the
<strong>CAP_SETUID</strong> capability in its user namespace), the real UID and saved
set-user-ID are also set.</p>
</blockquote>

<p>(In Rust this function is helpfully exposed via <a href="https://docs.rs/nix/0.22.0/nix/unistd/fn.setuid.html">the nix crate</a>.)</p>

<p>My normal shell user is <em>not</em> the <code class="language-plaintext highlighter-rouge">root</code> user and therefore in just about every
program I execute, they cannot use <code class="language-plaintext highlighter-rouge">setuid(2)</code> to change to <strong>any other user
id</strong>.</p>

<p>If processes spawned by my user cannot assume other user ids, how the heck does
<code class="language-plaintext highlighter-rouge">sudo</code> do it?</p>

<p>Based on the man page excerpt above, it’s easy to assume that perhaps the
process has the <code class="language-plaintext highlighter-rouge">CAP_SETUID</code> capability? Per-file capabilities are a Linux-ism which exposed to user-land via libcap:</p>

<blockquote>
  <p>Libcap implements the user-space interfaces to the POSIX 1003.1e capabilities
available in Linux kernels. These capabilities are a partitioning of the all
powerful root privilege into a set of distinct privileges.</p>
</blockquote>

<p>These capabilities <em>can</em> provide the desired <code class="language-plaintext highlighter-rouge">setuid</code> functionality, but they
can also be dangerous and lead to vulnerabilities.  <a href="https://steflan-security.com/linux-privilege-escalation-exploiting-capabilities/">This blog
post</a>
has a simple demonstration of using <code class="language-plaintext highlighter-rouge">CAP_SETUID</code> to gain root on a machine.  We
can check whether the <code class="language-plaintext highlighter-rouge">sudo</code> binary has this capability with the <code class="language-plaintext highlighter-rouge">getcap</code>
program, which is not always in the distribution’s default packages:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>❯ sudo getcap `which sudo`

❯
</code></pre></div></div>

<p>Unfortunately it looks like <code class="language-plaintext highlighter-rouge">sudo</code> has no special file capabilities,
fortunately there is <em>one</em> other way to for a program to run with <code class="language-plaintext highlighter-rouge">root</code>
privileges, but if you don’t know where to look, good luck finding it!</p>

<p>The file system contains a little bit more information about an executable
referred to as the “setuid bit”. Using <code class="language-plaintext highlighter-rouge">chmod</code> you take a binary that you own,
set the “setuid bit” and then whenever <em>any</em> user runs the binary, it will run
as your user. Naturally if a binary is owned by root with the setuid bit set,
any caller of that binary will run the binary <strong>as root</strong>.</p>

<p>This is exactly what sudo does, which you can check just by exampling the file mode:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>❯ ls -lah `which sudo`
-rwsr-xr-x 1 root root 181K May 27 07:49 /usr/bin/sudo
❯
</code></pre></div></div>

<p>Note the <code class="language-plaintext highlighter-rouge">rws</code> at the beginning of the mode string, this means read (<code class="language-plaintext highlighter-rouge">r</code>),
write (<code class="language-plaintext highlighter-rouge">w</code>), and setuid (<code class="language-plaintext highlighter-rouge">s</code>) are all set at the user level on the file.
Additionally the executable (<code class="language-plaintext highlighter-rouge">x</code>) bits allow all users within the <code class="language-plaintext highlighter-rouge">root</code> group,
and <em>everybody else</em> to execute the file.</p>

<p>Sudo basically starts out as <code class="language-plaintext highlighter-rouge">root</code> and then will read its configuration file
and validate/drop permissions as necessary. Since the program is running with
the <code class="language-plaintext highlighter-rouge">root</code> level privileges, it can easily call <code class="language-plaintext highlighter-rouge">setuid(2)</code> type functions and
change to <em>any</em> user on the system.</p>

<p>Should you wish to explore <em>exactly</em> how sudo works, the <a href="https://github.com/sudo-project/sudo">source code is on GitHub</a>.</p>

<hr />

<p>There is effectively no user hierarchy in Unix-inspired systems, there is
<em>root</em> and then there is everybody else. Unfortunately for my hacking this
means I cannot have a user (<code class="language-plaintext highlighter-rouge">1000</code>) launch a process with a “lateral” user
permission such as uid <code class="language-plaintext highlighter-rouge">1001</code>. I would first need to escalate privileges in
some manner from <code class="language-plaintext highlighter-rouge">1000</code> to <code class="language-plaintext highlighter-rouge">root</code> and then drop back down again to <code class="language-plaintext highlighter-rouge">1001</code>.</p>

<p>If you ever see a program that “drops privileges” such as Docker, Nginx, or Systemd, just
remember that in there is no dropping privileges without first escalating to
<code class="language-plaintext highlighter-rouge">root</code> in a Unix-inspired system!</p>]]></content><author><name>R. Tyler Croy</name></author><category term="security" /><category term="linux" /><category term="unix" /><summary type="html"><![CDATA[Today I fell into a rabbit hole around user and process permissions and had to learn how sudo actually worked. For the program I was working on I set out to figure out how to perform something akin to a “user swap” when launching subprocesses. On its face it seems simple enough, my program runs with user id 1000 and I wish to shunt child processes over to run as another user id. sudo can do it, so why can’t I? “For reasons” is the answer.]]></summary></entry><entry><title type="html">Ransomware is coming to a cloud near you</title><link href="https://brokenco.de//2021/06/03/cloud-ransomware.html" rel="alternate" type="text/html" title="Ransomware is coming to a cloud near you" /><published>2021-06-03T00:00:00+00:00</published><updated>2021-06-03T00:00:00+00:00</updated><id>https://brokenco.de//2021/06/03/cloud-ransomware</id><content type="html" xml:base="https://brokenco.de//2021/06/03/cloud-ransomware.html"><![CDATA[<p>Ransomware is the most significant and dangerous evolution of computer-based
crime I have seen, and it’s going to get worse. Ransomware attacks have
compromised <a href="https://www.complianceweek.com/cyber-security/colonial-pipeline-fallout-thwarting-ransomware-attacks-requires-collective-defense/30438.article">oil
pipelines</a>,
<a href="https://www.bbc.com/news/technology-35880610">hospitals</a>, and
<a href="https://arstechnica.com/gadgets/2021/06/attack-on-meat-supplier-came-from-revil-ransomwares-most-cut-throat-gang/">beef</a>.
While they’re nothing new over the past two years, targets have become
increasingly high-profile and the adverse impacts of ransomware have similarly
become more dire. Based on my read of the reports and incident reviews, these
attacks seem to largely be affecting physical infrastructure assets:
workstations, servers sitting in closets, and small-scale data center
operations. Given this trend, it might be easy conclude that running in AWS, Azure, or Google
Cloud offers some level of protection. I strongly doubt it, and I think
ransomware is about to get <strong>worse</strong>.</p>

<p>The mythos of “cloud-native” technology is not nearly wide-spread as its
practitioners would like to admit. I posit that <strong>most</strong> of the workloads
running in a public cloud like AWS are fairly simplistic “Infrastructure as a
Service” (IaaS) deployments. Rather than using higher-level cloud-native
platforms, most of what makes up the “cloud” are: virtual disks, network
devices, and machines. There is <strong>nothing</strong> inherently safer about running a
virtual machine in AWS compared to an on-premise machine. A cloud-based virtual
machine does make it easier to take disk snapshots and restore machines, but
that’s only if you <em>use</em> those features. I would guess that most don’t.</p>

<p>I believe the nightmare scenario that corporate IT departments are experiencing
will soon be visiting tech companies and others that have migrated into cloud
environments. The worst-case scenario that nags at me goes something like this (using AWS terminology):</p>

<ul>
  <li>An attacker finds an “in”, through a leaked set of IAM keys or other exploit.</li>
  <li>The attacker disables S3 object versioning, RDS snapshots, or other safe-guards that have been enabled.</li>
  <li>The attacker then starts walking through stored data, downloading, deleting,
or encrypting it along the way.</li>
  <li>At some point it is “zero day” and the final push of deleting/encrypting of “live” data is complete and the organization is paralyzed.</li>
</ul>

<p>As long as the attacker is able to compromise an account with a high enough access level, there is unfathomable amount of damage that could be done. Segmented accounts can provide bulkheads against the damage, but based on the “digital transformations” I have seen over the past five years the two things typically left behind when enterprises migrate to the cloud are: security and disaster recovery.</p>

<p>In fact, I would guess that for many cloud users if the data attackers were
compromising wasn’t in a “hot” access path, the attackers could remain
undetected inside the account for long periods of time, similar to the
on-premise enterprises hit by ransomware.</p>

<p>Ransomware is <strong>lucrative</strong> and will not be going anywhere soon. The cloud doesn’t inherently protect you but it <em>does</em> provide a <em>lot</em> of mechanisms that allow for better security practices, intrusion detection, policy violations, and disaster recover. The big question I would encourage any infrastructure engineer to be asking themselves right now is: <strong>how can I reduce the impact of an attack</strong>.</p>

<p>Because like it or not, they’re coming.</p>]]></content><author><name>R. Tyler Croy</name></author><category term="opinion" /><category term="security" /><summary type="html"><![CDATA[Ransomware is the most significant and dangerous evolution of computer-based crime I have seen, and it’s going to get worse. Ransomware attacks have compromised oil pipelines, hospitals, and beef. While they’re nothing new over the past two years, targets have become increasingly high-profile and the adverse impacts of ransomware have similarly become more dire. Based on my read of the reports and incident reviews, these attacks seem to largely be affecting physical infrastructure assets: workstations, servers sitting in closets, and small-scale data center operations. Given this trend, it might be easy conclude that running in AWS, Azure, or Google Cloud offers some level of protection. I strongly doubt it, and I think ransomware is about to get worse.]]></summary></entry><entry><title type="html">Software-defined networks with FreeBSD Jails</title><link href="https://brokenco.de//2021/03/07/sdn-with-freebsd-jails.html" rel="alternate" type="text/html" title="Software-defined networks with FreeBSD Jails" /><published>2021-03-07T00:00:00+00:00</published><updated>2021-03-07T00:00:00+00:00</updated><id>https://brokenco.de//2021/03/07/sdn-with-freebsd-jails</id><content type="html" xml:base="https://brokenco.de//2021/03/07/sdn-with-freebsd-jails.html"><![CDATA[<p>As a comprehensive operating system FreeBSD never ceases to impress me, the
recent iterations of <a href="https://docs.freebsd.org/en/books/handbook/jails/">FreeBSD
Jails</a> as an example have been an
absolute joy to use. The introduction of the
<a href="https://www.freebsd.org/cgi/man.cgi?query=vnet&amp;apropos=0&amp;sektion=0&amp;manpath=FreeBSD+12.2-RELEASE+and+Ports&amp;arch=default&amp;format=html">vnet(9)</a>
network subsystem has completely transformed what I had originally thought
about software-defined networking. My previous exposure to the concept of
<a href="https://en.wikipedia.org/wiki/Software-defined_networking">software-defined
networking</a> was
through both <a href="https://www.openstack.org/">OpenStack</a> and Docker, two very
different approaches to the broad domain of “SDN”. FreeBSD’s <code class="language-plaintext highlighter-rouge">vnet</code> system has
resonated most strongly with me and has allowed me some measure of success in
deploying real production-grade virtualized networks.</p>

<p>One of the first blog posts I read was <a href="https://klarasystems.com/articles/virtualize-your-network-on-freebsd-with-vnet/">this post by klara
systems</a>,
which despite being painfully devoid of diagrams, does a fairly good job of
introducing the vnet feature set. In short, vnet allows jails to each have their
own isolated networking stack <em>including</em> the ability to assign virtual or
<strong>hardware interfaces</strong> to the jail. I cannot overstate how much of a super-power this is.</p>

<p>Consider an example where you want to do some testing of network security
tooling whether your firewall rules, intrusion detection, etc. You can set up a
bridge device to act as a “virtual switch” along with
<a href="https://www.freebsd.org/cgi/man.cgi?query=epair&amp;apropos=0&amp;sektion=0&amp;manpath=FreeBSD+12.2-RELEASE+and+Ports&amp;arch=default&amp;format=html">epair(4)</a>
devices to act as “virtual patch cables” to create a completely virtualized
network as diagrammed below:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>+---------------------+
| Host                |
|                     |
| +-----------------+ |
| | ixl0 (ethernet) | |  +-------------+
| +-----------------+ |  | snort-jail  |
|                     |  |             |
| +---------+     +------&gt; epair1b     |
| | bridge0 |     |   |  +-------------+
| |         |     |   |
| | epair1a +-----+   |  +-----------------+
| | epair2a +------+  |  | some other jail |
| |         |      |  |  |                 |
| +---------+      +-----&gt; epair2b         |
+---------------------+  +-----------------+
</code></pre></div></div>

<p>In the setup described above, <code class="language-plaintext highlighter-rouge">epair1a</code> and <code class="language-plaintext highlighter-rouge">epair2a</code> would not have IP addresses:</p>

<ul>
  <li><code class="language-plaintext highlighter-rouge">bridge0</code>: could be set up with <code class="language-plaintext highlighter-rouge">10.0.100.1</code></li>
  <li><code class="language-plaintext highlighter-rouge">epair1b</code>: would then be assigned <code class="language-plaintext highlighter-rouge">10.0.100.2</code></li>
  <li><code class="language-plaintext highlighter-rouge">epair2b</code>: would also take <code class="language-plaintext highlighter-rouge">10.0.100.3</code></li>
</ul>

<p>If it was desired to have <code class="language-plaintext highlighter-rouge">snort-jail</code> be able to route out to the network
connected on the host’s <code class="language-plaintext highlighter-rouge">ixl0</code> interface, then the host would need to be
configured as a
<a href="https://docs.freebsd.org/en_US.ISO8859-1/books/handbook/network-routing.html">gateway</a>
and <code class="language-plaintext highlighter-rouge">snort-jail</code> would the treat <code class="language-plaintext highlighter-rouge">10.0.100.1</code> as its default router.</p>

<p>Contrary to my experiences with Docker-based networking this is natively part
of the FreeBSD network stack, rather than iptables-style smoke and mirrors. As
a result, I have found this set up not only to be incredibly reliable but also
<em>very</em> nestable. It’s totally feasible to nest these virtualized network setups
to create completely isolated test networks or all manners of different network
topologies for development or production workloads.</p>

<p>On top of that, because you can assign <strong>hardware interfaces</strong> to Jails, it is
also possible to take another NIC (e.g. <code class="language-plaintext highlighter-rouge">ixl1</code>) and assign it into a Jail such
that the Jail could manage its own network, <code class="language-plaintext highlighter-rouge">pf</code>, and gateway configuration,
completely separate from the jail host. Thereby allowing a fairly strong
segmentation of <em>physical</em> networks by way of the FreeBSD vnet jails
infrastructure.</p>

<p><strong>So cool!</strong></p>

<p>For my own purposes I have been using an approach which is much better
documented in <a href="https://alpha.pkgbase.live/howto/jails.html">this page</a> for
isolating network services. For example, I run a <a href="https://torproject.org">Tor</a> relay inside
of a Jail with an isolated network, and then use
<a href="https://www.freebsd.org/cgi/man.cgi?query=pf&amp;apropos=0&amp;sektion=0&amp;manpath=FreeBSD+12.2-RELEASE+and+Ports&amp;arch=default&amp;format=html">pf</a>
on the jail host to restrict that network from ever reaching out onto my LAN.
It can only talk to the public internet, no other virtual or physical networks.
From a security standpoint this <em>should</em> mean even a compromise of the Tor
daemon would limit the blast-radius to that jail alone.</p>

<p>Since there are few practical limits on how many bridges and <code class="language-plaintext highlighter-rouge">epair</code> devices I
can create, I have taken the path of creating multiple independent networks for
Jails which don’t need to talk to each other or the same set of shared
resources. This gives me some level of comfort when trying out new pieces of
software or sharing services with other users that have varying trust profiles.</p>

<p>Overall I am immensely pleased with what has been possible between FreeBSD,
Jails, and <code class="language-plaintext highlighter-rouge">pf</code> to create rich topologies of software-defined networks for my
production and development needs. If you are curious to learn more, <a href="https://klarasystems.com/articles/virtualize-your-network-on-freebsd-with-vnet/">this blog
post</a>
and <a href="https://alpha.pkgbase.live/howto/jails.html">this page</a> are definitely worth reading!</p>]]></content><author><name>R. Tyler Croy</name></author><category term="freebsd" /><category term="security" /><summary type="html"><![CDATA[As a comprehensive operating system FreeBSD never ceases to impress me, the recent iterations of FreeBSD Jails as an example have been an absolute joy to use. The introduction of the vnet(9) network subsystem has completely transformed what I had originally thought about software-defined networking. My previous exposure to the concept of software-defined networking was through both OpenStack and Docker, two very different approaches to the broad domain of “SDN”. FreeBSD’s vnet system has resonated most strongly with me and has allowed me some measure of success in deploying real production-grade virtualized networks.]]></summary></entry><entry><title type="html">Multiple Let’s Encrypt domains in a single Nginx server block</title><link href="https://brokenco.de//2021/02/15/dynamic-certificates-nginx.html" rel="alternate" type="text/html" title="Multiple Let’s Encrypt domains in a single Nginx server block" /><published>2021-02-15T00:00:00+00:00</published><updated>2021-02-15T00:00:00+00:00</updated><id>https://brokenco.de//2021/02/15/dynamic-certificates-nginx</id><content type="html" xml:base="https://brokenco.de//2021/02/15/dynamic-certificates-nginx.html"><![CDATA[<p><a href="https://nginx.org">Nginx</a> is a fantastic web server and reverse proxy to use
with <a href="https://letsencrypt.org/">Let’s Encrypt</a>, but when dealing with multiple
domains it can be a bit tedious to configure.  I have been moving services into
more <a href="https://freebsd.org">FreeBSD</a> jails as I alluded to <a href="/2021/02/02/freebsd-pkg-with-an-offline-jail.html">in my previous
post</a>, among them the
general Nginx proxy jail which I have serving my HTTP-based services. Using
Let’s Encrypt for TLS, I found myself declaring multiple <code class="language-plaintext highlighter-rouge">server</code> blocks inside
my virtual host configurations to handle the apex domain (e.g.
<code class="language-plaintext highlighter-rouge">dotdotvote.com</code>), the <code class="language-plaintext highlighter-rouge">www</code> subdomain, and vanity domains (e.g.
<code class="language-plaintext highlighter-rouge">dotdot.vote</code>). With the help <code class="language-plaintext highlighter-rouge">Membear</code> and <code class="language-plaintext highlighter-rouge">MTecknology</code> in the <code class="language-plaintext highlighter-rouge">#nginx</code>
channel on <a href="https://freenode.net">Freenode</a>, I was able to refactor multiple
largely redundant <code class="language-plaintext highlighter-rouge">server</code> blocks into one.</p>

<p>With Nginx 1.15.9 a feature I have seen referred to as “dynamic certificates”
was released. Originally the <code class="language-plaintext highlighter-rouge">ssl_certificate</code> and <code class="language-plaintext highlighter-rouge">ssl_certificate_key</code> were
loaded when Nginx <em>started</em>. This meant that you could not refer to any of the
<a href="http://nginx.org/en/docs/varindex.html">Nginx variables</a> when creating the
setting. With dynamic certificates, the resolution of the <code class="language-plaintext highlighter-rouge">ssl_certificate</code>
directive is done <em>later</em> by the worker(s) process(es). It’s a <em>very</em> handy
feature!</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>server {
  listen 443 ssl http2;
  listen [::]:443 ssl http2;
  server_name dotdotvote.com www.dotdotvote.com dotdot.vote;
  ##
  # Certificates
  ssl_certificate     /usr/local/etc/letsencrypt/live/$ssl_server_name/fullchain.pem;
  ssl_certificate_key /usr/local/etc/letsencrypt/live/$ssl_server_name/privkey.pem;

  # ... snip ...
  location / {
    # ... snip ...
    proxy_pass http://dotdotvote;
  }
}
</code></pre></div></div>

<p>In the example above, I’m using the <code class="language-plaintext highlighter-rouge">$ssl_server_name</code> variable which will
correspond to the server name requested in the SNI part of the TLS payload.
This ensures that the right hostname’s certificate is utilized.</p>

<p>My first attempt was with <code class="language-plaintext highlighter-rouge">$server_name</code>, which I recommend you avoid using
<code class="language-plaintext highlighter-rouge">$server_name</code> since that will not be computed per request. For example, in the
block below the <code class="language-plaintext highlighter-rouge">$server_name</code> variable will <em>always</em> be <code class="language-plaintext highlighter-rouge">dotdotvote.com</code> and
requests served on <code class="language-plaintext highlighter-rouge">www.dotdotvote.com</code> will use the incorrect certificate.:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>server {
    server_name dotdotvote.com www.dotdotvote.com;

    ssl_certificate '/some/path/$server_name/fullchain.pem';
}
</code></pre></div></div>

<p>When I was originally setting this up, I also stumbled into some “Permission
denied” errors from Nginx. With the static certificate declaration, the <em>main</em>
Nginx process would load the file. That process would run as root before
dropping privileges to the <code class="language-plaintext highlighter-rouge">www</code> user for the Nginx worker(s). To address this
I needed to go change filesystem ownership in order for the <code class="language-plaintext highlighter-rouge">www</code> user to
properly read the certificate files.</p>

<hr />

<p>In retrospect, this feature seems relatively simple to use if you have a good
understanding of the Nginx process and permissions model. Suffice it to say, I
wouldn’t have figured this out without a bit of help from the folks in
<code class="language-plaintext highlighter-rouge">#nginx</code>. With the change in place my Nginx configurations are now much more
succinct and readable!</p>]]></content><author><name>R. Tyler Croy</name></author><category term="nginx" /><category term="freebsd" /><category term="security" /><summary type="html"><![CDATA[Nginx is a fantastic web server and reverse proxy to use with Let’s Encrypt, but when dealing with multiple domains it can be a bit tedious to configure. I have been moving services into more FreeBSD jails as I alluded to in my previous post, among them the general Nginx proxy jail which I have serving my HTTP-based services. Using Let’s Encrypt for TLS, I found myself declaring multiple server blocks inside my virtual host configurations to handle the apex domain (e.g. dotdotvote.com), the www subdomain, and vanity domains (e.g. dotdot.vote). With the help Membear and MTecknology in the #nginx channel on Freenode, I was able to refactor multiple largely redundant server blocks into one.]]></summary></entry><entry><title type="html">Intentionally leaking AWS keys</title><link href="https://brokenco.de//2021/01/15/leaking-aws-keys.html" rel="alternate" type="text/html" title="Intentionally leaking AWS keys" /><published>2021-01-15T00:00:00+00:00</published><updated>2021-01-15T00:00:00+00:00</updated><id>https://brokenco.de//2021/01/15/leaking-aws-keys</id><content type="html" xml:base="https://brokenco.de//2021/01/15/leaking-aws-keys.html"><![CDATA[<p>“Never check secrets into source control” is one of those <em>rules</em> that are 100%
correct, until it’s not. There are no universal laws in software, and recently
I had a reason to break this one. I checked AWS keys into a Git repository. I
then pushed those commits to a <em>public</em> repository on GitHub. I did this
<strong>intentionally</strong>, and lived to tell the tale. You almost certainly should
never do this, so I thought I would share what happens when you do.</p>

<p>I can imagine you thinking: “this guy posted his AWS credentials on purpose? He
must be an idiot.” I don’t disagree with your conclusion, but just let me
explain!</p>

<p>My use-case is pretty simple: the
<a href="https://github.com/delta-io/delta-rs">delta-rs</a> project needed a real S3
bucket to do some integration testing. I decided to set up a real S3 bucket for
our (read-only) integration tests. Fortunately our tests just needed to
retrieve objects from a bucket to confirm that an S3 bucket is presenting
itself as a Delta table properly. I would have <em>never</em> done this if we needed
“write” operations on the bucket.</p>

<h2 id="preparing">Preparing</h2>

<p>AWS has an integral access control framework called IAM, not to be confused
with an anagram of “AMI” which <a href="https://twitter.com/QuinnyPig">Corey Quinn</a> can
help you learn how to pronounce. IAM allows crafting policies and roles for
just about everything in AWS a dozen or more different ways. It slices, it
dices, it keeps your buckets safe. It is also configured with JSON, which is
awful, but I’ll have to save those rantings for another blog post. Anyways,
here’s the read-only policy that I set up for the bucket:</p>

<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w">
    </span><span class="nl">"Version"</span><span class="p">:</span><span class="w"> </span><span class="s2">"2012-10-17"</span><span class="p">,</span><span class="w">
    </span><span class="nl">"Statement"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
        </span><span class="p">{</span><span class="w">
            </span><span class="nl">"Sid"</span><span class="p">:</span><span class="w"> </span><span class="s2">"VisualEditor0"</span><span class="p">,</span><span class="w">
            </span><span class="nl">"Effect"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Allow"</span><span class="p">,</span><span class="w">
            </span><span class="nl">"Action"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
                </span><span class="s2">"s3:GetLifecycleConfiguration"</span><span class="p">,</span><span class="w">
                </span><span class="s2">"s3:GetBucketTagging"</span><span class="p">,</span><span class="w">
                </span><span class="s2">"s3:GetInventoryConfiguration"</span><span class="p">,</span><span class="w">
                </span><span class="s2">"s3:GetObjectVersionTagging"</span><span class="p">,</span><span class="w">
                </span><span class="s2">"s3:ListBucketVersions"</span><span class="p">,</span><span class="w">
                </span><span class="s2">"s3:GetBucketLogging"</span><span class="p">,</span><span class="w">
                </span><span class="s2">"s3:ListBucket"</span><span class="p">,</span><span class="w">
                </span><span class="s2">"s3:GetAccelerateConfiguration"</span><span class="p">,</span><span class="w">
                </span><span class="s2">"s3:GetBucketPolicy"</span><span class="p">,</span><span class="w">
                </span><span class="s2">"s3:GetObjectVersionTorrent"</span><span class="p">,</span><span class="w">
                </span><span class="s2">"s3:GetObjectAcl"</span><span class="p">,</span><span class="w">
                </span><span class="s2">"s3:GetEncryptionConfiguration"</span><span class="p">,</span><span class="w">
                </span><span class="s2">"s3:GetBucketObjectLockConfiguration"</span><span class="p">,</span><span class="w">
                </span><span class="s2">"s3:GetBucketRequestPayment"</span><span class="p">,</span><span class="w">
                </span><span class="s2">"s3:GetObjectVersionAcl"</span><span class="p">,</span><span class="w">
                </span><span class="s2">"s3:GetObjectTagging"</span><span class="p">,</span><span class="w">
                </span><span class="s2">"s3:GetMetricsConfiguration"</span><span class="p">,</span><span class="w">
                </span><span class="s2">"s3:GetBucketOwnershipControls"</span><span class="p">,</span><span class="w">
                </span><span class="s2">"s3:GetBucketPublicAccessBlock"</span><span class="p">,</span><span class="w">
                </span><span class="s2">"s3:GetBucketPolicyStatus"</span><span class="p">,</span><span class="w">
                </span><span class="s2">"s3:ListBucketMultipartUploads"</span><span class="p">,</span><span class="w">
                </span><span class="s2">"s3:GetObjectRetention"</span><span class="p">,</span><span class="w">
                </span><span class="s2">"s3:GetBucketWebsite"</span><span class="p">,</span><span class="w">
                </span><span class="s2">"s3:GetBucketVersioning"</span><span class="p">,</span><span class="w">
                </span><span class="s2">"s3:GetBucketAcl"</span><span class="p">,</span><span class="w">
                </span><span class="s2">"s3:GetObjectLegalHold"</span><span class="p">,</span><span class="w">
                </span><span class="s2">"s3:GetBucketNotification"</span><span class="p">,</span><span class="w">
                </span><span class="s2">"s3:GetReplicationConfiguration"</span><span class="p">,</span><span class="w">
                </span><span class="s2">"s3:ListMultipartUploadParts"</span><span class="p">,</span><span class="w">
                </span><span class="s2">"s3:GetObject"</span><span class="p">,</span><span class="w">
                </span><span class="s2">"s3:GetObjectTorrent"</span><span class="p">,</span><span class="w">
                </span><span class="s2">"s3:GetBucketCORS"</span><span class="p">,</span><span class="w">
                </span><span class="s2">"s3:GetAnalyticsConfiguration"</span><span class="p">,</span><span class="w">
                </span><span class="s2">"s3:GetObjectVersionForReplication"</span><span class="p">,</span><span class="w">
                </span><span class="s2">"s3:GetBucketLocation"</span><span class="p">,</span><span class="w">
                </span><span class="s2">"s3:GetObjectVersion"</span><span class="w">
            </span><span class="p">],</span><span class="w">
            </span><span class="nl">"Resource"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
                </span><span class="s2">"arn:aws:s3:::deltars"</span><span class="p">,</span><span class="w">
                </span><span class="s2">"arn:aws:s3:::deltars/*"</span><span class="w">
            </span><span class="p">]</span><span class="w">
        </span><span class="p">}</span><span class="w">
    </span><span class="p">]</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>

<p>I also set up an <a href="https://aws.amazon.com/aws-cost-management/aws-budgets/">AWS
Budget</a> to alert me
should this start to ever cost real money. My currently monthly costs in this
AWS account are almost $1.50, so my budget is set such that if/when this
starts costing me more than a couple of dollars a month, AWS will email me so I
can figure out what to do in order to save my snapple money.</p>

<p>Finally, I created an IAM user for the integration tests. This IAM user has a
single IAM policy attached to it, listed out above. I then took the AWS access
key and secret key ID for the IAM user and checked those into Git.</p>

<hr />

<p><strong>2021-01-19 update:</strong> An anonymous reader points out:</p>

<p><em>Certain AWS APIs cannot be disabled via IAM, <a href="https://docs.aws.amazon.com/STS/latest/APIReference/API_GetCallerIdentity.html">including
<code class="language-plaintext highlighter-rouge">sts:GetCallerIdentify</code></a>
which in turn allows anyone with the public credentials to run the AWS
equivalent of <code class="language-plaintext highlighter-rouge">whoami</code>:</em></p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>% AWS_PROFILE=rtyler aws sts get-caller-identity
{
    "UserId": "AIDAX7EGEQ7F24XVIBAAL",
    "Account": "547889645515",
    "Arn": "arn:aws:iam::547889645515:user/deltars-ro"
}
</code></pre></div></div>

<p><em>AWS account numbers and IAM user ARNs are not especially privileged but be
aware that publishing access keys has a side effect of disclosing those too.</em></p>

<hr />

<h2 id="boom-goes-the-dynamite">Boom goes the dynamite</h2>

<p>After preparing the integration tests, I pushed <a href="https://github.com/delta-io/delta-rs/pull/63">my pull
request</a> at <strong>13:05 PST</strong>. When
pushing code to GitHub, anything that looks like an AWS access key is
immediately identified by robots around the world, most of them
malicious in intent, but a few designed to help developers like me who make silly mistakes.</p>

<p>At <strong>13:05:36 PST</strong>, an AWS Support Case was opened in my account:</p>

<blockquote>
  <p>Dear AWS customer,</p>

  <p>We have become aware that the AWS Access Key AKIAX7EGEQ7FT6CLQGWH, belonging to IAM User deltars-ro, along with the corresponding Secret Key is publicly available online at https://github.com/rtyler/delta.rs/blob/b3581ee06eee26d971bd3b76bb788c85ecf0c6c0/rust/tests/s3_test.rs .</p>

  <p>Your security is important to us and this exposure of your account’s IAM credentials poses a security risk to your AWS account, could lead to excessive charges from unauthorized activity, and violates the AWS Customer Agreement or other agreement with us governing your use of our Services.</p>

  <p>To protect your account from excessive charges and unauthorized activity, we have applied the “AWSCompromisedKeyQuarantine” AWS Managed Policy (“Quarantine Policy”) to the IAM User listed above. The Quarantine Policy applied to the User protects your account by limiting permissions for high risk AWS services.
You can view the policy by going here: https://console.aws.amazon.com/iam/home#policies/arn:aws:iam::aws:policy/AWSCompromisedKeyQuarantine$jsonEditor?section=permissions .</p>

  <p>For your security, DO NOT remove the Quarantine Policy before following the instructions below. In cases where the Quarantine Policy is causing production issues you may detach the policy from the user. NOTE: Only users with admin privileges or with access to iam:DetachUserPolicy may remove the policy. For instructions on how to remove managed policies go here: https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_manage-attach-detach.html#remove-policies-console . In the event of the unauthorized use of your AWS account, we may, at our sole discretion, provide you with concessions. However, a failure to follow the instructions below may jeopardize your ability to receive a concession.</p>

  <p>If you believe you’ve received this note in error, please contact us immediately via the support case.</p>

  <p>PLEASE FOLLOW THE INSTRUCTIONS BELOW TO SECURE YOUR ACCOUNT:</p>

  <p>Step 1: Delete or rotate the exposed AWS Access Key AKIAX7EGEQ7FT6CLQGWH. To delete IAM User Keys go to your AWS Management Console here: https://console.aws.amazon.com/iam/home#users . To delete Root User Keys go here: https://console.aws.amazon.com/iam/home#security_credential .</p>

  <p>If your application uses the exposed Access Key, you need to replace the Key. To replace the Key, first create a second Key (at that point both Keys will be active) and then modify your application to use the new Key.
Then disable (but do not delete) the exposed Key by clicking on the “Make inactive” option in the console. If there are any problems with your application, you can reactivate the exposed Key. When your application is fully functional using the new Key, please delete the exposed Key.</p>

  <p>NOTE: Only rotating or deleting the exposed Key may not be sufficient to protect your account, see Step 2.</p>

  <p>Step 2: Check your CloudTrail log for unsanctioned activity such as the creation of unauthorized IAM users, policies, roles or temporary security credentials. To secure your account please delete any unauthorized IAM users, roles and policies, and revoke any temporary credentials.</p>

  <p>To delete unauthorized IAM User, navigate to https://console.aws.amazon.com/iam/home#users . To delete unauthorized policies go here: https://console.aws.amazon.com/iam/home#/policies . To delete unauthorized roles go here: https://console.aws.amazon.com/iam/home#/roles  .</p>

  <p>Unauthorized temporary credentials may have been created for the IAM User deltars-ro with the exposed AWS Access Key AKIAX7EGEQ7FT6CLQGWH. You can revoke temporary credentials by following instructions outlined here: https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_control-access_disable-perms.html#denying-access-to-credentials-by-issue-time . Temporary credentials can also be revoked by deleting the IAM User. NOTE: Deleting IAM Users may impact production workloads and should be done with care.</p>

  <p>Step 3: Check your CloudTrail log to review your AWS account for any unauthorized AWS usage, such as unauthorized EC2 instances, Lambda functions or EC2 Spot bids. You can also check usage by logging into your AWS Management Console and reviewing each service page. The “Bills” page in the Billing console can also be checked for unexpected usage. https://console.aws.amazon.com/billing/home#/bill</p>

  <p>Please keep in mind that unauthorized usage can occur in any region and that your console may show you only one region at a time. To switch between regions, you can use the dropdown in the top-right corner of the console screen.</p>

  <p>Please take steps to prevent any new credentials from being publicly exposed. See Best Practices of Managing your Access Keys at http://docs.aws.amazon.com/general/latest/gr/aws-access-keys-best-practices.html .</p>

  <p>WE RECOMMEND THAT YOU ENABLE AMAZON GUARDDUTY:</p>

  <p>Amazon GuardDuty is an AWS threat detection service that helps you continuously monitor and protect your AWS accounts and workloads. Enabling Amazon GuardDuty on your accounts gives you further visibility into malicious or unauthorized activity, alerting you to take action in order to reduce the risk of harm. To learn more, visit: https://aws.amazon.com/guardduty .</p>

  <p>If you have any questions, you can contact us by accessing the newly created Support Case in your account’s Support Center. If you do not see a new case, you can create a case from the Support Center here: https://console.aws.amazon.com/support/home?#/</p>

  <p>Thank you for your immediate attention to this matter.</p>
</blockquote>

<p>I <em>also</em> got emails from two third party services
<a href="https://gitguardian.com">GitGuardian</a> at <strong>13:09 PST</strong> and
<a href="https://leakd.io">leakd.io</a> at <strong>14:56 PST</strong>. Nice try folks, but AWS was
already on top of it within literal seconds of my git push.</p>

<p>I ignored the third party services and responded to the AWS Support Case to let
them know that my disclosure was in fact intentional. The support person surely
rolled their eyes before reminding me that I would be responsible for charges
on the account and still recommended that I:</p>

<ul>
  <li>Change the password for the root account.</li>
  <li>Delete and rotate all access keys.</li>
  <li>Check for possible unauthorized usage.</li>
</ul>

<hr />

<p>Normally this story doesn’t end well. I did this on purpose and planned
accordingly. There is one incidence of leaking AWS keys on GitHub which I personally know the details of (friend of a friend, I swear!).
An errant <code class="language-plaintext highlighter-rouge">git add</code>
resulted in a local credentials file being pushed to a personal, but public
repository. Because the email account linked to the AWS account was not
regularly checked, the key was used abusively to rack up a few hundred dollars
on an AWS bill before the keys were revoked.</p>

<p>If you add anything that looks like AWS keys to a public repository, website,
or really anything on the internet, malicious actors will download the keys and
try to launch services in your AWS account. Typically cryptocurrency miners or
spam gateways, anything that costs a lot of money which they’re happy you’ve
volunteered to pay for.</p>

<p>Don’t check your AWS credentials into GitHub!</p>

<p>But if you must, do it safely :)</p>]]></content><author><name>R. Tyler Croy</name></author><category term="security" /><category term="github" /><category term="aws" /><summary type="html"><![CDATA[“Never check secrets into source control” is one of those rules that are 100% correct, until it’s not. There are no universal laws in software, and recently I had a reason to break this one. I checked AWS keys into a Git repository. I then pushed those commits to a public repository on GitHub. I did this intentionally, and lived to tell the tale. You almost certainly should never do this, so I thought I would share what happens when you do.]]></summary></entry><entry><title type="html">Passing credentials to Otto steps</title><link href="https://brokenco.de//2020/10/28/otto-credentials.html" rel="alternate" type="text/html" title="Passing credentials to Otto steps" /><published>2020-10-28T00:00:00+00:00</published><updated>2020-10-28T00:00:00+00:00</updated><id>https://brokenco.de//2020/10/28/otto-credentials</id><content type="html" xml:base="https://brokenco.de//2020/10/28/otto-credentials.html"><![CDATA[<p>One of the major problems I want to solve with
<a href="https://github.com/rtyler/otto">Otto</a> is that in many CI/CD tools secrets and
credentials can be inadvertently leaked. Finding a way to allow for the secure
use of credentials without giving developers direct access to the secrets is
something <em>most</em> CI/CD systems fail at today. My hope is that Otto will succeed
because this is a problem being considered from the beginning.  In this post,
I’m going to share some of the thoughts I currently have on how Otto can pass
credentials around while removing or minimizing the possibility for them to be
leaked by user code.</p>

<p>In the course of my career I have spent a <strong>lot</strong> of time on this problem
already. In the <a href="https://jenkins.io">Jenkins project</a> we have had to deal with
a number of “credential integrity” problems where plugins, or more frequently
users’ pipelines end up leaking credentials. I have some thoughts about this
problem with Jenkins written down already in these posts:</p>

<ul>
  <li><a href="/2019/04/30/pipeline-secure-enclave.html">Thoughts about a secure enclave for Jenkins Pipeline</a>, musing on how Jenkins could improve or otherwise restrict credentials access.</li>
  <li><a href="/2019/04/15/trust-and-jenkins.html">Jenkins should not be the only line of defense</a>, noting the importance of defense in depth and siloing “build” from “deploy” after a high profile breach via an open source project’s CI system.</li>
  <li><a href="/2019/02/22/its-not-credentials-stealing.html">It’s not stealing when you’re giving them
away</a>, outlining the key
problem with exposing credentials to user-defined code in a pipeline. Choice
quote: “<em>fundamentally, exposing credentials to user-defined code will always
have the possibility for disclosure.</em>”</li>
  <li><a href="/2018/08/14/gitops-and-secrets.html">It’s no secret where GitOps falls down</a>, discussing the challenges of GitOps and secrets management when “everything is code.”</li>
</ul>

<p>Ultimately, the key challenge is drawing the boundary between “arbitrary
user-defined code” and credentials. Consider an open source project which
accepts pull requests and uses their CI system to validate proposed changes.
In this contrived example, pretend that the CI scripting for the project will:</p>

<ul>
  <li>Build the package</li>
  <li>Run the tests</li>
  <li>Push a snapshot binary <em>somewhere</em> for any manual testing that may be desired.</li>
</ul>

<p>Presumably only the last step requires any type of credential. For
demonstration, I will use Jenkins Pipeline but the problem is present in many
other systems:</p>

<div class="language-groovy highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">sh</span> <span class="s1">'make package'</span>
<span class="n">sh</span> <span class="s1">'make tests'</span>
<span class="n">withCredentials</span><span class="o">([</span><span class="n">string</span><span class="o">(</span><span class="nl">credentialsId:</span> <span class="s1">'mytoken'</span><span class="o">,</span> <span class="nl">variable:</span> <span class="s1">'TOKEN'</span><span class="o">)])</span> <span class="o">{</span>
    <span class="n">sh</span> <span class="s1">'make deploy'</span>
<span class="o">}</span>
</code></pre></div></div>

<p>There are various mechanisms to prevent a pull request from modifying the
<code class="language-plaintext highlighter-rouge">Jenkinsfile</code> (in this example), and also ways to prevent outputting the
<code class="language-plaintext highlighter-rouge">mytoken</code> credentials into the log stream. Ultimately however, since the
credential is being exposed as an <em>environment variable</em> which is then being
made available to user-defined code, in this case a <code class="language-plaintext highlighter-rouge">Makefile</code>. The only two ways to prevent a malicious pull request from using the token would be to:</p>

<ul>
  <li>Not run the <code class="language-plaintext highlighter-rouge">make deploy</code> script for pull requests</li>
  <li>Only use the <code class="language-plaintext highlighter-rouge">main</code> branch version of the <code class="language-plaintext highlighter-rouge">Makefile</code> when running the pull request. (<em>note</em>: Jenkins basically does a variant of this by default with the <code class="language-plaintext highlighter-rouge">Jenkinsfile</code>)</li>
</ul>

<p>My thinking right now is that <strong>Otto should never expose credentials as
environment variables</strong>.</p>

<h2 id="how-this-could-work">How this could work</h2>

<p>In <a href="/2019/02/22/its-not-credentials-stealing.html">this blog post</a> I alluded to the solution I am thinking about for Otto:</p>

<blockquote>
  <p>The simplest [pattern], the one which exposes those credentials to user-defined
processes is fundamentally flawed. In Jenkins, an administrator can define
credentials when configuring some plugins such as the Slack Notification
plugin. In the case of Slack, the user cannot access the Slack credentials, but
can invoke <code class="language-plaintext highlighter-rouge">slackSend</code> from within their Jenkins Pipeline. I strongly believe
that variations on this pattern, wherein <strong>an administrator defines a credential
and only allows administrator-governed code to utilize that credential, are the
only ways in which credentials exposure can be avoided</strong> (barring bugs in the
code, etc).</p>
</blockquote>

<p>(<em>added emphasis</em>)</p>

<p>This is effectively <em>exactly</em> what I am thinking for Otto: credentials can
never be accessed as environment variables or by user-defined steps. Instead
they can be accessed from within the context of an administrator approved step
library.</p>

<p>Otto’s notion of <a href="/2020/10/18/otto-steps.html">step libraries</a> means that there
is <em>plenty</em> of room for both administrator and user-defined code, but Otto
should always know where a step comes from, and that coupled with a credential
access pattern <em>should</em> provide sufficient safety.</p>

<p>Using the above example, imagine that <code class="language-plaintext highlighter-rouge">make deploy</code> is essentially just running:</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>curl <span class="nt">-H</span> <span class="s2">"Token: </span><span class="nv">$TOKEN</span><span class="s2">"</span> <span class="nt">-X</span> PUT <span class="nt">-d</span> @artifact-20201028-312.tar.gz https://example.com/api/archive
</code></pre></div></div>

<p>In Otto, that code would need to be wrapped in a step library of some form, for
example a <code class="language-plaintext highlighter-rouge">deploy-snapshot</code> step.</p>

<p>The step below is a sketch in Ruby for ease of understanding:</p>

<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">#!/usr/bin/env ruby</span>

<span class="nb">require</span> <span class="s1">'json'</span>
<span class="nb">require</span> <span class="s1">'yaml'</span>
<span class="c1"># Imagine that this gem provides the agent control socket shim</span>
<span class="nb">require</span> <span class="s1">'otto/control'</span>

<span class="c1"># Load the invocation file which defines where the control socket lives</span>
<span class="n">invoke_data</span> <span class="o">=</span> <span class="no">YAML</span><span class="p">.</span><span class="nf">load</span><span class="p">(</span><span class="no">File</span><span class="p">.</span><span class="nf">read</span><span class="p">(</span><span class="no">ARGV</span><span class="p">.</span><span class="nf">first</span><span class="p">))</span>
<span class="n">control</span> <span class="o">=</span> <span class="n">invoke_data</span><span class="p">[</span><span class="s1">'configuration'</span><span class="p">][</span><span class="s1">'ipc'</span><span class="p">]</span>
<span class="n">artifact</span> <span class="o">=</span> <span class="n">invoke_data</span><span class="p">[</span><span class="s1">'parameters]['</span><span class="n">artifact</span><span class="s1">']

# Send the HTTP request for the credential
response = Otto::Control::send(
                {:type =&gt; :RetrieveCredential,
                :name =&gt; '</span><span class="n">mytoken</span><span class="s1">'})

exit 1 unless response.ok?

# Grab the token to use
token = response['</span><span class="n">mytoken</span><span class="s1">']['</span><span class="n">value</span><span class="s1">']

# This could be done with Net::HTTP, but just for symmetry sake
# shell out to curl
system("curl -H '</span><span class="no">Token</span><span class="p">:</span> <span class="c1">#{token}' -X PUT -d @#{artifact} https://example.com/api/archive")</span>
</code></pre></div></div>

<p>The above would then be packaged as a <code class="language-plaintext highlighter-rouge">deploy-snapshot-&lt;version&gt;.tar.gz</code> and
uploaded to the wherever Otto stores its step libraries (yet to be defined).</p>

<p>In order for this step to work properly, an administrator would need to <strong>explicitly</strong>
state that the <code class="language-plaintext highlighter-rouge">deploy-snapshot</code> can access:</p>

<ul>
  <li>Credentials of specific names, e.g. <code class="language-plaintext highlighter-rouge">mytoken</code>.</li>
  <li>All credentials.</li>
</ul>

<p>In the above example, if <code class="language-plaintext highlighter-rouge">mytoken</code> was explicitly granted, any other credential
retrieval requests would be denied and the step would error out since it did
not receive the credential.</p>

<p>I can imagine somebody granting “all credentials” access to a standard library step such as <code class="language-plaintext highlighter-rouge">sh</code>. I believe in this case the credentials would still be safe so long as:</p>

<ul>
  <li>User-defined code cannot locate the agent control socket.</li>
  <li>User-defined code cannot interact with the agent control socket. Right now
there’s not an authentication scheme applied to the agent/step control
socket, but it is trivial to add a pre-shared secret that is given to each
step’s internal machinery to use for accessing the socket.</li>
</ul>

<p>There may also be cases where an administrator is comfortable with allowing a
user-defined step, such as one that is located in the local source tree or at a
remote URL (e.g. on GitHub). For these cases, I think the initial behavior has
to be a “deny and record the request.” The administrator should then be able to
approve a user-defined step for credentials access at a specified revision such
as <code class="language-plaintext highlighter-rouge">deploy-snapshot@main</code> or <code class="language-plaintext highlighter-rouge">deploy-snapshot@1ff9033</code>.</p>

<h2 id="how-this-could-not-work">How this could not work</h2>

<p>The reason environment variables are frequently used for passing secrets around
is because it’s <em>easy</em>. Well, not only that. Developers are <em>lazy</em>. The
forceful demarcation line between which code does and does not have credential
access hasn’t been validated with lazy developers other than myself. I don’t
yet know whether the bar I am setting is too high, or just high enough for
real-world usage.</p>

<p>If the barrier to entry is too high, I can easily imagine somebody writing a
<code class="language-plaintext highlighter-rouge">withSecrets</code> step which basically just takes whatever secrets you want and
dumps them into environment variables. This would still require an
administrator to approve the step, but the system would still allow the
<code class="language-plaintext highlighter-rouge">withSecrets</code> step code to retrieve credentials and then handle them in any
insecure manner they please.</p>

<hr />

<p>I believe that the approach I describe above will prevent the easy and foolish
cases where credentials get leaked.  Ultimately, I don’t think Otto can 100%
prevent credential leakage by somebody determined to shoot themselves in the
foot.</p>

<p>If you’re a pipeline user or CI/CD systems administrator and have thoughts on
how you would want to use or lock-down credentials, please join <code class="language-plaintext highlighter-rouge">#otto</code> on
Freenode or <a href="/about">email me</a> with your thoughts!</p>]]></content><author><name>R. Tyler Croy</name></author><category term="otto" /><category term="security" /><summary type="html"><![CDATA[One of the major problems I want to solve with Otto is that in many CI/CD tools secrets and credentials can be inadvertently leaked. Finding a way to allow for the secure use of credentials without giving developers direct access to the secrets is something most CI/CD systems fail at today. My hope is that Otto will succeed because this is a problem being considered from the beginning. In this post, I’m going to share some of the thoughts I currently have on how Otto can pass credentials around while removing or minimizing the possibility for them to be leaked by user code.]]></summary></entry><entry><title type="html">Getting started with a Yubikey on openSUSE</title><link href="https://brokenco.de//2019/12/25/yubikey-opensuse.html" rel="alternate" type="text/html" title="Getting started with a Yubikey on openSUSE" /><published>2019-12-25T00:00:00+00:00</published><updated>2019-12-25T00:00:00+00:00</updated><id>https://brokenco.de//2019/12/25/yubikey-opensuse</id><content type="html" xml:base="https://brokenco.de//2019/12/25/yubikey-opensuse.html"><![CDATA[<p>If the people I know tweet enough about something, eventually I’m bound to
breakdown and just buy the thing. It happened with the Intel NUC, and now it’s
happened with <a href="https://www.yubico.com">Yubikey</a>. The Yubikey is a USB-based
security device that can do a <em>lot</em> of things, but in my case I just need it to
act as a security key for a number of websites such as GitHub, Google, and
Twitter. Much to my dismay it did not work exactly as I expected right out of
the box on my openSUSE-based laptop.</p>

<p>With recent versions of Firefox Quantum, basically Firefox released over the
past couple years, Chrome/Chromium, and Opera (apparenlty) a Yubikey can be
used directly within web browsers as an external security key. In my case,
every time I tried to configure the device on sites loaded in either Chromium or Firefox, I simply couldn’t make it work. The Yubikey would 
instead spit out a random string of garbage. This is one <em>valid</em> mode of the Yubikey,
where it acts like a pretend keyboard and generates One-Time Passwords (OTP).
In my case, I wanted it to act like a Universal 2-Factor authentication device
(U2F).</p>

<p>First try was using the Yubikey manager to poke at the device. This application
can be installed with <code class="language-plaintext highlighter-rouge">zypper in yubikey-manager-qt</code>. The installed binary is
frustratingly <strong>not</strong> called <code class="language-plaintext highlighter-rouge">yubikey-manager</code> but instead is named
<code class="language-plaintext highlighter-rouge">ykman-gui</code>. With that hurdle overcome, I tried to inspect the “FIDO2”
application on the Yubikey but that consistently fail.</p>

<p>Utilizing a mixture of “opensuse” “yubikey” and “u2f” I stumbled into an
openSUSE Buildservice link for a promising package named <code class="language-plaintext highlighter-rouge">u2f-host</code>. As luck
would have it, the <code class="language-plaintext highlighter-rouge">u2f-host</code> package ended up being the missing link between
Firefox and my Yubikey. After executing <code class="language-plaintext highlighter-rouge">zypper in u2f-host</code> and restarting
Firefox, I was able to successfully register and use my Yubikey across a myriad
of websites!</p>

<p>A tiny (nano, actually) Christmas miracle! :)</p>]]></content><author><name>R. Tyler Croy</name></author><category term="opensuse" /><category term="yubikey" /><category term="security" /><summary type="html"><![CDATA[If the people I know tweet enough about something, eventually I’m bound to breakdown and just buy the thing. It happened with the Intel NUC, and now it’s happened with Yubikey. The Yubikey is a USB-based security device that can do a lot of things, but in my case I just need it to act as a security key for a number of websites such as GitHub, Google, and Twitter. Much to my dismay it did not work exactly as I expected right out of the box on my openSUSE-based laptop.]]></summary></entry><entry><title type="html">JKS? jfc. Adding a root certificate</title><link href="https://brokenco.de//2019/09/28/jks-jfc.html" rel="alternate" type="text/html" title="JKS? jfc. Adding a root certificate" /><published>2019-09-28T00:00:00+00:00</published><updated>2019-09-28T00:00:00+00:00</updated><id>https://brokenco.de//2019/09/28/jks-jfc</id><content type="html" xml:base="https://brokenco.de//2019/09/28/jks-jfc.html"><![CDATA[<p>TLS certificates have the largest “complexity/importance” scores imaginable.
Everything about them is error prone and seemingly over-engineered from top to
bottom, yet they are one of the most important pieces of security and
authentication in our software architectures. From an engineering management
standpoint, I am finding myself adopting the rule of: estimates for any project
involving certificates should be multiplied tenfold. If the project involves
the Java Virtual Machine (JVM) and the Java Key Store (JKS), multiply by
another ten I suppose. For my own future convenience, in this blog post I would
like to outline how to add a root certificate to a Java Key Store in Red
Hat-derived environments.</p>

<p>Like many corporate environments, we have our own internal Certificate
Authorities (CAs) which all derive their chain of trust from our internal root
certificate. Accessing internal services requires that the operating system has
that root certificate, or when accessing those internal services from anything
running atop the JVM, the default JKS must have the root certificate.</p>

<p>If you search around the web for how to add root certificates, you might find
the <code class="language-plaintext highlighter-rouge">update-ca-certificates</code> command, whose CentOS/RHEl manpage has the
following:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>The directory /etc/pki/ca-trust/extracted/java/ contains a CA
certificate bundle in the java keystore file format. Distrust information
cannot be represented in this file format, and distrusted certificates are
missing from these files. File cacerts contains CA certificates trusted for TLS
server authentication.
</code></pre></div></div>

<p>You might assume, as I did, that this means the <code class="language-plaintext highlighter-rouge">update-ca-certificates</code> tool
is going to create files that the JVM picks up properly and your default JKS
will have the root certificate in place.</p>

<p>This is <em>false</em>. At least in the environments which I have tested this.</p>

<p>Digging further I found <a href="https://connect2id.com/blog/importing-ca-root-cert-into-jvm-trust-store">this blog post</a> and used the following command to import the root certificate into JKS after installing it on the system at large:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>keytool -importcert -alias startssl -keystore $JAVA_HOME/jre/lib/security/cacerts -storepass changeit -file ca.der
</code></pre></div></div>

<p>Using the SSLPoke tool referenced in <a href="https://confluence.atlassian.com/kb/unable-to-connect-to-ssl-services-due-to-pkix-path-building-failed-779355358.html">this Atlassian knowledgebase
article</a>
I was then <em>finally</em> able to access the same internal services from native
utilities (e.g. <code class="language-plaintext highlighter-rouge">curl</code>) and from the Java-based services which I was working
with at the time.</p>

<p>In my situation, the fact that all of this was happening within Docker
containers further complicated the debugging: multiple by another 2-5 on that
engineering estimate.</p>

<p>Certificates are too important to be this painful.</p>]]></content><author><name>R. Tyler Croy</name></author><category term="security" /><category term="tls" /><category term="java" /><category term="scribd" /><summary type="html"><![CDATA[TLS certificates have the largest “complexity/importance” scores imaginable. Everything about them is error prone and seemingly over-engineered from top to bottom, yet they are one of the most important pieces of security and authentication in our software architectures. From an engineering management standpoint, I am finding myself adopting the rule of: estimates for any project involving certificates should be multiplied tenfold. If the project involves the Java Virtual Machine (JVM) and the Java Key Store (JKS), multiply by another ten I suppose. For my own future convenience, in this blog post I would like to outline how to add a root certificate to a Java Key Store in Red Hat-derived environments.]]></summary></entry><entry><title type="html">It’s not stealing when you’re giving them away</title><link href="https://brokenco.de//2019/02/22/its-not-credentials-stealing.html" rel="alternate" type="text/html" title="It’s not stealing when you’re giving them away" /><published>2019-02-22T00:00:00+00:00</published><updated>2019-02-22T00:00:00+00:00</updated><id>https://brokenco.de//2019/02/22/its-not-credentials-stealing</id><content type="html" xml:base="https://brokenco.de//2019/02/22/its-not-credentials-stealing.html"><![CDATA[<p>My exact words were “<a href="/2019/02/02/codevalet-the-failed-experiment.html">I could have likely filled half a book with thoughts and
practices on good security for
Jenkins</a>” and were I to write
that book, my
colleague <a href="https://github.com/daniel-beck">Daniel</a> would certainly be a guest
author of a few chapters, such as one on the <a href="https://jenkins.io/blog/2019/02/21/credentials-masking/">limitations of credentials
masking</a>. In that
linked post Daniel highlights some of the ways in which credentials can be
misused in Jenkins. As I <a href="https://twitter.com/agentdero/status/1098735822977650688">mentioned on
Twitter</a>, this is not
a unique problem to Jenkins, any system which allows user-defined build
processes <em>and</em> allows the use of credentials in those build processes
<strong>will</strong> allow exposure of those credentials.</p>

<p>Both Jenkins and Travis CI for example, can expose credentials or
secrets as environment variables, which can then be used in the <code class="language-plaintext highlighter-rouge">Jenkinsfile</code> or
<code class="language-plaintext highlighter-rouge">.travis.yml</code> respectively. Of course, once those credentials are exposed in
<em>any form</em> to the user-defined build process, an incompetent or malicious user
can leak those credentials.</p>

<p>Take this example from Daniel’s blog post:</p>

<div class="language-groovy highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">withCredentials</span><span class="o">([</span><span class="n">usernamePassword</span><span class="o">(</span><span class="nl">credentialsId:</span> <span class="s1">'topSecretCredentials'</span><span class="o">,</span>
                               <span class="nl">passwordVariable:</span> <span class="s1">'PWD'</span><span class="o">,</span>
                               <span class="nl">usernameVariable:</span> <span class="s1">'USR'</span><span class="o">)])</span> <span class="o">{</span>
    <span class="n">echo</span> <span class="s1">'Encrypting the password with base64, h4xx0r!'</span>
    <span class="c1">// will print e.g. dDBwczNjcjN0Cg=</span>
    <span class="c1">// which is trivially converted back to the top secret password</span>
    <span class="n">sh</span> <span class="s1">'echo $PWD | base64'</span>
<span class="o">}</span>
</code></pre></div></div>

<p>OH NOES HACKED! The only way to be certain credentials cannot be disclosed is
to never expose them to the user-defined build process in the first place.</p>

<h3 id="mitigations">Mitigations</h3>

<p>Of course, there are ways to reduce the surface area for possible credential
leakage. One approach which Daniel discusses in the blog post, is segmenting
where credentials are used:</p>

<blockquote>
  <p><em>Credentials can be defined in different scopes: Credentials defined on the root
Jenkins store (the default) will be available to all jobs on the instance. The
only exception are credentials with System scope, intended for the global
configuration only, for example, to connect to agents. Credentials defined in a
folder are only available within that folder (transitively, i.e. also in
folders inside this folder).</em></p>

  <p><em>This allows defining sensitive credentials, such as deployment credentials, on
specific folders whose contents only users trusted with those credentials are
allowed to configure: Directly in Jenkins using Matrix Authorization Plugin and
by limiting write access to repositories defining pipelines as code.</em></p>
</blockquote>

<p>There are a couple other approaches which can help:</p>

<ol>
  <li>Utilize the built-in <code class="language-plaintext highlighter-rouge">Jenkinsfile</code> “Branch Source” configuration to allow
only trusted contributors’ Pull Requests/Branches to be evaluated. This
extends a modicum of trust, basically that if somebody can write to the
repository, we trust them not to be foolish with credentials.</li>
  <li>Isolate the use of credentials to the <code class="language-plaintext highlighter-rouge">master</code> branch only. In many cases, I
doubt that pull requests will need credentials for deployment. If the need
for credentials is only once things land in master, for example, then we can
use Jenkins Pipeline to further restrict where Pipelines might use credentials. Namely,
only allowing the use once code has been merged, e.g.:
    <div class="language-groovy highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">stages</span> <span class="o">{</span>
    <span class="cm">/* .. snip .. */</span>

    <span class="n">stage</span><span class="o">(</span><span class="s1">'Deploy'</span><span class="o">)</span> <span class="o">{</span>
        <span class="n">when</span> <span class="o">{</span> <span class="n">branch</span> <span class="s1">'master'</span> <span class="o">}</span>
        <span class="n">environment</span> <span class="o">{</span>
            <span class="n">SECRET_KEY</span> <span class="o">=</span> <span class="n">credentials</span><span class="o">(</span><span class="s1">'deploy-key'</span><span class="o">)</span>
        <span class="o">}</span>
        <span class="n">steps</span> <span class="o">{</span>
            <span class="cm">/* .. */</span>
        <span class="o">}</span>
    <span class="o">}</span>
<span class="o">}</span>
</code></pre></div>    </div>
    <p>This only reduces the possible disclosure surface area, but does not remove
it entirely.</p>
  </li>
  <li>Utilize a “deployment” environment, separate from the common “build”
environment. This is more or less the approach we take in the Jenkins
project. There is a non-public Jenkins instance which actually has deployment
keys for a number of projects, and the <code class="language-plaintext highlighter-rouge">Jenkinsfile</code> for those projects is
actually run in two different Jenkins environments. This <em>also</em> only reduces
the possible disclosure surface area, since it only prevents problematic
disclosures via the Console Output or archived artifacts. It would still be
possible to run a script which POSTed to a third party service with those
credentials.</li>
  <li>Utilize <a href="https://learn.hashicorp.com/vault/secrets-management/sm-dynamic-secrets">dynamic secrets with a tool like HashiCorp
Vault</a>.
This reduces the effect of disclosure, since disclosed secrets would be invalid
after their use in the Pipeline. This is not always possible, since many of the
things we need secrets for are out of our control, such as GitHub, Slack, or
other third-party service secrets.</li>
</ol>

<p>Fundamentally, exposing credentials to user-defined code <strong>will always have the
possibility for disclosure</strong>.</p>

<h3 id="the-only-solution">The Only Solution</h3>

<p>Both Jenkins and Travis CI have multiple ways in which to use credentials. The
simplest, the one which exposes those credentials to user-defined processes is
fundamentally flawed. In Jenkins, an administrator can define credentials
when configuring some plugins such as the <a href="https://plugins.jenkins.io/slack">Slack Notification
plugin</a>. In the case of Slack, the user
cannot access the Slack credentials, but <em>can</em> invoke <code class="language-plaintext highlighter-rouge">slackSend</code> from within
their Jenkins Pipeline.  I strongly believe that variations on this pattern,
wherein an administrator defines a credential and only allows administrator-governed
code to utilize that credential, are the only ways in which credentials
exposure can be avoided (barring bugs in the code, etc).</p>

<p>Unfortunately, in Jenkins this approach only applies to plugins. There is no
way to expose credentials <strong>only</strong> to a Pipeline Shared Library, which would be 
administrator-approved code. This means there are some unfortunate limits to
the reduction in surface area that administrators can provide.</p>

<p>As I have been thinking about this problem and some <a href="/2019/02/02/codevalet-the-failed-experiment.html">other design challenges in
Jenkins</a>, and as a result
have been considering alternative system designs which would obviate many
classes of security challenges Jenkins and other CI/CD systems face. Balancing
user flexibility and security is among the harder problems in the space, and
with continuous delivery becoming the standard, I believe we need to find safer
ways to handle the <em>deployment</em> end of the pipeline.</p>]]></content><author><name>R. Tyler Croy</name></author><category term="jenkins" /><category term="security" /><category term="otto" /><summary type="html"><![CDATA[My exact words were “I could have likely filled half a book with thoughts and practices on good security for Jenkins” and were I to write that book, my colleague Daniel would certainly be a guest author of a few chapters, such as one on the limitations of credentials masking. In that linked post Daniel highlights some of the ways in which credentials can be misused in Jenkins. As I mentioned on Twitter, this is not a unique problem to Jenkins, any system which allows user-defined build processes and allows the use of credentials in those build processes will allow exposure of those credentials.]]></summary></entry></feed>