<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom" ><generator uri="https://jekyllrb.com/" version="3.10.0">Jekyll</generator><link href="https://brokenco.de//feed/by_tag/postgresql.xml" rel="self" type="application/atom+xml" /><link href="https://brokenco.de//" rel="alternate" type="text/html" /><updated>2026-10-31T17:58:32+00:00</updated><id>https://brokenco.de//feed/by_tag/postgresql.xml</id><title type="html">rtyler</title><subtitle>a moderately technical blog</subtitle><author><name>R. Tyler Croy</name></author><entry><title type="html">Crashing the index with PostgreSQL custom types</title><link href="https://brokenco.de//2026/08/14/custom-types-index.html" rel="alternate" type="text/html" title="Crashing the index with PostgreSQL custom types" /><published>2026-08-14T00:00:00+00:00</published><updated>2026-08-14T00:00:00+00:00</updated><id>https://brokenco.de//2026/08/14/custom-types-index</id><content type="html" xml:base="https://brokenco.de//2026/08/14/custom-types-index.html"><![CDATA[<p>My current work project is one of the first Aurora/PostgreSQL deployments and
already the <strong>largest</strong>. Success brings <em>scaling challenges</em> and the types of
things seasoned operators classify as “good problems to have.” Compared to
previous scaling work around <a href="https://delta.io">Delta Lake</a> the knobs and ways
in which online relational databases scale is a muscle that has atrophied in
the past decade. The relational database landscape is also quite different
especially with <a href="https://aws.amazon.com/video/watch/f2a0fcbccb8/">Amazon Aurora which is quite well
designed</a>in the past decade.
The relational database landscape is also quite different especially with
<a href="https://aws.amazon.com/video/watch/f2a0fcbccb8/">Amazon Aurora which is quite well
designed</a>.</p>

<p>At some point it doesn’t matter how “well architected” the underlying database engine was, the laws of physics still apply. The speed of light between CPU cores, storage, and memory still imposes limitations. Similar to my work
related to <a href="/2026/02/13/screaming-in-the-cloud.html">Engineering around Extreme S3
scale</a>, the difference between systems
which handle hundreds of millions of <em>something</em> and hundreds of <strong>billions</strong>
of something is substantial.</p>

<p>My original database design relied upon <a href="https://www.postgresql.org/docs/current/sql-createtype.html">Custom
types</a>, which I
really enjoy using for easily mapping complex data types into PostgreSQL.
Similar to a <code class="language-plaintext highlighter-rouge">struct</code> in Rust, at it’s most simple a custom type is a
collection of associated fields:</p>

<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">struct</span> <span class="n">Coordinate</span> <span class="p">{</span>
    <span class="n">brand</span><span class="p">:</span> <span class="n">Brand</span><span class="p">,</span>
    <span class="n">doc_id</span><span class="p">:</span> <span class="nb">u64</span><span class="p">,</span>
    <span class="n">page</span><span class="p">:</span> <span class="nb">Option</span><span class="o">&lt;</span><span class="nb">u32</span><span class="o">&gt;</span><span class="p">,</span>
<span class="p">};</span>
</code></pre></div></div>

<p>Can easily map into the database with:</p>

<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">CREATE</span> <span class="k">TYPE</span> <span class="n">coordinate</span> <span class="k">AS</span> <span class="p">(</span><span class="n">brand</span> <span class="n">BRAND</span> <span class="k">NOT</span> <span class="k">NULL</span><span class="p">,</span> <span class="n">doc_id</span> <span class="nb">BIGINT</span> <span class="k">NOT</span> <span class="k">NULL</span><span class="p">,</span> <span class="n">page</span><span class="p">:</span> <span class="nb">INTEGER</span><span class="p">);</span>
</code></pre></div></div>
<p>The utility of custom types is so <em>alluring</em> for modeling richer data in at the
database level because so much of our mental models for systems are driven by
data encapsulation. I see similar design patterns from my colleagues when working with Delta Lake tables. A preference for creating columns of deeply nested structs, despite my protestations around flattening the into wider table schemas with more columns.</p>

<p>The biggest performance downside with custom types that I observed was when
using custom types in <strong>indexes</strong> but the issue was obscured by Aurora’s
exceptional performance until we crossed into the <em>billions</em> scale.</p>

<p>For the PostgreSQL engine version we were running, custom types in indexes would only be used correctly if the full type was present in the query. Using the example above this meant that a query would <strong>not use the index correctly</strong> when querying on <code class="language-plaintext highlighter-rouge">doc_id</code> alone! Instead the query was required to construct a type inline, e.g.:</p>

<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">SELECT</span> <span class="o">*</span> <span class="k">FROM</span> <span class="n">some_table</span> 
    <span class="k">WHERE</span> <span class="k">c</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'scribd'</span><span class="p">,</span> <span class="mi">123</span><span class="p">,</span> <span class="k">NULL</span><span class="p">)::</span><span class="n">COORDINATE</span>

</code></pre></div></div>

<p>The inefficiencies of queries with a partial custom type predicate wasn’t
noticed until queries in hitting billion row tables started to slow down. Those
production query plans made it fairly obvious that the index wasn’t being used,
but it was not clear to me <em>why</em> until after we really considered the offending query plans.</p>

<p><strong>Storage</strong> of the custom type was also not something I considered until after
the billions scale where the indexes and table sizes ballooned. The <a href="https://www.postgresql.org/docs/current/datatype-enum.html">enumerated
type</a> occupies four
bytes on disk. Combined with the other two entries in the type each row
required 16 bytes of storage. With smaller scale data that’s kind of a “who
cares” amount of data but 10 billion rows that equates to around 149GiB to
store nothing but this one custom type in the table.</p>

<hr />

<p>The solution we ended up with to resolve the inefficiencies of the custom type
for query and storage performance was to switch towards a bit-masked 64-bit
integer with the custom type’s properties encoded within.</p>

<p>My colleague experimented with using a binary encoded field which had similar
benefits over a custom type. Ultimately the bitmask approach allowed us to
halve the storage requirements and improved query performance since the index
moved from a hash index to a <a href="https://www.postgresql.org/docs/current/indexes-types.html">b-tree
index</a> for
operations on that column.</p>

<p><strong>Billions</strong> is such an astronomically large number that I often explain that
the “physics of billions” is so fundamentally different that it’s like going
from Newtonian to Quantum physics: the old rules no longer apply. In our online
data system, we have tables that vary from hundreds of millions of rows, to
tens of billions of rows, and our modelling has to account for <em>both</em> which has
been an exciting challenge.</p>

<center><iframe width="560" height="315" src="https://www.youtube-nocookie.com/embed/XCv9Tviftgo?si=BzXciAynsuGwsshX" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen=""></iframe></center>

<p>The system would not have gotten this far without some of the superpowers that
PostgreSQL has to offer. Originally we deployed to PostgreSQL 17 but after
watching the above session we were convinced to upgrade to PostgreSQL 18 and
reaped the additional performance rewards offered by the upgrade.</p>

<p>PostgreSQL by default is quite fast. Aurora/PostgreSQL by default is
exceptionally fast. As a data system grows the “nice problems to have” start
popping up and query plans, APM traces, and all other sorts of telemetry become
crucial to understanding how and where to start improving performance.</p>

<p>I would not recommend avoiding custom types in PostgreSQL, there is a time and
a place for them, and we’re still using them quite heavily, just not as indexed
columns!</p>]]></content><author><name>R. Tyler Croy</name></author><category term="postgresql" /><category term="data" /><category term="dataeng" /><summary type="html"><![CDATA[My current work project is one of the first Aurora/PostgreSQL deployments and already the largest. Success brings scaling challenges and the types of things seasoned operators classify as “good problems to have.” Compared to previous scaling work around Delta Lake the knobs and ways in which online relational databases scale is a muscle that has atrophied in the past decade. The relational database landscape is also quite different especially with Amazon Aurora which is quite well designedin the past decade. The relational database landscape is also quite different especially with Amazon Aurora which is quite well designed.]]></summary></entry><entry><title type="html">Elephant in the room</title><link href="https://brokenco.de//2026/02/21/elephant-in-the-room.html" rel="alternate" type="text/html" title="Elephant in the room" /><published>2026-02-21T00:00:00+00:00</published><updated>2026-02-21T00:00:00+00:00</updated><id>https://brokenco.de//2026/02/21/elephant-in-the-room</id><content type="html" xml:base="https://brokenco.de//2026/02/21/elephant-in-the-room.html"><![CDATA[<p>just use postgres</p>]]></content><author><name>R. Tyler Croy</name></author><category term="software" /><category term="dataeng" /><category term="postgresql" /><summary type="html"><![CDATA[just use postgres]]></summary></entry><entry><title type="html">Trait not bound errors with Diesel</title><link href="https://brokenco.de//2020/09/15/surprise-nullable-types-diesel.html" rel="alternate" type="text/html" title="Trait not bound errors with Diesel" /><published>2020-09-15T00:00:00+00:00</published><updated>2020-09-15T00:00:00+00:00</updated><id>https://brokenco.de//2020/09/15/surprise-nullable-types-diesel</id><content type="html" xml:base="https://brokenco.de//2020/09/15/surprise-nullable-types-diesel.html"><![CDATA[<p>Recently I have been exploring using <a href="https://diesel.rs">Diesel</a> for a
simple Rust web application. I quickly ran into a very confusing <code class="language-plaintext highlighter-rouge">trait
bound</code> error, listed below when integrating with <code class="language-plaintext highlighter-rouge">chrono</code>. It took me a
while to understand and fix the error, which I thought I should write down
for later!</p>

<p>The error only started showing up when I queried a specific model:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>error[E0277]: the trait bound `DateTime&lt;Utc&gt;: FromSql&lt;diesel::sql_types::Nullable&lt;diesel::sql_types::Timestamptz&gt;, Pg&gt;` is not satisfied
  --&gt; src/main.rs:85:64
   |
85 |         let choices: Vec&lt;Choice&gt; = Choice::belonging_to(&amp;poll).get_results(&amp;pgconn).expect("Failed to get relations");
   |                                                                ^^^^^^^^^^^ the trait `FromSql&lt;diesel::sql_types::Nullable&lt;diesel::sql_types::Timestamptz&gt;, Pg&gt;` is not implemented for `DateTime&lt;Utc&gt;`
   |
   = help: the following implementations were found:
             &lt;DateTime&lt;Utc&gt; as FromSql&lt;diesel::sql_types::Timestamptz, Pg&gt;&gt;
   = note: required because of the requirements on the impl of `Queryable&lt;diesel::sql_types::Nullable&lt;diesel::sql_types::Timestamptz&gt;, Pg&gt;` for `DateTime&lt;Utc&gt;`
   = note: required because of the requirements on the impl of `Queryable&lt;(diesel::sql_types::Integer, diesel::sql_types::Text, diesel::sql_types::Integer, diesel::sql_types::Nullable&lt;diesel::sql_types::Timestamptz&gt;), Pg&gt;` for `(i32, std::string::String, i32, DateTime&lt;Utc&gt;)`
   = note: required because of the requirements on the impl of `Queryable&lt;(diesel::sql_types::Integer, diesel::sql_types::Text, diesel::sql_types::Integer, diesel::sql_types::Nullable&lt;diesel::sql_types::Timestamptz&gt;), Pg&gt;` for `models::Choice`
   = note: required because of the requirements on the impl of `LoadQuery&lt;PooledConnection&lt;ConnectionManager&lt;PgConnection&gt;&gt;, models::Choice&gt;` for `diesel::query_builder::SelectStatement&lt;schema::choices::table, query_builder::select_clause::DefaultSelectClause, query_builder::distinct_clause::NoDistinctClause, query_builder::where_clause::WhereClause&lt;diesel::expression::operators::Eq&lt;schema::choices::columns::poll_id, diesel::expression::bound::Bound&lt;diesel::sql_types::Integer, &amp;i32&gt;&gt;&gt;&gt;`

error: aborting due to previous error; 1 warning emitted
</code></pre></div></div>

<p>I stumbled into <a href="https://github.com/diesel-rs/diesel/issues/2445">this closed issue</a>, which didn’t really help me, but it did inspire me to review the <code class="language-plaintext highlighter-rouge">src/schema.rs</code>:</p>

<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nd">table!</span> <span class="p">{</span>
    <span class="nf">choices</span> <span class="p">(</span><span class="n">id</span><span class="p">)</span> <span class="p">{</span>
        <span class="n">id</span> <span class="k">-&gt;</span> <span class="n">Int4</span><span class="p">,</span>
        <span class="n">details</span> <span class="k">-&gt;</span> <span class="n">Text</span><span class="p">,</span>
        <span class="n">poll_id</span> <span class="k">-&gt;</span> <span class="n">Int4</span><span class="p">,</span>
        <span class="n">created_at</span> <span class="k">-&gt;</span> <span class="n">Nullable</span><span class="o">&lt;</span><span class="n">Timestamptz</span><span class="o">&gt;</span><span class="p">,</span>
    <span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>

<p>The <code class="language-plaintext highlighter-rouge">created_at</code> column is what is causing issues for me, and I noticed that it’s wrapped in a <code class="language-plaintext highlighter-rouge">Nullable</code>. Meanwhile, my struct looks like:</p>

<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nd">#[derive(Associations,</span> <span class="nd">Debug,</span> <span class="nd">Identifiable,</span> <span class="nd">Queryable,</span> <span class="nd">Serialize)]</span>
<span class="nd">#[belongs_to(Poll)]</span>
<span class="k">pub</span> <span class="k">struct</span> <span class="n">Choice</span> <span class="p">{</span>
    <span class="n">id</span><span class="p">:</span> <span class="nb">i32</span><span class="p">,</span>
    <span class="n">details</span><span class="p">:</span> <span class="nb">String</span><span class="p">,</span>
    <span class="n">poll_id</span><span class="p">:</span> <span class="nb">i32</span><span class="p">,</span>
    <span class="n">created_at</span><span class="p">:</span> <span class="n">DateTime</span><span class="o">&lt;</span><span class="n">Utc</span><span class="o">&gt;</span><span class="p">,</span>
<span class="p">}</span>
</code></pre></div></div>

<p>Turns out the <code class="language-plaintext highlighter-rouge">Nullable</code> wrapper around the <code class="language-plaintext highlighter-rouge">Timestamptz</code> SQL type was causing behavior that I didn’t want. The underlying problem ended up being that my table definition didn’t include a <code class="language-plaintext highlighter-rouge">NOT NULL</code> despite including a <code class="language-plaintext highlighter-rouge">DEFAULT NOW()</code>.</p>

<p>Naturally, altering the statement to be <code class="language-plaintext highlighter-rouge">created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()</code> caused my compilation error to disappear, fun!</p>]]></content><author><name>R. Tyler Croy</name></author><category term="rust" /><category term="diesel" /><category term="postgresql" /><summary type="html"><![CDATA[Recently I have been exploring using Diesel for a simple Rust web application. I quickly ran into a very confusing trait bound error, listed below when integrating with chrono. It took me a while to understand and fix the error, which I thought I should write down for later!]]></summary></entry><entry><title type="html">Streaming HTTP data with PostgreSQL and ExpressJS</title><link href="https://brokenco.de//2018/12/06/streaming-results-node-pg.html" rel="alternate" type="text/html" title="Streaming HTTP data with PostgreSQL and ExpressJS" /><published>2018-12-06T00:00:00+00:00</published><updated>2018-12-06T00:00:00+00:00</updated><id>https://brokenco.de//2018/12/06/streaming-results-node-pg</id><content type="html" xml:base="https://brokenco.de//2018/12/06/streaming-results-node-pg.html"><![CDATA[<p>One of the <a href="https://github.com/jenkins-infra/uplink">little applications</a> which
I built earlier this year ended up more useful than I originally anticipated.
Useful enough to have hit its first performance bottleneck! Performance
problems I generally grumble “nice problem to have” which profiling and
refactoring, but in this case I know what the performance problem was, but
lacked the appropriate solution.</p>

<p>This little application, Uplink, receives anonymous telemetry information from
short-lived “trials” defined within the <a href="https://github.com/jenkinsci/jenkins">Jenkins core
application</a>. The entire end-to-end system is
defined by the design document
<a href="https://github.com/jenkinsci/jep/blob/master/jep/214/README.adoc">JEP-214</a>.
What the JEP does not describe is how we use and analyze the data on the other
end. At the moment the “data science” behind Uplink has been exporting large
dumps of JSON information from Uplink, and then bash scripting the heck out of
it. As time has gone on the amount of data ingested, and therefore exportable,
has increased quite a bit. This growth in data has required numerous iterations
on the “Export” functionality, whilst everything else remained largely
unchanged.</p>

<p><strong>First iteration</strong></p>

<p>The first cut at “export” functionality was as simple and straightforward as
possible:</p>

<ol>
  <li>Receive authorized “Export” HTTP request</li>
  <li>Send the database <code class="language-plaintext highlighter-rouge">SELECT * FROM events WHERE ...</code></li>
  <li>Receive results</li>
  <li>Format an HTTP response with the right <code class="language-plaintext highlighter-rouge">Content-Disposition</code> headers, etc.</li>
</ol>

<p>This worked for much longer than I honestly thought it would. The Node
application lives close enough to the database to retrieve large datasets
within an HTTP timeout and deliver those to the client. Once the <strong>total</strong>
dataset exceeded a couple hundred megabytes, things stopped working.</p>

<p><strong>Second iteration</strong></p>

<p>The consumer of this data was, and still is a single person wielding bash
scripts a’plenty. To keep things as simple as possible, we changed the frontend
to require that any “Export” define a date range to export. Initially the Data
Scientist™ would request a whole week at a time, and when that stopped
working, they would request individual daily exports instead. Eventually this also
stopped working, somewhere around a <em>daily</em> dataset size of a couple of hundred
megabytes.</p>

<p><strong>Third iteration</strong></p>

<p>Clearly loading big stupid <code class="language-plaintext highlighter-rouge">SELECT * FROM</code> result sets into the application to
format and serve them to clients was not scalable. For the third iteration I
resolved to implement a direct stream from the database through the web
application to the client. In effect, I wanted the PostgreSQL database
connection to give me results <em>immediately</em> which would then be written
directly to the HTTP output stream; no in-memory storage.</p>

<p>I discovered a very useful Node package to solve the first part of the problem:
<a href="https://github.com/brianc/node-pg-query-stream/#pg-query-stream">pg-query-stream</a>.
The pg-query-stream package uses a database-side cursor to avoid the need to
create large datasets in memory on the database or web application.</p>

<p>The “trick”, which to be honest isn’t a very incredible trick since Node
streams are designed to be pluggable in this way, was to connect the
<code class="language-plaintext highlighter-rouge">pg-query-stream</code> directly to ExpressJS <code class="language-plaintext highlighter-rouge">Response</code> which looks like a writable
stream. To form a proper HTTP response, the ExpressJS handler must first write
the response code and headers, for which <code class="language-plaintext highlighter-rouge">response.send()</code> will not work, so
<code class="language-plaintext highlighter-rouge">response.writeHead</code> is used instead:</p>

<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">const</span> <span class="nx">query</span> <span class="o">=</span> <span class="k">new</span> <span class="nx">QueryStream</span><span class="p">(</span><span class="dl">'</span><span class="s1">SELECT * FROM events WHERE ...</span><span class="dl">'</span><span class="p">);</span>
<span class="kd">const</span> <span class="nx">datastream</span>  <span class="o">=</span> <span class="nx">dbConnection</span><span class="p">.</span><span class="nx">query</span><span class="p">(</span><span class="nx">query</span><span class="p">);</span>

<span class="nx">response</span><span class="p">.</span><span class="nx">writeHead</span><span class="p">(</span><span class="mi">200</span><span class="p">,</span> <span class="p">{</span>
    <span class="dl">'</span><span class="s1">Content-Disposition</span><span class="dl">'</span> <span class="p">:</span> <span class="s2">`attachment; filename=</span><span class="p">${</span><span class="nx">req</span><span class="p">.</span><span class="nx">body</span><span class="p">.</span><span class="nx">type</span><span class="p">}</span><span class="s2">-</span><span class="p">${</span><span class="nx">req</span><span class="p">.</span><span class="nx">body</span><span class="p">.</span><span class="nx">startDate</span><span class="p">}</span><span class="s2">.json`</span><span class="p">,</span>
    <span class="dl">'</span><span class="s1">Content-Type</span><span class="dl">'</span><span class="p">:</span> <span class="dl">'</span><span class="s1">application/json</span><span class="dl">'</span><span class="p">,</span>
<span class="p">});</span>

<span class="cm">/*
 * Pipe the data to JSONStream to convert to a proper JSON string first.
 *
 * Once it has been formatted, _then_ pipe to the ExpressJS response object
 */</span>
<span class="nx">datastream</span><span class="p">.</span><span class="nx">pipe</span><span class="p">(</span><span class="nx">JSONStream</span><span class="p">.</span><span class="nx">stringify</span><span class="p">(</span><span class="kc">false</span><span class="p">)).</span><span class="nx">pipe</span><span class="p">(</span><span class="nx">response</span><span class="p">);</span>
<span class="nx">datastream</span><span class="p">.</span><span class="nx">on</span><span class="p">(</span><span class="dl">'</span><span class="s1">end</span><span class="dl">'</span><span class="p">,</span> <span class="p">()</span> <span class="o">=&gt;</span> <span class="p">{</span> <span class="nx">response</span><span class="p">.</span><span class="nx">end</span><span class="p">();</span> <span class="p">])</span>
</code></pre></div></div>
<p>(<em>You can view the actual code used <a href="https://github.com/jenkins-infra/uplink/blob/7a4b6377552d901b850c4c39570a67dd86b0a209/src/controllers/export.ts#L19-L32">here</a></em>)</p>

<hr />

<p>This approach is, as far as I can tell, “infinitely” scalable. So long as the
database can stream data to the Node application, the Node application will
continue to write data into the response for the end-user.</p>

<p>I was so worried that I was going to have to find some way to generate bulk
files on the server with some background job processing system, or something
else equally complex. I’m thrilled that the solution simply required connecting
one streamy thing to another streamy thing, which Node is quite well suited
for.</p>

<p>Neat!</p>]]></content><author><name>R. Tyler Croy</name></author><category term="javascript" /><category term="expressjs" /><category term="postgresql" /><summary type="html"><![CDATA[One of the little applications which I built earlier this year ended up more useful than I originally anticipated. Useful enough to have hit its first performance bottleneck! Performance problems I generally grumble “nice problem to have” which profiling and refactoring, but in this case I know what the performance problem was, but lacked the appropriate solution.]]></summary></entry></feed>