Chris’ talk below at North Bay Python was, as his always are, well-delivered and worth consideration.

The conclusion that he draws towards the end is similar to something I was noodling last year:

At some point somebody, somewhere, is going to have to actually understand how things work.

Chris makes the point, as he typically does, much more thoughtfully and with a stronger philosophical base.

Had some discussions with the delta-kernel-rs developers after they mistakenly added a ton of new files to tests/ blowing up test cycle times. Another community member shared this great overview about not using Cargo integration tests.

Catching up on Daniel’s thoughts on Data Quality and reconsidering the domain. The generation of slop has resulted in renewed discussions of “but how do we ensure correctness?” which is a great question to be trying to answer, but I am still rather disappointed with the state of the art for data quality tooling.

I recommend this blog post which has some good citations for negative AI behaviors affecting free and open source communities.

This is going to be a difficult problem to solve, more difficult than the email spam problem we have been unable to solve after 30 years of working on it.

This is also a very important problem, we are currently in an age where we have access to information that most people couldn’t even dream of 30 years ago. We also have disinformation that combines some of the worst aspects of authoritarian regimes throughout history combined with the worst aspects of cult brainwashing. If we lose access to the information but the disinformation remains (or get worse) then the result will be terrible.

I really enjoy Planet Debian as an aggregator of an international set of voices from the Debian community. I get exposed to so many different view points from around the free software ecosystem, which I really value. This past week I read this blog post by a debian maintainer which I was so flummoxed by I wrote out my thoughts on the topic here

Streaming tar over SSH is one of the more novel Unix tricks I don’t get to use much anymore. Drew Devault shared some helpful tips for using it without needing to use incantations of rsync(1).

Compressed Apache Arrow tables over HTTP

When discussing transport protocols for sending data between services at work recently, a colleague asked “why can’t we just yeet Arrow over HTTP?” It turns out, you absolutely can and Arrow IPC streams even have a registered MIME type:

Content-Type: application/vnd.apache.arrow.stream

Understanding Parquet format for beginners

A great introduction to the Apache Parquet format and why it makes so many things better with large data storage systems like Delta Lake. I have written on this topic before and encourage you to take another read through this blog post by some maintainers of the parquet crate.

Every layer of review makes you 10x slower

Every layer of approval makes a process 10x slower [..]

Just to be clear, we’re counting “wall clock time” here rather than effort. Almost all the extra time is spent sitting and waiting.

• Code a simple bug fix: 30 minutes
• Get it code reviewed by the peer next to you: 300 minutes → 5 hours → half a day
• Get a design doc approved by your architects team first: 50 hours → about a week
• Get it on some other team’s calendar to do all that (for example, if a customer requests a feature): 500 hours → 12 weeks → one fiscal quarter

This inspired these thoughts which I shared with the delta-rs community:

“what if we didn’t require code review for merging into main”

I’m exploring the thought more about what we might need to make that happen. “Why would you do such a thing, code review is so valuable!”  I do find code reviews valuable but we do seem to lose a lot of flow time due to timezones, differing work schedules, and a number of other things. For something without a lot of changes, especially bug fixes that come with tests I would be much more comfortable with maintainers merging once CI goes green.

Some pieces of the puzzle that I think would be needed:

• Soft caps on pull requests. I saw this mentioned somewhere else, but implementing a soft cap of <500 lines per pull request can help people avoid massive unreviewable changes that are simpler to integrate.
• Incorporating some of the benchmarking work into CI that has already been explored. If performance of key operations is not affected and the build is green, go for it.
• Stronger semantic version checks: if our APIs have not changed and all tests pass, I’m generally comfortable with landing stuff by maintainers.
• Implementing Apache Software Foundation style release candidates and voting: this is where we would put a mandatory bottleneck, rather than some jokey slack emojis like I tend to do, implementing a true release candidate process that requires review and vote before we push something to users.

All of this is to say that reviews can still be requested, but I would love to see us land more improvements faster and I think we have a bunch of different schedules that can make pushing each change through a review queue a lot slower than necessary.

Conditional Impls in Rust

It’s possible in Rust to conditionally implement methods and traits based on the traits implemented by a type’s own type parameters. While this is used extensively in Rust’s standard library, it’s not necessarily obvious that this is possible.

I have been vaguely aware of this functionality but haven’t really taken the time to consider it, so I really appreciated this post walking through the conditional impl functionality in Rust.

Microblogging is basically RSS with a slight change in conventions to support more Twitter-style postings, with a somehow sillier name. When I was exploring the concept, I came across numerous posts of folks trying out microblogging only to find that their feeds had gone very stale. They couldn’t bridge the gap between their existing community in sites like Twitter with their microblog set up. This excludes some users I have seen on micro.blog, but I’m not about to pay $5/month for something that primitive.

My set up is built around me existing blog (you’re reading it!) with some tweaks to make it easier to author microblog entries. I then integrated twitter-together to ensure that my microblog posts are duplicated automatically into my Twitter account. Not everything I post to Twitter goes through my Microblog however, replies and retweets don’t make much sense to me outside of the context of Twitter. But practically anything I would typically share via my desktop I can now broadcast via both channels!

Subscribe to my microblog, or don’t, it’s a free internet after all. :)

After trying to find some way to make newsboat read a different protocol, racking my brains thinking of different ways to set up a stub HTTP proxy, I finally succumbed and read the manpage.

As my luck would have it, the urls file that newsboat stores its URLs supports a special exec syntax for shelling out to run a command to fetch the feed, for example:

~/.newsboat/urls

"exec:ssh shellhost 'cat /srv/www/rss.xml'"
"exec /usr/bin/torify curl ftp://someftp/rss.xml"
"exec:/usr/bin/torify curl gopher://example.com/0/news.atom.xml"

(Side note: do you have any idea how many protocols curl supports? Lots! On my machine: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp)

The exec syntax is certainly a novel feature. As I have been pondering it more, I have been thinking about is using it to run arbitrary shell scripts which would generate reports for review. Some ideas that have come to mind:

• Reading the root’s mbox on my local and remote machines to get better visibility into the status of cron jobs.
• Executing some aws-cli and az scripts to grab generate some daily cost reports.
• Retrieving error logs from remote machines to tabulate a daily error report.

There are other possibilities that come to mind, but it all basically boils down to generating information dashboards which will help me keep tabs on more and more things, all from within my feed reader.

I have only just started to experiment with this idea, but I’m looking forward to poking around with this more.