Linux

On and off over the last few weeks I have been migrating this site from a Django app written by me to a static site generator called Hugo. If you don't recognise at least one of the technical words in the previous sentence, then you won't find anything interesting to read here. But if you find anything that is broken on here, it's probably something I broke during the migration, and you might want to let me know.

Migration wasn't hard

I had to move all my old content into the new site while not breaking any links. I used Markdown for all the content in the old Django app (in case I wanted to do something like this some day), so there was not much conversion required there. I had my own special snowflake markup for handling images and YouTube videos, which required a little work to convert.

The core of the migration was 161 lines of Python code (and then another 20 lines of throwaway Python to fix my breaking all the OpenGraph images). If I had entire days to spend on it and an unlimited attention span I might have finished the exporter in a day. I didn't have that, because I have a day job and a stupid car project and a mind that says well I'm going to have to Google this thought unrelated to any particular thing I am doing, but it was much more work thinking about than doing.

The slightly harder part was learning what I needed to learn about Hugo and doing a basic theme for it, but all of that might have been only a couple of long days - if, again, I had been focused on it.

After the migration, I rather wish I had used Hugo from the start. This site was a Django app, because Django is my day job and it is what I know; I figured I would more time fiddling with Hugo (or some other static site generator) than I would writing my own Django app. But I definitely spent far more time writing tests for my Django app than I did migrating this to Hugo!

Authoring is nicer

I like writing things in my favourite text editor. It seems a more obvious way to write, because that is what I do with most of the rest of my time at a computer. Actually, that was partly the way I was writing already; I was writing posts in a text editor, then copy-pasting into the Django admin interface when I was finished. And if I needed to do non-trivial edits I was copy-pasting it into my text editor, editing it, and then pasting it back.

This is not because Django's admin interface is bad. Although it wins no beauty contests, it's rather better than most web-based interfaces, because text boxes in the Django admin behave as text boxes have in your operating system for about 30 years, rather than the fallout of React meeting some UX designer's reinvention of "type text into a computer". But to those people for whom "favourite text editor" is a valid sequence of words, writing text into a web browser is not as natural as working on files in one's favourite text editor.

As with all static site generators, Hugo's master copy is just a bunch of text files. A bunch of text files can be version controlled with Git, which is also what I do with everything else I write at a computer. That, too, is what I do with anything else I write on a computer, and so it's a better way of working for me.

It's much less maintenance

This is the real reason for the upgrade; I wanted less work for myself in the long run. As said, Django is what I know. I like it a lot! But I have little interest in maintaining a Django app when I am not being paid for it.

It's easier to secure a static site than it is to secure a Django app. My web server is just serving some files from disk; it's nothing the unattended upgrades of my OS can't handle. Being a responsible citizen requires that I kept the various non-OS dependencies of a Django app up to date. That became a chore I didn't want, and if I'm honest I ended up getting Dependabot email alert blindness.

OS version updates were a source of dread. Serving static files does not change much, if at all, between major operating system versions. Serving a Django app does. That requires, invariably, a major Python version update and a PostgreSQL update as well.

Backups are easier

My old site was backed up multiple ways. Snapshots of the entire virtual machine (what Digital Ocean calls "droplets") were made daily, which cost money. That they would restore to a usable site was a matter of faith. I also periodically pulled the database and media files (such as images) to my local machine, which in turn is backed up elsewhere.

The entirety of the source code of this site is in a Git repository. That includes the theme, and all of the site's text and images. Simply by virtue of how this is deployed, that makes at least three up-to-date copies of the site in normal circumstances: one on my computer, one on GitHub, and one on the server. I'm getting those "for free", but then my local machine gets backed up, which makes four.

I also have the statically-linked Hugo binary checked in to the repo. This gives me some faith that, so long as I am on an x86-64 Linux machine, I will always be able to rebuild the entire site from source in a few seconds and re-deploy to a new server very quickly. This means I am more likely to have backups that can be restored.

Dear God it's fast

Really really fast.

It should load just about instantly on any device and any Internet connection that is likely to be in use. The technical people out there will say "of course it does", because of course it does; it's serving small files from a solid state drive.

The old site wasn't slow. I know how to make Django sites fast, and I did. But, without adding more piles of caching on top, it will never be as fast as directly serving files from a disk.

I decided to make that even faster, by inlining all of the CSS for the site on every page. It adds less than two kilobytes per page and saves a render-blocking HTTP request. Fast internet connections won't see this difference, but slow mobile ones will. I also added some magic tricks which make every link within this site to be prefetched, so once you actually click on it it'll load even faster.

"Faster" - of the "just serving file from disk kind" - also means I need fewer computing resources to serve it, which means...

It's slightly cheaper

...I could run this on the same host as a bunch of other static sites. Previously it was running on its own virtual server, because I figured a Django app would need it. Virtual servers are so cheap these days that they may as well give them away with Happy Meals, but that's still a few quid saved. I could reduce that to near-zero if I used S3, and actual zero if I used GitHub Pages or some other free host, but I like having a server I control.

Arguably the reason the Django app was maintenance in the first place was because I like having servers, rather than some Docker container running in some platform-as-a-service that I don't entirely understand. As it was, when it was time to upgrade the operating system on my server, it was also time to upgrade the database along with it, and also the Python version. The trick here is that when I upgraded the OS on my desktop I had to do the same thing on the server, and vice-versa. That could have been made easier by using a managed database, and a Dockerised Python. If I was doing this as a job I probably would. For something I am having fun with I do not like that loss of control.

I did a bunch of other things while I was there

I have some JavaScript on this site. It's used to handle the privacy-respecting, no-script-friendly YouTube embeds, such as this one. It was originally written in Vue, version 2. That version of Vue went out of support a while ago.

I didn't really want to maintain a JavaScript build system anymore. It felt like effort, and it turned out I could replace it with not many more lines of plain JavaScript than it was in Vue code; it's so small (like 500 bytes gzipped) that I didn't even bother minifying it. I can count on plain JS working in browsers for more-or-less ever. I can't count on a JS build system still working on my machine next year, so I do not feel much like maintaining one for a personal project. This fits the plan of making less long-term work for myself.

Not installing anything from NPM means I don't have any linters anymore. I don't feel good about this, but I feel far worse about the utterly insane dependency tree of ESLint and stylelint, and only slightly less bad about some of the (non-dependency) insanities of JSLint.

I did a minor visual revamp. The home page now lists every post, which means you can find any post by using Ctrl+F, and also go to all of the categories immediately. I also added support for high-contrast rendering, for those who have that preference set in their operating system. And the whole site now uses a monospace font, because I are programmer.

I could have done these things earlier, but I didn't. I didn't consider them meaningful enough differences to be worth doing by themselves. A rewrite of the site seemed like a good time to do that.

Hugo is really good

I like Hugo. Actually I tend to like anything that comes out of the Go programming community; it tends towards generating high-quality software.

I looked into Hugo in the past, and liked it a bit. As I recall its templating system was not as good back whenever that was as it is now. It, and I may be misremembering this, had nothing that would allow one to override blocks of a base template; instead, it only allowed including other templates, which is not slightly the same thing. That made it feel nerfed compared to Django or Jinja2 templates. Now that it supports extending from and overriding blocks of other templates I like it a lot.

Hugo is wonderfully documented and every maintainer should aspire to writing documentation this comprehensive.

Hugo builds to a standalone executable, which should continue to work on any of my Linux boxes into the indefinite future; for as long as I have a libc6 x86-64 Linux system, I'll be able to use the same Hugo binary I'm using today. I checked in to my repo with LFS in case it disappears.

Anyway

It all works! Please let me know if you find something broken.

TL;DR: Use imapsync with DavMail.

So my subscription for Office 365 (or whatever it's called these days) was due for renewal in January. I only ever used it for email on my domain. Given how increasingly hateful the Outlook web client has been in recent years (worst among them was when it randomly disabled the ability to send plain-text email for some time), and my general desire to move as much of my life away from software I have no control over, I decided to migrate my shit to Migadu. Partly because Migadu is kinda cheap, partly because of a good recommendation, but mostly because I really like their beautifully simple and functional website. I'm shallow!

Obviously, I didn't want to lose any of my email in this process. What should have happened is that I would enable IMAP on Office365, plug in some command line options to imapsync, and by the magic of open standards email would be moved from one place to another. Open standards are good! People who write open source software are awesome!

Of course, one half of that involved dealing with Microsoft. That and the rule of time estimates (increase the unit, halve the quantity) and the rule of everything is bullshit and never fucking works meant that I wasted more than a few hours trying to make that happen. Greetings from 3am!

The short version is if you use Office 365 in the same manner I did, which is that you were both the administrator of the domain and a user on the domain (because it was for personal email and you're the only person on the domain), nothing you do will make IMAP work for your account. Not if you navigate six levels of Enterprise-ness and a UI with decade-old branding to enable 2FA and then to enable "app passwords", not if you go disable security defaults and probably still nope if you run some PowerShell magic spells that I've seen around.

Give up trying to make IMAP work directly and use DavMail. I did!

DavMail is a proxy between Microsoft's proprietary protocols and protocols such as IMAP that the rest of the world uses. There's a package in the Ubuntu repositories called davmail, but because everything is bullshit and never fucking works the Ubuntu package is broken if you don't install the openjdk-11-jre package first (through no fault of DavMail's authors).

Anyway, once you've worked out that last bit from reading some random Debian bug report, you will want to go into DavMail's settings, under the "Main" tab, and change "Exchange Protocol" to "O365Interactive".

Then run a magic spell...

./imapsync \
  --host1 localhost \
  --port1 1143 \
  --user1 'you@dealingwithbullshit.com' \
  --debugimap1 \
  --password1 'hunter2' \
  --host2 'imap.migadu.com' \
  --user2 'you@dealingwithbullshit.com' \
  --password2 'hunter3'

...and DavMail will pop up a with a link to open in your browser, and that will in turn redirect to a URL that you copy-and-paste into DavMail. Some time later your mail will appear in Migadu or whatever else you decide to migrate to.

So, that was some bullshit. But it's done now! And I only have to do that once. And if I ever do have to do it again I won't waste hours of my time and hours of someone else's time trying to work out why IMAP isn't working, because, reminder, if your account is an admin account for an O365 domain it absolutely never will.

Anyway, imapsync and DavMail are good! They made me happy! I've only been dealing with mail migration bullshit for one long night, but the authors of both have been dealing with it for years. Here's a figurative toast to those fine people, and here's to many years of Migadu and me!

This is a quick recommendation for the XP-Pen G640 drawing tablet, for those fellow Linux users who like to do a bit of a scribble. It costs a penny shy of £40 as I write this, and it works flawlessly out of the box on Ubuntu 18.04 and 20.04 with no drivers or any other fiddling required. Just plug it in and go.

This is a great little tablet for doodling, whiteboarding in the working-at-home era, pwning noobs on Skribblio and generally messing about. For all I know someone might even buy one of these, download Super Street's car colouring pages^[1], fire up MyPaint and burn a few hours in the evening when lockdown gives them nothing better to do.

I shall give a necessary disclaimer that I occasionally doodle stuff just for myself (it helps my concentration in read-only tasks, and when I am in deep thought) and that I am not an artist at all. If I was I might want something like a few dedicated buttons for commonly-used operations like undo and erase. And I'm sure that in many other ways, someone using this for actual artwork would find this vastly inferior to things costing several times as much money. But for me, this cheap tablet works great!

[1] Alternative source as Super Street have, for GDPR reasons, blocked the whole of Europe from accessing their site. :(

This, known as bastet to its friends, has been around since at least 2005. I heard of it today. It's a variety of Tetris, but one that will always queue up the optimally worst next piece. ("So kinda like normal Tetris then", I thought.) And there's a devilish mode in which it will not tell you what piece is up next either. That one was too hardcore even for me.

It's available in the Ubuntu package repositories as bastet, and on Homebrew for the Mac, and apparently there is even a Windows version.

You may have heard of the Tetris effect; I wonder what damage this hilariously evil variant could do if you played it enough. The stuff of literal nightmares?

Links:

• Homepage
• GitHub
• Hell

I know, it is the Current Year, you don't need to worry about disk space. Until you do! Via a comment on this article I found this to purge a bunch of old log entries (on my personal machines I don't care about anything more than a day old, tweak according to your situation):

$ sudo journalctl --vacuum-time=1d
[literally six million lines of text]
Vacuuming done, freed 616.1M of archived journals from /var/log/journal/8ab90b50ecb94b5ba09cacf15a486a8e

Yay!