TinyPilot

Management

• Prepared processes for the Voyager 2 launch
• Scrambled to fix a part shortage
  • We almost had to stop sales because we ran out of tiny ribbon cables!
  • Fortunately, our vendor expedited our order and got it to us within days.
• Continued onboarding with Gusto and started offboarding with Justworks
• 1:1 with developer
• 1:1 with local staff member
• Sync meeting with distributor
• Sync meeting with 3D printing vendor
• Sync meeting with EE vendor

Software development

• Reviewed a PR that adds custom logging messages for sensitive data
  • We routinely ask users to upload their logs when asking for support in the help forum, so we want to make it easy for them to remove sensitive data from their logs like usernames or keystrokes.
• Reviewed a Github Action to auto-merge changes from TinyPilot Community -> TinyPilot Pro
  • This used to be a tedious, manual job that fell to me
  • Now, we have it set up
• Worked on design document for overhaul of TinyPilot’s update/install system
• Fixed pylint’s rules so that long lines are okay for URL strings
  • Our style guide says they’re okay, and pylint makes exceptions for URLs, but it’s default exception is very narrow.
• Got rid of pylint rules that were redundant to our other linters
• Reformatted pylintrc for readability

Customer support

• Updated instructions for enabling read-only filesystem on the TinyPilot

Product research

• Met with EE vendors for Voyager 3 design

Sales

• Met with potential enterprise customer

Misc

• Fixed a break in the office printer
  • It turned out that the Pi we were using as a print server crashed and needed to be rebooted

mtlynch.io

• Wrote most of November retrospective
• Bought parts for a homelab NAS build
  • Blog post to follow
• Removed Google Analytics
• Removed the proxy funny business I was using to prevent users from ad-blocking Plausible
  • I did it mostly out of curiosity, but my new policy is that if users want to block, they can block.

What Got Done

• Migrated from AppEngine+Firestore to fly.io+SQLite+Litestream
  • This caused a 2-20x speedup in most What Got Done requests
• Switched to an auto-update mechanism for retrieving pageviews
  • What Got Done displays pageviews at the bottom of each post, and it does that by querying data from Google Analytics
  • Google Analytics is slow, so we don’t do it in real time.
  • On AppEngine, I set up a cron job to refresh the numbers every five minutes, but fly.io doesn’t have any cron-like mechanism.
  • I realized the better solution is to just cache the numbers for X minutes and then refresh the values when the cache expires.
  • This is a lot more elegant and results in far fewer requests and database writes
• Optimized database writes for pageview statistics
  • “Optimized” is maybe embellishing it.
  • On Firestore, updating pageview stats required updating 600+ values in Firestore, which is too large for their batched write operation.
  • On SQLite, writing to the database 600 times in quick succession was causing the database to lock.
  • The solution was to just write all 600 values in a single query instead of 600 separate UPDATE queries.
• Discovered a bug in gorilla/handlers
  • And made a fork to work around it.
  • The gorilla/handlers framework adds a ProxyHeaders handler that will update the properties of a proxied request to make it look like a regular request.
  • I needed this handler for What Got Done logging because otherwise, the logs show all the requests as coming from fly.io’s edge proxy server
  • It turns out that ProxyHeaders populates the URL with a weird value, and that causes gorilla’s other library to reject it as a potential CSRF attack.
  • Integrated my forked library into What Got Done
• Added server-side checks for 404ed requests
  • Previously, the server just returned HTTP 200 for everything and let the frontend show a “not found” message on missing routes, but it’s better to return proper 404 for definitely non-existent routes.
• Added support for exporting all data (admin only)
  • And then removed it
  • I only needed it as a one-time operation to export all of What Got Done’s data from Firestore
  • It turned out that exhausted AppEngine memory, so I briefly had to [scale](https://github.com/mtlynch/whatgotdone/pull/697] up AppEngine resources
• Modified my test data injector script to support JSON as well as YAML
  • The export functionality exported to JSON, so I used my test data injector script to populate a new SQLite database that matched the data I exported from Firestore
• Converted some timestamp properties to proper time.Time values instead of RFC-3399 strings.
  • I couldn’t do this easily when I was on Firestore because I had less control over serialization and deserialization to the datastore.
• Removed cleanup logic for leaked channels
  • It turns out that you don’t really need to clean up channels in Go
• Fixed a bug in media uploading that I must have introduced a few weeks ago
• Set up log exporting from fly.io to Datadog
• Increased Litestream’s snapshot retention to 30 days
• Added logic to print build settings during the build

LogPaste

• Fixed a bug that I introduced in my migration to the Docker alpine image
• Fixed the --skip-build flag in my e2e test script

WanderJest

• Added a null check in the frontend to prevent the UI from getting into a state where it can’t submit performance details and can’t fix the error.

Misc

• Experimented with Gitlab as a CI provider
  • It was disappointing. Builds are very slow and I found the syntax unintuitive.
• Made a simple feature submission to Litestream to sort snapshot output