's update for the week ending on Friday, Feb 7, 2020
Inkpad
Responded to reviews and support emails (pretty heavy week because of the ongoing release, still catching up)
Discovered a bug in our Stripe upgrade flow for Android WebViews, but gave up when I realized it only affects really old versions of Chrome (e.g. version 33)
Searched for the cause of a slow memory leak, but finally gave up and switched our App Engine instance type back to F4 for now.
Instances still get shut down periodically, but F4’s increased RAM will make it a less common occurrence. Less frequent shutdown means less double-billing for the overlapping 15 minute period when a full RAM instance is shutdown and a fresh one is started up.
Hopefully the migration to Python 3 will reveal/remove whichever library is the cause
Disabled the more expensive dedicated memcache service (there was no meaningful difference in hit-rate compared to the cheaper “best-effort” memcache)
Lots of app and server debugging, spelunking through logs.
Removed App Engine’s appstats builtin. This was part of the memory leak search, but it needed to go anyway as part of the move to Python 3
Misc
Reviewed our GCP billing logs - discovered an old virtual machine that’s been eating a few dollars/month and was intended for temporary use