Weekly Update

General

• Most of this week was spent researching options for upgrading our various App Engine projects to Python 3 before Python 2 is deprecated at the end of the year. So this is more of a series of preliminary notes on “What needs to be done” rather than “What got done”:
  • Google had previously hinted they were working on a version of ndb (App Engine’s datastore library/ORM) for the Python 3, but years had gone by without any official declaration. I was just resigning myself to migrating off of ndb this week when I discovered ndb for Python 3 is now in beta
    • Our dependency on ndb was our top blocker for migrating to Python 3 (we depend heavily on ndb’s built in features like data-models, transactional decorators, hooks for post-put-processing, etc.)
  • After migrating, we can use Python libraries written in C for performance-critical functionality like password hashing, allowing us to deprecate our separate Golang fast-utils project.
  • Migration would require a full-scale refactoring, with lots of new code to replace services that are currently tightly coupled with AE Python 2.
    • ndb2 -> ndb3:
      • We can keep our existing models, with some minor changes.
      • We’ll need to replace ndb 2’s integrated memcache (we depend in particular on automatic memcaching of fetched entities). Earlier migration docs mentioned that work was underway to add access to an App Engine Python 3 memcache service, but newer docs now suggest configuring a Redis instance.
    • Task queues: would need to migrate to Google’s non-app engine queuing service.
    • Cron: same as above.
    • Memcache: Python 3 standard environment doesn’t include a built in memcache service. Replace with Redis.
    • webapp2 framework dependencies would need to be migrated, probably to Flask.
      • For UserKit we have pretty deep dependecies on a class based framework (we inherit from, extend, and use mixins with, webapp2’s RequestHandler class).
  • Dev environment:
    • Our tests currently use AE2’s TestBed library, which is tightly coupled with the old datastore emulator.
      • There is a new experimental datastore emulator, but it may not be possible to migrate our old dev data, in which case we would have to setup the development environment again.
      • We can use “normal” test libraries instead, but it’s not clear yet to me how to connect the dots between this and the new emulator for our existing unit tests.
    • Without TestBed, we may need to handle datastore initialization and reset at a lower level between tests.
    • Keeping the dev environment fully encapsulated might no longer be possible, or at least would reduce the fidelity of our dev and test setup vs production.
      • For example, replacing memcache with Google’s recommended Redis configuration would add an external project/service dependency. This may be impossible to emulate locally.
  • UserKit dashboard depends on the UserKit Python SDK, so we need to officially support Python 3 (something we should do anyway).

Summary

• The App Engine Python 2 standard environment comes prepackaged with a set of integrated services and simple, easy to use libraries. The Python 3 standard environment is more akin to a docker container with built-in automatic scaling but requires developers to use Google’s standard services.
• More research required on all fronts.

UPDATE FOR PREVIOUS WEEK
(I accidentally posted last weeks update for this week, but didn’t want to pollute the stream by re-posting it separately)

General

• Support emails.
• Spent most of the week dealing with accounting and tax issues.
  • Most of this felt like treading water, but at least we now also have the paperwork in place if we decide to contribute to a multi-user SEP-IRA account.
  • I’m also now armed with a better understanding of the rules around making contributions.

Feedbacker

• Added the new SurveyResponse datastore model.
  • Accomplished by renaming an existing model (AnswerCounter -> SurveyResponse) which was already used to keep track of how many survey questions a user had answered, and whether that user received points already for their answers. Also added a property to the Answer model create a relationship between each Answer and it’s parent SurveyResponse entity.
  • Migrated existing dev and production data.

Misc

• Finally overcame the long-tail of car issues!
  • Although it turns out most of these problems were actually just one larger one: mice. They had eaten through much of the car’s wiring, destroying every airbag, and damaging some of the other electronics like the stereo and antenna.