Monthly Archives: October 2015

RESTful App Engine

Continuing with my trend of posting presentations I gave a while ago, last year at Twin Cities DevFest I gave a presentation about building RESTful JSON services on Google App Engine.

The presentation is designed to both explain the ideas of REST, including the following topics:

  • How REST differs from RPC-style APIs
  • The pros and cons of JSON versus XML
  • What HTTP verbs are appropriate for which operations, including PATCH witch is seen less often in the wild
  • What HTTP status codes should be used for which scenarios
  • Tools to use when developing RESTful APIs
  • Python Code examples implementing the same API in straight Webapp2, Google Cloud Endpoints, and Webapp2 + Pytracts

In the talk I introduce my JSON serialization library Pytracts (which was called ProtoPy at the time of the presentation).

Slides here and video of the presentation here. I’m planning on recording a screencast of the presentation so the audio quality is a bit better.

Data Migration on the App Engine Datastore

I gave a presentation a couple years ago at the Twin Cities DevFest conference and I’ve been meaning to post the slides.

The gist of the talk is that with web frameworks like Rails and Django, data migration is a feature of the data model tools. With App Engine Datastore (now Cloud Datastore) you have to do the work yourself. In the talk I give Python examples of how to update the NDB models, how to use deferred tasks and mapper/mapreduce jobs to update existing entities.

The slides are here:

http://documents.morlok.net/data-migration-on-the-appengine-datastore

I’m hoping to record myself giving the presentation soon.

Setting up port forwarding on Mac OS X El Capitan for Google App Engine local development

I’m going to preface this post with the fact that I’m not an expert with pf the tool I’m using here to do this. I’ve just hacked together something that works from other tutorials I’ve found online.

By default the App Engine local development server runs on port 8080 locally, which is fine, but our app has some domain regex rules that are hard to test when the URL isn’t similar to how its deployed in production. To make things more realistic I edited my /etc/hosts file to give me “real” domains for my local dev environment. That solves part of the issue but the other part is getting things running on the right port. The first 1024 ports on *nix are restricted, so directly running the development app server on port 80 would be a pain, so I setup port forwarding.

The above linked tutorials got me going in the right direction, but didn’t quite work for me. Here are my steps.

First, create a new rules file in pf.anchors:

sudo vim /etc/pf.anchors/local-appengine

Paste the following in the file and save it (note that you just change 8080 if you are using a different port):

rdr pass on lo0 inet proto tcp from any to any port 80 -> 127.0.0.1 port 8080

Now edit /etc/pf.conf which should look like this when you start:

#
# Default PF configuration file.
#
# This file contains the main ruleset, which gets automatically loaded
# at startup.  PF will not be automatically enabled, however.  Instead,
# each component which utilizes PF is responsible for enabling and disabling
# PF via -E and -X as documented in pfctl(8).  That will ensure that PF
# is disabled only when the last enable reference is released.
#
# Care must be taken to ensure that the main ruleset does not get flushed,
# as the nested anchors rely on the anchor point defined here. In addition,
# to the anchors loaded by this file, some system services would dynamically 
# insert anchors into the main ruleset. These anchors will be added only when
# the system service is used and would removed on termination of the service.
#
# See pf.conf(5) for syntax.
#


#
# com.apple anchor point
#
scrub-anchor "com.apple/*"
nat-anchor "com.apple/*"
rdr-anchor "com.apple/*"
rdr-anchor "forwarding"
dummynet-anchor "com.apple/*"
anchor "com.apple/*"
anchor "forwarding"
load anchor "com.apple" from "/etc/pf.anchors/com.apple"
load anchor "forwarding" from "/etc/pf.anchors/local-appengine"

Update the non-comments part of the file to look like this:

scrub-anchor "com.apple/*"
nat-anchor "com.apple/*"
rdr-anchor "com.apple/*"
rdr-anchor "forwarding"
dummynet-anchor "com.apple/*"
anchor "com.apple/*"
anchor "forwarding"
load anchor "com.apple" from "/etc/pf.anchors/com.apple"
load anchor "forwarding" from "/etc/pf.anchors/local-appengine"

Note that you are just adding the following lines:

rdr-anchor "forwarding"
anchor "forwarding"
load anchor "forwarding" from "/etc/pf.anchors/local-appengine"

but the order of commands in the file matters, so it has to look roughly like the above.

Finally, enable port forwarding from bash with the following command:

sudo pfctl -ef /etc/pf.conf

You can disable it with the following command:

sudo pfctl -df /etc/pf.conf