Category Archives: OSX

Resolving problems with PIP after Upgrading to OS X El Capitan

After I upgraded my Mac to El Capitan, I was having some problems installing new packages. I was getting access denied errors when some packages tried to upgrade (and hence remove) existing packages.

For those not in virtualenvs, I had package installed to the default Python site packages directory (/Library/Python/2.7/site-packages in my case). This was causing problems because El Capitan included a new feature called System Integrity Protection (also called rootless) that prevents you (even as root via sudo) from modifying files in a number of system directories, which seemed to be affecting this.

Below are the steps I took to resolve the issue, which is a general outline for how you can resolve this issue for yourself:

  1. Capture a list of all packages you have installed. Use pip freeze > some-file-to-keep-results
  2. Disable System Integrity Protection, which involves rebooting into recovery mode (hold Command+R), launch a terminal, use the command csrutil disable and reboot back into normal mode.
  3. Uninstall all packages from pip. Use pip freeze | xargs sudo pip uninstall -y or uninstall the package manually.
  4. Ensure that all the packages in the system site-packages directory are gone (/Library/Python/2.7/site-packages), remove any remaining packages manually.
  5. Re-enable System Integrity Protection using the same procedure as #2, with the csrutil enable command
  6. Once again rebooted in normal mode again, install a version of python that’s not the one that comes with OS X. brew install python will do that if you have the Homebrew package manager installed. This is better for development uses for Python anyway.
  7. Install pip manually by downloading the get-pip.py file from the link, and running it with python get-pip.py. You can also install pip via Homebrew, but there are some reasons to do it the manual way.
  8. Finally, and this might not be required in your case, pip still wasn’t available via the shell, so I needed to manually create a command to invoke it. I created a script pip in /usr/local/bin and made it invoke the pip package:
    #!/bin/bash
    python /usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pip/__main__.py $@
    

    Finally, I modified the script to executable with chmod uga+x pip.

  9. After pip is back in place and working, I re-installed the packages I previously had with pip install -r some-file-to-keep-results

    That’s it. Hopefully its at least that easy for you.

Setting up port forwarding on Mac OS X El Capitan for Google App Engine local development

I’m going to preface this post with the fact that I’m not an expert with pf the tool I’m using here to do this. I’ve just hacked together something that works from other tutorials I’ve found online.

By default the App Engine local development server runs on port 8080 locally, which is fine, but our app has some domain regex rules that are hard to test when the URL isn’t similar to how its deployed in production. To make things more realistic I edited my /etc/hosts file to give me “real” domains for my local dev environment. That solves part of the issue but the other part is getting things running on the right port. The first 1024 ports on *nix are restricted, so directly running the development app server on port 80 would be a pain, so I setup port forwarding.

The above linked tutorials got me going in the right direction, but didn’t quite work for me. Here are my steps.

First, create a new rules file in pf.anchors:

sudo vim /etc/pf.anchors/local-appengine

Paste the following in the file and save it (note that you just change 8080 if you are using a different port):

rdr pass on lo0 inet proto tcp from any to any port 80 -> 127.0.0.1 port 8080

Now edit /etc/pf.conf which should look like this when you start:

#
# Default PF configuration file.
#
# This file contains the main ruleset, which gets automatically loaded
# at startup.  PF will not be automatically enabled, however.  Instead,
# each component which utilizes PF is responsible for enabling and disabling
# PF via -E and -X as documented in pfctl(8).  That will ensure that PF
# is disabled only when the last enable reference is released.
#
# Care must be taken to ensure that the main ruleset does not get flushed,
# as the nested anchors rely on the anchor point defined here. In addition,
# to the anchors loaded by this file, some system services would dynamically 
# insert anchors into the main ruleset. These anchors will be added only when
# the system service is used and would removed on termination of the service.
#
# See pf.conf(5) for syntax.
#


#
# com.apple anchor point
#
scrub-anchor "com.apple/*"
nat-anchor "com.apple/*"
rdr-anchor "com.apple/*"
rdr-anchor "forwarding"
dummynet-anchor "com.apple/*"
anchor "com.apple/*"
anchor "forwarding"
load anchor "com.apple" from "/etc/pf.anchors/com.apple"
load anchor "forwarding" from "/etc/pf.anchors/local-appengine"

Update the non-comments part of the file to look like this:

scrub-anchor "com.apple/*"
nat-anchor "com.apple/*"
rdr-anchor "com.apple/*"
rdr-anchor "forwarding"
dummynet-anchor "com.apple/*"
anchor "com.apple/*"
anchor "forwarding"
load anchor "com.apple" from "/etc/pf.anchors/com.apple"
load anchor "forwarding" from "/etc/pf.anchors/local-appengine"

Note that you are just adding the following lines:

rdr-anchor "forwarding"
anchor "forwarding"
load anchor "forwarding" from "/etc/pf.anchors/local-appengine"

but the order of commands in the file matters, so it has to look roughly like the above.

Finally, enable port forwarding from bash with the following command:

sudo pfctl -ef /etc/pf.conf

You can disable it with the following command:

sudo pfctl -df /etc/pf.conf