2013
08.13

Selenium WebDriver makes it easy to write automation tests for your web applications, but with lots of apps using asynchronous means of loading data, it can be hard to determine when a certain call is finished.

The code below shows how to wait until all ajax calls are complete by calling out to jQuery.active via webdriver’s execute_script method.

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.common.exceptions import WebDriverException
 
def ajax_complete(driver):
    try:
        return 0 == driver.execute_script("return jQuery.active")
    except WebDriverException:
        pass
 
def my_automation_test():
    ff_driver = webdriver.Firefox()
 
    ff_driver.get("http://domain.tld")
 
    #wait for ajax items to load
    WebDriverWait(ff_driver, 10).until(
             ajax_complete,  "Timeout waiting for page to load")
 
    assert "ajax loaded string" in ff_driver.page_source
2013
01.13

Like many others, the news of Aaron Scwartz’s suicide, struck me to the core. Although I didn’t know him personally, I figured I owe it to him to as least put down my thoughts, like he did so many times.

As you traverse the many projects that Aaron was a part of, you quickly see a pattern

web.py is a web framework for Python that is as simple as it is powerful. web.py is in the public domain; you can use it for whatever purpose with absolutely no restrictions.”

That was his theme: “Its public, use it for whatever you like, no restrictions.” Some might argue a little simple and idealistic, but you know where he stood.

After reading the statement from aaron’s family, combined with Lawrence Lessig’s piece and Alex Stamos, I wanted to see what Carmen Ortiz and the Massachusetts Federal District Attorneys Office had been up to that they have the time to prosecute someone like Aaron. As the DA’s office site says, “U.S. Attorney Ortiz’s top priorities include terrorism and national security, civil rights, and violent and white collar crime reduction – encompassing public corruption, financial and healthcare fraud.”

Foolishly I then assumed we would have publicly available court records that I could see to show that our Justice Department was hard at work busting the public corruption that their site talks about, but I was greeted with this message:

To locate information on a case, please log on to PACER, the U.S. Courts electronic records database. If you do not have a log on ID to PACER, you may register for one. Please note that there is a charge to access the system.

I started to think that the charges brought against Aaron were less about justice, and more about retaliation. Didn’t Aaron try to liberate the PACER information before, for us?

In regard to Aaron’s current charges, Ortiz had previously been quoted as saying, “stealing is stealing, whether you use a computer command or a crowbar, and whether you take documents, data or dollars”. At the very least, it shows she has a very archaic way of viewing digital information. It also does not distinguish between someone stealing credit cards to enrich themselves and someone who is trying to distribute information data. Neither JSTOR or MIT were victimized to the tune of 35 years in jail and $1million dollar fine (which is what Aaron was facing). In fact, JSTOR is now starting to liberate some of the same information that Aaron allegedly stole.

What makes more sense is that Carmen Ortiz, Stephen P. Heymann and the US D.A.’s office were out send a message. They dont want government to be open and transparent, and if you try to free public records, or help to keep the internet more open, they are gonna throw the book at you.

No one will know Aaron’s final thoughts. I don’t see how 35 years didn’t play some sort of a role. It was also no secret that Aaron dealt with depression. I hope he is no longer tortured.

Carmen Ortiz and Stephen P. Heymann dont represent me, and I think they should be haunted by his death.

It is a testament to Aaron to see the outpouring of grief the past few days.

Its pretty obvious we all lost someone special.

Edit 1-15-2013:
Marcia Hofmann breaks down the way in which the government was using the CFAA to charge Aaron and how “hacking” laws should be changed.

2012
11.14

Why Flask-Classy?

This is a guest post by the awesome @apiguy, aka Freedom Dumlao. You can read more by him at his site Dash Rocket.

If you’ve spent any time thinking about developing web applications in Python, no doubt you’ve considered, and maybe even tried Flask. It’s an amazing little framework with massive appeal. I love it’s minimal and a la carte approach to features, and I love that it makes getting started easier than any other framework I’ve worked with in the past.

One place where Flask really distinquishes itself from the rest of the pack is it’s approach to supporting larger applications. The framework includes two methods for organizing your views beyond the global scope of the main module, either Blueprints, a novel aproach using a mock application object in seperate Python modules, or using special class based views from the flask.view module. Both are useful in many cases, but for me neither really quite fit.

Blueprints work well for grouping similar views together, such as all the views for your admin interface. It’s a great help when trying to organize a large project, but suffers from a lack of a managed context. I wanted to be able to specify data and behaviors that were important and applicable only to a certain set of views and outside of putting stuff in the request object my options were limited.

The flask.view View classes are pretty interesting and provide some class based context, but they don’t help much with grouping similar views together. I found myself writing much more code than seemed reasonable when using them and started to lose the feeling of working with a simple framework that got out of my way and let me focus on what my application was trying to do.

Enter Flask-Classy. My idea seemed simple. Let me group my application views in a way that was both logical and provided a context that was extensible. I examined my own projects and those of my colleagues to find patterns that could easily be simplified by use of a few basic, common conventions.

The most common pattern I found was creating RESTful endpoints for resources in an application. There were a few solutions out there but the most common pattern I found was one I had used myself on more than a few occasions:

@route(‘/entry/<id>’, methods=[‘GET’, ‘PUT’])
def entry(id):
    if request.method == “GET”:
        …
    elif request.method == “PUT”:
        …

Or alternatively:

@route(‘/entries/<id>’, methods=[‘GET’])
def get_entry(id):
    …

@route(‘/entries/<id>’, methods=[‘PUT’])
def put_entry(id):
    …

Neither of these methods is easy to manage once your application has more than a few entities. So task #1 for Flask-Classy was to make that better. I achieved this by making Flask-Classy aware of common RESTful CRUD operations, so you could easily just define some methods of a class and get the common functionality for free:

class EntriesView(FlaskView):
    def get(id):
        …
    def put(id):
        …

This way all of your related views could be grouped together in classes that are located anywhere in your application. You get complete control over the structure of your code and you get to write less of it!

After the basic CRUD stuff was in there I went ahead and added support for adding your own methods and getting automatic routing for them as well. I’ve added a custom routing decorator and subdomain support to make sure that experienced Flask developers don’t have to leave any of their favorite features behind, and I’m contiuing to add new features with the help of the amazing Flask community all the time.

So for your next web app, give Flask and Flask-Classy a try. I’d love your feedback, feature ideas, and criticisms. (OK if we’re being honest I probably don’t want your criticism, but I’ll take it anyway.)

2012
11.12

A couple years ago I made the slow process of moving all DNS configurations to one central location and I chose Zerigo as my dns management provider. Before that, I had dns entries all over the place like Slicehost, Rackspace, Route 53, and places including GoDaddy, where some client may have bought a name and the dns was left there out of convenience. Its one thing if its someone else’s responsibility to make dns changes, but when you are in charge, its a nightmare to have such a large number of places and interfaces you have to work with. Some of those interfaces dont work that well either.

I don’t make changes all that often, but when I do, I now have just one place to look and a simple interface to use. Obviously it costs more money to have managed dns service, but I think its worth the trade off. Zerigo, let me import most of my dns config, makes it easy to roll out a new domain with templates, and has an api. Just tonight, I found out that you can easily re-route a naked domain to a www address. To accomplish this before, I used to point a lot of my naked domains at a generic linux box that would redirect them to the corresponding www address because I didn’t think it was possible to do via dns (maybe I’m just a meatball).

Overall I have had no issues that I am aware of with Zerigo, and I’m a proud paying customer. Force yourself to start moving your dns to one spot. It will make your life easier.

2012
11.11

After seeing Eric Maundu’s video about his aquaponics setup in Oakland, I was pretty inspired to finally try and put my arduino to good use. In the past, I had hit walls using the arduino because I failed get it setup to a wifi network, and therefore never got to the point of programming it to do anything useful for me.

This summer I grabbed a raspberryPi, which allows me to write all my code in python and has everything that comes with linux, like painless networking. Although the pi is the coolest thing since sliced bread, it has no analog inputs on its GPIO, so there is no easy way to measure inputs like temperature, only digitally (off/on). You can, however, easy communicate between the pi and the arduino serially over usb to use the arduino’s analog inputs.

Ultimately, I would like my aquaponics setup to be out in the back garage, which means a wifi connection to my raspberryPi to post data (I dont want the pi to be my SOR). Again, I struggled to get the wifi adapter setup. Below is the wpa configuration that ultimately worked to get my Edimax WiFi Adapter connected to my Cisco wifi router.

Once you have the drivers installed for the wifi adapter. Add these lines to your /etc/network/interfaces file:

auto wlan0
iface wlan0 inet dhcp
pre-up wpa_supplicant -Dwext -i wlan0 -c /etc/wpa_supplicant/wpa_supplicant.conf -B

now you must go create or edit your wpa_supplicant.conf file, the configuration that worked for me looked like this:

ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
network={
ssid="my ssid"
scan_ssid=1
key_mgmt=WPA-PSK
psk="my secret passcode"
}

replacing my ssid and my secret passcode with your routers settings. Make sure the router is broadcasting it’s ssid.

Reboot your raspberryPi and you should see the edimax light up blue and grab an ip address from your router. This now allows me to ditch the keyboard and mouse, and put my raspberryPi out in the garage to control my garden. When I need to update anything, I can ssh into the pi.

Using the Python Arduino Prototyping library, I’m able to read the analog inputs of the arduino on the pi by connecting to it via usb. I have an arduino Uno, so the arduino shows up on the pi as '/dev/ttyACM0'. If you are unsure which device is the usb, run ls /dev with the aruduino plugged in, and then again with it disconnected. Go find the device that disappeared; it probably begins with ‘tty’.

NOTE: if you are ssh’ing into the pi and using a non-root user (e.g. not the default “pi” user), make sure you add your user to the dialout group, before you will be able to access the arduino over usb. You can add yourself to the group by issueing the following command (replace “pi” with your username):
sudo usermod -a -G dialout pi
Log in again to make group changes take effect.

Now I am able to run this python code on the pi and control the arduino:
mattotodd@raspberrypi ~/garden-project $ python
Python 2.7.3rc2 (default, May 6 2012, 20:02:25)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from arduino import Arduino
>>> a = Arduino('/dev/ttyACM0')
>>> a.setHigh(7)
True

Its a very small step to make an LED light up, but I feel like I’ve moved the first brick. I hope to log more of the progress here as the project moves along.

Here’s a little demo

2012
08.09

TLDR; its pretty easy to use the same chrome dev tools from you desktop to debug android chrome

I used firefox for longer than I should have purely because firebug was so good. Now that I’ve swithched over to Chrome, I will ‘prolly stay here in the cozy familiar dev tools I’m used to than switch to something new. They make it too easy for me.

Tonight, I was trying to see how well Bootstrap and Backbone.js played together for a mobile web app. It was the first time I had to debug Chrome on Android and saw it had a usb debugging option in the settings. Low and behold, if you already have the android SDK installed, you are just a few short steps away from using your desktop Chrome dev tools to debug your android chrome web pages!

  • Enabled USB debugging in Android Chrome settings
  • Issue cmd on SDK enabled machine: adb forward tcp:9222 localabstract:chrome_devtools_remote
  • Open desktop Chrome and navigate to localhost:9222
2012
04.13

I’ve spent hours of my life trying to install PIL on different macs. Each time I fail miserably and relearn the steps to correct the issues. Once again I hit the same compilation error, this time while trying to install PyObj.

macbook$ easy_install pyobjc
....
things are going great
yada yada yada
...
command 'gcc' failed with exit status 1

This is as much a reminder for me as it is for others.

If you are having issues running installs via easy_install or running the command “python setup.py install” and your macintosh keeps barking about gcc, it means the installation process having trouble building one of the dependencies of your package. First make sure you have the latest and greatest version of Xcode. If you are still having an issue, which I always am, then make sure you are letting gcc know what architecture you are trying to build for. You can do this by prefixing your install command like so:

For Intel Macs:
macbook$ sudo env ARCHFLAGS="-arch x86_64" easy_install python-imaging
macbook$ sudo env ARCHFLAGS="-arch x86_64" python setup.py install

For PPC:
macbook$ sudo env ARCHFLAGS="-arch i386" easy_install python-imaging
macbook$ sudo env ARCHFLAGS="-arch i386" python setup.py install

2012
03.12

If you spend any amount of time in the python shell IPython could probably make your experience better. Syntax highlighting, gaining access to shell commands, and tab completion are just a few of the small things it can do for you as a developer. On the more complex end, the things it can help mathematicians and scientists do is beyond my comprehension.

To be honest, IPython is not my default python shell just because of habit. After watching Fernando Pérez’s presentation at PyCon last week I was fascinated by what they are doing with the IPython Notebook. Its fiddle for IPython, with the ability to rerun snippets of code, and display inline graphical data in a browser. You can save you sessions, and easily run any portion of your code.

I can wait to play with it more and you should check out the presentation. The demo starts here

2012
02.27

Flask’s simplicity is great.

from flask import Flask
app = Flask(__name__)
 
@app.route("/")
def hello():
    return "Hello World!"
 
if __name__ == "__main__":
    app.run()

But once you’re up and running you’ll find you need to refine your servers settings to your liking. I had to make some changes to get PyCharm’s debugger hitting breakpoints.

The python argparse module makes it easy to add a little flavor to your command line params.

#my_app.py
from flask import Flask
app = Flask(__name__)
 
@app.route("/")
def hello():
    return "Hello World!"
 
if __name__ == '__main__':
    import argparse
    parser = argparse.ArgumentParser(description='Development Server Help')
    parser.add_argument("-d", "--debug", action="store_true", dest="debug_mode",
                  help="run in debug mode (for use with PyCharm)", default=False)
    parser.add_argument("-p", "--port", dest="port",
                  help="port of server (default:%(default)s)", type=int, default=5000)
 
    cmd_args = parser.parse_args()
    app_options = {"port": cmd_args.port }
 
    if cmd_args.debug_mode:
        app_options["debug"] = True
        app_options["use_debugger"] = False
        app_options["use_reloader"] = False
 
    app.run(**app_options)

This allows you to accept command line parameters. A simple help query against your app will show you a friendly message:

~username$ python my_app.py -h
usage: my_app.py [-h] [-d] [-p PORT]

Development Server Help

optional arguments:
-h, --help show this help message and exit
-d, --debug run in debug mode (for use with PyCharm)
-p PORT, --port PORT port of server (default:5000)

This will allow you to pass “-d” as an additional argument in your PyCharm Debug Configuration and it will play nicely with the pycharm debugger. I also added a port param so that you can run it on a different port; django burned 127.0.0.1:8000 into my chrome history.

Keep in mind you need to be running PyCharm 2.0+ for the debugger to work correctly with flask. You also lose the automatic reloading of the server, but you need to restart the debugger if you make changes to your code anyways.

2011
11.23

I have been using mongoengine for the past year and life’s been a breeze. This week, however, I hit the first time that I needed an advanced query that couldn’t be expressed with the basic mongoengine api. It was time to go raw dog, and there’s no turning back. Mongoengine probably suffices in most situations, but sometimes you need a little more control over your query.

Lets say I have the following mongoengine Document:

from mongoengine import *
import User #another mongongine Document
import Team #another mongongine Document
 
class Item(Document):
    team = ReferenceField(Team)
    creator = ReferenceField(User)
    public = BooleanField(default=False)

Users of our system belong to a Team, and they create Items. A User should only have access to Items that he/she created or that are on the same Team and public.

So lets query for a list of Item’s a User has access to!

I had originally tried to do this with mongoengine’s advanced queries, but I had horrible results. After a full day of messing around with it I can not explain why the results were coming back as they were.

It was time to understand a little more about mongoDB advanced queries, and I feel like a better man for it. So I wanted to find all the Items that are part of a particular Team, that are either public or created by the logged in user:

....
user = logged_in_user
team = user.team
 
items_query = Item.objects(__raw__={ 'team.$id' :team.id, '$or': [ { 'public' : True }, { 'creator.$id' : user.id } ] })
...

Took me a while to figure out what property mongoengine was storing the referenced object id as ($id) and figured it would be of help to someone else. Thanks to apiguy for that!

I get the best of both worlds, the fine tooth comb that the raw query provides and the result comes back wrapped up as my mongoengine Documents.