Audrey M. Roy

I wish I could post screenshots of the app I'm working on with Fuzzy Rainbow.  But I can't talk too much about it yet.

It's a Python/AppEngine + OpenSocial + JavaScript + jQuery UI app.  All the interaction & graphics are done with pngs and crazy JavaScript tricks because @shazow introduced me to the magic of jQuery & jQuery UI and convinced me that JavaScript could do anything.  Working on Fluffy Puffy Cloud Pets also taught me this to some extent, although I did nothing but draw clouds (and integrate my teammates' code) for that project, and I was skeptical of JS back then.

No canvas, no Flash.  If this turns out to be the wrong approach, I'm not afraid to throw away 2 months of code and rewrite it. 

The app is pretty odd.  Things do funny things.  It's either a game or a tool, depending on the user's perspective.

More pieces are coming down the line.  There will likely be a Pinax-based standalone site centered around the app.  There will also likely be a mobile client (maybe PhoneGap).  And more that I absolutely can't mention.  It's pretty ambitious, but it's just a matter of doing it piece-by-piece and cutting out pieces if needed. 

All the seemingly-scattered hacks & experiments that I've done over the summer are now coming together.  It is much more planned and well-thought-out than it might have appeared.  I am excited that I can say this much & can't wait to reveal more.

I can't bring myself to finish revising my book. It's 95% done, but
the remaining 5% is expanding infinitely. I can't stop making changes
to everything.  When I'm not doing that, I'm procrastinating on social
networks.
 
I've decided to give myself a video ultimatum to force myself to
finish by October 8:

 
October 8 is the day that my JetBlue all-you-can-jet pass expires.
Being on planes a lot has helped a lot, since I tend to pull out my
book instead of my laptop on flights. So I'm making that day my
deadline.

Update 9/15:  One of the folks on the GAE team saw this!  First he gave me more apps. Then he sent me another email saying he "just pushed a change live which should fix deployments so that you actually have 250 (instead of the mysterious 100). Please give this a try and let me know!"

This totally changes things.  Wow!

****

I like GAE a lot.  But in the end, it's hard to stick with it for major, time-sensitive projects where you don't have a lot of time to waste on workarounds.  As a developer, you just don't have enough control over your work. 

My biggest troubles:

• You can't pack up and leave appspot.com.  There is the dev_appserver, which you can use to test locally.  But there's no easy, reliable way to run a GAE server clone on your own box.  Appdrop, a project that tried to address this, was last updated in 2008 and seems to have fallen behind the new GAE SDK releases (I could understand - what a pain to keep updating).  Want to run a GAE clone on EC2?  You can't without a lot of work.
• Lots of arbitrary limits.  Sure, they were made with good intentions, to improve performance.  But you're bound to run into one trap or another with no way out.  Max # of files in an app, short timeout for processes, max # of IN queries, etc.
• The local dev server is inadequate for testing.  You need to be able to test your app in an environment identical to your production setup.  I thought I could get away with using one of my 10 appspot apps as a test app, but I was hitting the max 100 deployments/day limit by 4pm and having to wait until midnight for more quota (and the max 250 deployments that they said is untrue, it's really 100).  I had issues with cross-domain requests being treated differently on appspot than on my local server.  When you're working with OpenSocial, you find that different social networks are finnicky about cross-domain requests in different, often surprising ways. 
• (on Python GAE) If an unsupported library isn't pure Python, you can't use it.  Workarounds for this involve using another server, which is irritating.  This is often inevitable - when you're starting a project, you often don't know what libraries you're going to need.
• And finally, you can only have 10 apps.  So it's inadequate for even little hobby or test projects.  As someone who experiments a lot, I usually create (and finish) at least 1-2 mini-projects per week.  I've been reusing my appspot apps, but there are ones that I'd like to leave up and running for others to see and use.

I do want to mention that GAE is a really amazing product that just has a few kinks that need to be worked out.  The GAE team has made great outreach efforts in the developer community and has always been helpful.  And GAE so close to being what I've always wanted.

I'll still use GAE every now and then, and I'm likely to move my OpenSocial app back to GAE in the future.  But GAE has too many roadblocks for rapid development.  I was spending more time on workarounds than on my actual app.

If App Engine were open-source, I could run my own clone of the production server somewhere.  I'd be able to get around all the roadblocks and develop my app using their framework and datastore quickly, in a setup basically identical to production, and then I'd be able to come back and optimize it to meet the appspot limitations if I wanted to have their automatic scaling/load balancing.  I'd also love to be able to peek under the covers for my own knowledge.

Alternately, if App Engine would just let me pay to get around all these issues, I'd get my company's credit card out right now.

Today I tried out Tornado, the Python non-blocking web server optimized for speed & traffic.

[Note: the confusing social media buzz around Tornado might make you think it includes social networking or twittering functionality.  That's incorrect...just setting the record straight here.]

Their chat demo seems pretty snappy, although I didn't see it under large loads or anything stressful.  The long polling stuff is super-cool.  You can start a request and leave it open for awhile, specifying a callback method to go to later.  It's like ajax inside out and backwards.

Testing out their framework...I like their blog demo.  There's a Tornado-App Engine version, which I modified slightly to create a blog to keep track of my canvas experiments (totally unrelated, just killing two birds with one stone)

Check it out: http://labs.fuzzyrainbow.com/graphicsexperiments/

[Note: if you're using Tornado with App Engine, you lose much of the functionality of Tornado that makes it interesting in the first place.  WSGI doesn't support non-blocking requests.  You'll want to use Tornado's own HTTP server.]

Tornado comes with a really simple web framework along the lines of Google App Engine's "webapp" framework.  I already like to use GAE+webapp for really simple projects, like basic API servers, and for projects where a more complex form/template system isn't useful, like very graphical games.  It doesn't have the forms fanciness of Django, but that's good sometimes.

I still have to finish writing the high scores server - the one that all the Fuzzy Rainbow mobile apps will connect to in order to send/receive high score data.  So far it's partly working on GAE+webapp (no auth yet, but I can post and get the scores from a mobile app).  I'm thinking now might be a good time to move it to Tornado. 

A game score tracking server sounds pretty trivial.  But imagine one sending and receiving updates in real-time.  You'd be able to keep tabs on your top players' while they're actually playing.  You could give them live updates on their competitors' scores. 

Not to mention you could run live tournaments and broadcast them publicly.  I didn't anticipate making anything this advanced, but Tornado would make a lot of the implementation easier.

This post is a mess and mainly for my own personal records.  I've been up all night installing Pinax v0.7rc1, integrating my Django art cataloguing app with one of the Pinax sample projects, and trying to deploy it on Webfaction. 

I installed Pinax 0.7rc1 locally as per the "development version" instructions on their documentation site. 

Installation notes:
- Install virtualenv and virtualenvwrapper before pinax, if you haven't already
- When you create the pinax-env directory, do it inside of ~/.virtualenvs/
- I ran into a dependency conflict - I had simplejson 1.9.2 installed, and it wanted 2.0.9.  I removed the old one via synaptic and then deleted the old egg.  When I tried rerunning the Pinax script that installs all the external dependencies, it completed successfully.

Then, I cloned their sample social networking website social_project and played with the site (at localhost:8000).  Its admin site at localhost:8000/admin also worked as expected.  This is great, I thought.  I was kind of surprised that I got it working so fast.  The trick is to read every word of the installation docs carefully - I was running into problems until I forced myself to stop skimming the instructions over.

Next, I decided to integrate my Django art cataloguing app with social_project. 

I did the following:
- added this to socialsite/urls.py: "    (r'^artpieces/', include('artapp.urls')),"
- dropped my artapp's urls.py into socialsite/apps/artapp/ and removed the 'artpieces/' start of each url since it's already relative now
- put artapp/ into socialsite/apps/
- put my artpieces/ (the artapp template folder) into socialsite/templates/.  I just put all the artpiece display code inside {% block body %}.  At the top I added {% extends "site_base.html" %} and {% load artapp_extras %}, where the latter loads my custom filters.  I believe I looked at social_project's tag_app and modeled these after its templates, but I don't remember and don't feel like looking it up.
- dropped media/ into site_media/ for local testing purposes.  media/ contains my css files.
- added 'artapp' to INSTALLED_APPS in socialsite/settings.py  (as well as a bunch of Amazon S3 stuff)
- dropped templatetags/ into socialsite/apps/artapp/.  this contains my Django custom filters.
- (I may have forgotten a couple other things)

This worked for the most part.  My art cataloguing app & its admin interface are now fully functional inside of the Pinax social_project app, except for the fact that thumbnails don't get uploaded to S3.  This is probably because of MEDIA_URL and MEDIA_ROOT in settings.py - they point to the just-for-testing site_media/ right now, I could imagine.  I didn't bother to look because I figured I had gotten far enough to try deploying my app on webfaction.

So, after getting it mostly working locally, I tried to get it working on Webfaction.  I really, really hope Webfaction starts supporting Pinax out-of-box because this is where I got stuck. 
1. I easy_installed virtualenv & virtualenvwrapper in my webfaction acct
2. $ wget http://kernel.org/pub/software/scm/git/git-1.6.4.1.tar.bz2
$ tar -xvjf git-1.6.4.1.tar.bz2
$ cd git-1.6.4.1
$ ./configure --prefix=$HOME
$ make && make install

and then kaboom...this is where I got stuck:
    LINK git-upload-pack
    CC var.o
    LINK git-var
    CC http-push.o
http-push.c:14:19: error: expat.h: No such file or directory
http-push.c:852: error: expected ‘;’, ‘,’ or ‘)’ before ‘*’ token
http-push.c: In function ‘lock_remote’:
http-push.c:936: error: ‘XML_Parser’ undeclared (first use in this function)
http-push.c:936: error: (Each undeclared identifier is reported only once
http-push.c:936: error: for each function it appears in.)
http-push.c:936: error: expected ‘;’ before ‘parser’
http-push.c:943: error: ‘parser’ undeclared (first use in this function)
http-push.c:946: error: ‘xml_cdata’ undeclared (first use in this function)
http-push.c: In function ‘remote_ls’:
http-push.c:1179: error: ‘XML_Parser’ undeclared (first use in this function)
http-push.c:1179: error: expected ‘;’ before ‘parser’
http-push.c:1186: error: ‘parser’ undeclared (first use in this function)
http-push.c:1189: error: ‘xml_cdata’ undeclared (first use in this function)
http-push.c: In function ‘locking_available’:
http-push.c:1262: error: ‘XML_Parser’ undeclared (first use in this function)
http-push.c:1262: error: expected ‘;’ before ‘parser’
http-push.c:1269: error: ‘parser’ undeclared (first use in this function)
make: *** [http-push.o] Error 1

(How do I install Git? https://help.webfaction.com/index.php?_m=knowledgebase&_a=viewarticle&kbarticleid=169&nav=0,33
WebFaction Django/mod_wsgi forum post: http://forum.webfaction.com/viewtopic.php?id=1119)

This is where I've left off for now.  I guess I have to install some XML library on webfaction in order to install git, and then I can get back to installing git on webfaction and continue through the rest of the instructions.

I find it a pain to have to open up GIMP or Photoshop to make gradient strip pngs while I'm in the middle of styling my CSS.  I like to go back and forth, tweaking the gradient strip stops and reloading the web page in my browser until it looks just right. 

Here's an easier way to generate them, if you prefer scripts.  When you're dealing with very transparent or subtle gradients that you can barely see, a graphical interface isn't that helpful anyway.  See the PyCairo docs for more on their gradients and color stops. 

#!/usr/bin/env python
import cairo

WIDTH, HEIGHT = 1, 54

surface = cairo.ImageSurface (cairo.FORMAT_ARGB32, WIDTH, HEIGHT)
ctx = cairo.Context (surface)

grad = cairo.LinearGradient (0, 0, 0, HEIGHT)   # change to 0,0,WIDTH,0 for horiz strips
grad.add_color_stop_rgba (0,0,0,0,0)
grad.add_color_stop_rgba (0.9,0,0,0,1)

ctx.rectangle(0, 0, WIDTH, HEIGHT)
ctx.set_source(grad)
ctx.fill()

surface.write_to_png("gradient.png")

***********

Update 8/24/2009:

Funny, just happened to find this by accident:
http://jtauber.com/blog/2008/05/18/creating_gradients_programmatically_in_python/

A few months ago, I wrote an app to catalog all of my artwork.  I've lost track of quite a bit of my sold art over time, which makes me teary-eyed.

My cataloging app is a Django app consisting of forms that I fill out with details about each art piece, e.g. dimensions, materials, where it's located, who owns it, any stories behind it, etc.  After filling out the details, I upload a variable number of photos of each piece via the app's uploader, which generates thumbnails and sticks them into Amazon S3. 

I didn't bother much with the front-end back then because I already had what I needed.  I hated js, css, html, ajax, etc., and the app was just for myself anyway.  I used it to catalog a bunch of my art, and then I forgot about it.

But last month I suddenly ramped up my front-end skills.  So it was a delight to pick up the app yesterday afternoon and see it in a new light.  I was up until 3am coding unnecessary things like transparent gradients overlaid on text, which I never had the motivation to do before.  It's still not publicly accessible, but chances are it may be soon.

The Google App Engine datastore bulkloader actually does work,
contrary to popular opinion on the issue tracker. It works for both
my local dev server datastore (dev_appserver.py) and my appspot.com
server. I have v1.2.3 of the GAE SDK.
 
If you're on Linux: put your GAE directory someplace nice. Mine is at
/usr/lib/google_appengine. I still don't know if that's the right
place, but it seems nice. Add it to PYTHONPATH, and add PYTHONPATH to
PATH if it's still playing hide-and-seek with you.
 
I had to add my GAE app's directory to PATH to get the bulkloader to
work. There's probably a better way, which I don't know of.
 
Follow their instructions exactly for creating models.py and
whatever_loader.py.
 
Authenticate into localhost using your computer's usual
login/password, which isn't your gmail address/appspot password.
 
Make sure your app.yaml contains this. I put it at the top of mine to
be extra safe. (The "/.*" URL handler messes with the other ones
below it, you know.)
- url: /remote_api
 script: $PYTHON_LIB/google/appengine/ext/remote_api/handler.py
 login: admin
 
I was able to successfully import a .csv full of data into my dev
server and my appspot server.