MichaelMoore.com
- October 7th, 2009
- Facebook
Python/Django
Social Media
Twitter
Web Applications -
We recently launched Michael Moore’s new online community at MichaelMoore.com - just in time for the opening of his new film - “Capitalism: A Love Story.”
The website is designed from the ground up with a social media focus, integrating Facebook, Twitter and a number of other Web 2.0 technologies. The audience is quite large - the site is in the top 10,000 websites in the United States by traffic and receives millions of hits. The social media footprint is also sizable - 313,289 Twitter followers, 131,253 Facebook Fans and numerous integration points with distributed social networks.
The project took about 5 weeks, which is breakneck speed for something this big. When assessing the initial requirements and timeline, we were really forced to think outside the box. Hard work might not be enough to get the site live by the deadline. How could we speed things up?
Although there was an existing site, we opted to re-architect from the ground up in order to take advantage of newer technologies. It may sound surprising, but this actually saved us a lot of time. After considering several web frameworks, we chose Django. It’s an environment that we know well and we were confident it was up to the task.
We began with a multi-server architecture, so the site could scale easily, and deployed the servers behind our own custom load balancer at Media Temple. We tried several different caching strategies, but settled on memcached in Apache as it let us handle the most users per server.
The structured content model we chose also saved us a lot of time. The new site layout was complicated by the existing website, which was really a number of separate websites bound loosely together. In order to overcome this, we created a simple content model and began migrating over the content, organizing as we went along. We scripted as much as we could, but with 30,000+ pages, there was a lot more done by hand than we would have liked.
With such a large amount of content, searchability was essential. For this we built a custom technology stack based on Solr, Lucene and Haystack. Then, to keep all the previous links pointing in the right direction, we created a customizable scheme of Django regular expressions and Apache mod_rewrites.
And, of course, all of this can be controlled from a robust admin interface.
We were working with an external design team, and found great synergy combining their design sense and our love of slick UI tools like jQuery. The Michael Moore folks were also a pleasure to work with. They had a lot of thoughtful improvement ideas as the development process rounded up.
Once the site was assembled, we ran some heavy load testing and were pleasantly surprised with the results. Our hard work and good architectural choices had paid off. A little browser compatibility testing, add water, and Voilà! Instant website.
To say it was an interesting project with some significant technical challenges would be an understatement. This was an all-out effort by some of our top developers to do the near impossible. It can be hard to accommodate such tight timelines, but our team rose to the task. Great job, guys!
Subscribe
27 Responses to “MichaelMoore.com”
Twitter Trackbacks for MichaelMoore.com | Concentric Sky [concentricsky.com] on Topsy.com wrote in to say...
[...] First Tweet 2 hours ago robhudson Rob Hudson Highly Influential Michael Moore site built in Django in 5 weeks: http://blog.concentricsky.com/2009/10/michaelmoore/ #django #eugene view retweet [...]
Eduardo de Oliveira Padoan (edcrypt) 's status on Thursday, 08-Oct-09 23:33:49 UTC - Identi.ca wrote in to say...
[...] @robhudson: Michael Moore site built in Django in 5 weeks: http://blog.concentricsky.com/2009/10/michaelmoore/ #django [...]
Simon Willison wrote in to say...
That’s a really impressive turnaround for a site of that size. Are you scaling entirely using memcached, or do you have some database replication going on as well? Are you using Solr replication at all?
Graham Dumpleton wrote in to say...
Are you using mod_python, mod_wsgi, mod_fastcgi, mod_fcgid, mod_proxy to pure Python WSGI server or something else to bridge Django to Apache? Site uses RedHat Apache which throws away the part of the response Server string which might partly indicate this.
Mike Biglan wrote in to say...
Thanks Simon!
Yes, we have several front-end web servers, a load balancer, memcached, and database replication (master/slave and MySQL Proxy). We originally used cmemcached to connect but it was buggy, as we discovered during load testing; we switched to python-memcached and all worked well. Our original plan was to replicate Solr, but we decided it wasn’t warranted. Static content is already on a separate subdomain, ready to be separated out if needed.
Mike Biglan wrote in to say...
Graham, we went with mod_wsgi and let Apache handle the the static content. Also, we gave it a separate subdomain in case we want to separate it onto a separate server in the future.
Mordy Ovits wrote in to say...
“we gave [the static content] a separate subdomain”
It would have been better to put it on a different domain, not just subdomain. That way, the requests for static content don’t include the (potentially large) cookies.
Mordy Ovits wrote in to say...
“We originally used cmemcached to connect but it was buggy, as we discovered during load testing”
Can you elaborate on that? What were the symptoms? Root cause?
links for 2009-10-08 « Object neo = neo Object wrote in to say...
[...] MichaelMoore.com | Concentric Sky 5 weeks. (tags: django development python) [...]
Wait till I come! » Blog Archive » TTMMHTM: Piano hacks, PHP and Ruby secured, Leisure Suit Larry in Canvas wrote in to say...
[...] Michael Moore’s new web site is based on Django and here’s some info how they made it scale. [...]
布里斯班 wrote in to say...
Another great website in django world
Gonzalo Delgado (gonzalodelgado) 's status on Friday, 09-Oct-09 11:38:36 UTC - Identi.ca wrote in to say...
[...] sitio de Michael Moore usa Django http://blog.concentricsky.com/2009/10/michaelmoore/ [...]
MichaelMoore.com | Concentric Sky wrote in to say...
[...] Continue reading here: MichaelMoore.com | Concentric Sky [...]
Yevgeny Ioffe wrote in to say...
Congrats! nice job!
Alex Capehart wrote in to say...
Great work and truly impressive in the amount of time!! Thank you for choosing (mt) Media Temple…you’re using some of our favorite dev tools (Django, JQuery) which makes it that much sweeter watching this scale up!
Cheers
Mike Biglan wrote in to say...
Mordy: During load testing when using cmemcached, under high load Django started to throw a variety of errors. This in turn broke the connectivity between Django and memcached until fully restarting Django. python-memcached, however, worked perfectly.
Also, the subdomain has worked fine for static content because we use almost no cookies, and the main cookie we do use is a short integer. Our goal was to set the stage for additional optimizations if necessary, though not to prematurely optimize.
Django Project Looks Toward the Future With Django 1.2 | Programming Blog wrote in to say...
[...] Django already powers numerous high-profile websites, from the geo-aware EveryBlock to Google’s App Engine tools. In a testament to Django’s rapid development capabilities, developers Concentric Sky recently rebuilt the high-traffic michaelmoore.com using Django. From proposal to launch was a mere five weeks; you can read more about building the site over at the Concentric Sky website. [...]
Media Blog » Blog Archive » Django Project Looks Toward the Future With Django 1.2 wrote in to say...
[...] Django already powers numerous high-profile websites, from the geo-aware EveryBlock to Google’s App Engine tools. In a testament to Django’s rapid development capabilities, developers Concentric Sky recently rebuilt the high-traffic michaelmoore.com using Django. From proposal to launch was a mere five weeks; you can read more about building the site over at the Concentric Sky website. [...]
Scot Hacker wrote in to say...
Fantastic work. Hoping you’ll submit the site to the directory at djangosites.org (this kind of visibility does a lot to create buy-in from the rest of the world).
Francisco Palm (mapologo) 's status on Sunday, 11-Oct-09 01:24:40 UTC - Identi.ca wrote in to say...
[...] http://blog.concentricsky.com/2009/10/michaelmoore/ a few seconds ago from Gwibber [...]
Django Project Looks Toward the Future With Django 1.2 wrote in to say...
[...] In a testament to Django’s rapid development capabilities, developers Concentric Sky recently rebuilt the high-traffic michaelmoore.com using Django. From proposal to launch was a mere five weeks; you can read more about building the site over at the Concentric Sky website. [...]
Rearchitecting MichaelMoore.com From the Ground Up With Django in 5 Weeks wrote in to say...
[...] Although there was an existing site, we opted to re-architect from the ground up in order to take advantage of newer technologies. After considering several web frameworks, we chose Django [...]
Register-Guard | Web firm lands big contract wrote in to say...
[...] The task of rebuilding the Web site for controversial filmmaker Michael Moore could have gone to developers anywhere in the world. But the work came to Eugene. “They called us and said they had heard that we did good work and asked us to give them an estimate,” said Wayne Skipper, founder of Concentric Sky - the Eugene firm that won the bid. [...]
Vijayinfo wrote in to say...
That’s really impressive!
Maykel Moya (maykelmoya) 's status on Tuesday, 13-Oct-09 19:37:08 UTC - Identi.ca wrote in to say...
[...] MichaelMoore.com in Django: http://blog.concentricsky.com/2009/10/michaelmoore/ [...]
Drupal or Django, Framework or CMS? — scot hacker’s foobar blog wrote in to say...
[...] michaelmoore.com, recently rebuilt in Django in five weeks. [...]
Concentric Sky Launches MichaelMoore.com « palmit.commentary wrote in to say...
[...] Recently, we used Django to implement Michael Moore’s new website and we couldn’t be happier with the results. Concentric Sky developed the back-end code for this site in Python/Django with a custom administrative interface and social media aware Content Management System (CMS). The site is optimized for millions of hits per day and it has been humming along nicely since it was launched. You can read more about this project on the Concentric Sky blog. [...]
Leave a Comment