Holovaty.com

New EveryBlock cities launched

We've just added two cities to EveryBlock: Charlotte and Philadelphia! I've written more over at the EveryBlock blog.

Request: Headless HTML rendering engine?

Warning: Seriously geeky request ahead!

I'm looking for a way to render arbitrary Web pages -- including CSS and JavaScript -- and access the resulting DOM tree programatically, i.e., in an automated/headless fashion. I want to be able to ask the following questions of the resulting DOM tree:

For a given element, what font family, size, and color is the text?
How tall and wide (in pixels) is a given <div>, <table>, etc.?
What are the x/y coordinates of a given element (from the upper-left corner of the page, or lower-left, or wherever)?
For a given element, what is its text content?

The rendering must be state-of-the-art, handling advanced CSS that Firefox, Safari and IE handle. It should work on Linux. Bonus points if there's a Python API for this magical DOM tree.

This is all stuff that standard in-page JavaScript could accomplish, but the catch with me is that I need to be able to do it in a completely automated way, on arbitrary pages, on a headless server.

I know Gecko and Webkit provide this, but I'm not sure where to start with them. The docs and articles I've read seem to be focused more on embedding the full browser window in a GUI application than embedding the rendering engine itself and manipulating the resulting pages.

Help! If you have any clues, I'd be grateful if you left a comment or got in touch with me.

Check out my Radiohead remix

Radiohead is holding a "contest" called Radiohead Remix, in which they're inviting fans to remix the song called "Nude" from their latest album. They've released the raw tracks -- separate, isolated audio clips of vocals, guitar, percussion, etc. -- and are encouraging people to remix the tracks to create something different, then upload it to radioheadremix.com. I put "contest" in quotes because there's no prize other than a guarantee that the band members will listen to your remix. But that's still kind of a cool prize.

I listened to a bunch of the submitted remixes on Wednesday and was kind of disappointed that none of the ones I listened to did anything interesting musically. Most of them retained the same techno/electronica feel of the original song, kept the song's melody intact and added a couple of drum beats. So tonight, I gave a shot at making my own remix.

For context, I'd suggest listening to the original song first. You can find it on the "In Rainbows" album or listen to this remix to get an idea of the song's melody/mood.

My remix is called "Nude (jazzy acoustic)," and you can listen to it on the site or using this embedded player:

It uses only Thom Yorke's vocal track from the recording, which means I was able to change the song's chords from the classic Radiohead melancholy to something a bit happier/jazzier. It has four guitar tracks -- two rhythm, one bass and one fingerstyle melody part for the extended fadeout. I cut up the vocal track in six places to fit the rhythm better, but they're in the same order as on the original recording, including the extended wordless vocals at the end.

It's kind of soulful, in a weird falsetto-Eddie-Vedder-ish way, especially compared to the original recording (which is also soulful, but in a much different way!).

If you like it, please vote and tell all your friends to vote! I'd love for Radiohead to hear this. :-)

EveryBlock hiring a Python screen-scraping expert

Attention Python screen-scraping experts! We're looking to hire another full-time developer at EveryBlock. Our site, which just launched a few weeks ago, compiles a wealth of granular geographic data and publishes it on a block-by-block basis. We offer a distinct Web page (plus an RSS feed and e-mail alerts) for every city block in Chicago, New York and San Francisco. We're expanding to more cities and more data sources. And we have a ton of fun features and projects up our sleeves.

This position involves contributions to all of our site's technology and data, with a concentration on screen-scraping public data from government Web sites. Some specifics we're looking for are:

Mastery of screen-scraping
Experience programming in Python
Experience with geographic data

Experience with Django is a nice-to-have.

For more on EveryBlock, check out our launch announcement and this recent interview.

This is an opportunity to work on an exciting and important project with a talented and experienced Web development team. We're currently only four people, so you'll have a lot of freedom and opportunities to make a difference.

This is a full-time, salaried position, on-location in our modest downtown Chicago office. We're a startup, funded by a grant, trying to make the world a better place. Please contact me if you're interested or have any questions. Tell me about the gnarliest site you've ever scraped.

A couple of EveryBlock interviews

Back in 2006, I had a very enjoyable interview with Robert Niles at Online Journalism Review. Now, Robert and I have gotten back together for another e-mail conversation about my latest project, EveryBlock: check it out.

And there's more! Earlier today, Rex Sorgatz published an interview with me about EveryBlock, with more of a technology focus.

Thanks to both Robert and Rex for the great questions.