Server Troubles

Our server has been crashing lately--I apologize if you tried to come to the site before 3pm today and found that it wasn't here. I'm working actively on trying to find the cause of these crashes and find solutions. I think a move to a different server, one with mod_perl, may be in order.

Metadata Scorecard, New Way of Polling Weblogs, And Email Updates

I added a new section of the site today called the Metadata Scorecard. The idea came from Jon Udell: he wanted to see if we could encourage participation in metadata creation by giving it a score board. If people could easily see who was contributing the most, maybe people would spend a few minutes adding data so that they too might appear on the list. It's worth a try. I also took the chance to explain a bit more about the system for getting new types of metadata into the dropdown box--basically, any metadata label that appears in the database more than once will appear in the dropdown. This, I thought, would prevent it from filling with irrelevant labels, and also allow for the more popular ones to become normalized a bit. I didn't want to see everyone thinking of a different way to label a book's website, for instance. Anyway, it's still pretty simple, and could be easilty abused, but so far nobody has bothered.

I also completely re-wrote the way the hourly poll goes out and finds out which weblogs are talking about which books. Up until yesterday, the poll would run every hour, saving up links in a big hash-table and then at the very end save them all to the database. Occasionally the server would get slow and I'd have to kill polls in order to lessen the load... but it killed me because I'd lose all of the data that it had already collected during its run. Well, I won't go into the boring details, but now that's no longer a problem. And hopefully it won't hog as much memory as it did before either. That said, if you see anything strange happening, let me know.

Finally, some of you have opted to receive emails whenever one of your friends mentions a new book on their website. Those emails will now also include the excerpt from their weblog that mentions the book, if I can find one. There are still some glitches, so I apologize if you already received one and it looked really ugly. To sign up for these emails, create an account, add some friends, and sign up for the email in the Control Panel.

One last thing. I've been hearing reports that some of the titles for items on the site have been wrong. For example, while the cover image for "Linked: The New Science of Networks" was correct, it was displaying the title, "Personal
Responsibility Counselling and Therapy: An Integrative Approach". I tracked this down to a bug in Amazon's web services. They're aware of the problem and the fix should have been released. Now I just have to wait for my cached information to expire.

Okay, that's it for today. Oh yes--I also found a roommate for SXSW, and will be staying at the Hampton Inn. Thanks for the replies.

Looking for a roommate at SXSW

I'm going to be in Austin, Texas from March 7th-11th for SXSW, but can't afford to pay for the full cost of a hotel room. I'm staying in a double at the Hampton Inn, which is about 3 blocks from the convention center. If you're going to SXSW during this time and need a room, let me know.

SXSW Finalist

This site has been selected as a finalist in SXSW's Developer's Resources category. I've submitted sites for the last three years, and this is the first time I've actually had something get nominated, so I'm really excited. There's a People's Choice Award too--so if you feel like it, you can help All Consuming get the popular vote, which to me would be cooler than winning the actual award.

In any case, I'll be attending SXSW for the first time this year. It's a little scary since I'll be meeting most of these internet people for the first time, but I think it'll be fun. If anyone who's attending has suggestions for what I should be doing, let me know.

New Applications Using All Consuming Data

I just got back from vacation only to find that DJ Adams has begun to do some very interesting things with All Consuming's SOAP API. First, he created booktalk, a script that will check to find which of your books have recently been commented on by others. Next, he wrote a simple Perl Module that can log you in to All Consuming and add books to your collection: Allconsuming::Agent. He talks about his results here. It wouldn't be too difficult to add this functionality to the SOAP API so that scripts like these didn't break if I changed the way log-in happens. Very nice though. He used this script to automatically synchronize his Amazon.com's wishlist with his All Consuming "Currently Reading" list... which is something I've been meaning to allow people to do as well but had hesitated to implement since Amazon's current Web Services don't have a straight-forward way of getting all the books from your wishlist. I should take a look at WWW::Amazon::Wishlist.

Kellan from Laughingmeme.org has also created a script that pulls his currently reading list from All Consuming via SOAP: here's the script. Very exciting stuff.

In other news, I've updated the weekly RSS feed to provide more information and look a little nicer in news readers. If people preferred the old way (it was a much smaller file before), let me know. Next step might be to make it update only once a day instead of every hour since some people report that there's too much of the same stuff showing up every hour. I don't use a news reader myself, so anyone that does, please let me know if this is an improvement or whether there's something else I could do that would be better.

I've still got a backload of email to get through so I apologize if I haven't responded to everyone who's written in yet. My next feature to add will be a sort of scorecard for the site that will display some statistics on visitors, status on the hourly updates of scraped sites, and also create some lists that display the users who've contributed the most metadata... this is all information that I can check up on (and usually do multiple times a day), but I think others might be interested in it as well. If there are other stats you'd like to see surfaced, let me know.

Out of the Website Jan 10th-27th

I'm going on a two week vacation in the middle east (Dubai in particular), so will not be around to maintain this site during that time. Think pools, beaches, 80 degree weather, and cool drinks. Hopefully the site doesn't die--I've raised it to be independant and kind to strangers, so hopefully you'll all get along fabulously while I'm gone and I won't even be missed. If anything does go horribly wrong, I'll be checking email so please let me know (contact address listed at the bottom of the page).

I'll be back on the 27th of January. In the meantime, my friends, keep reading. I've got Middlesex, Revising Fiction, and Interface Culture to fulfill my reading, writing, and technology thirsts while I'm in between diligent and thorough beach-inspections.

The Disruptive Web, Use my XML, Top Referrals, & All Consuming Ping Form

Jon Udell has written an article titled The Disruptive Web that mentions All Consuming in the context of his own LibraryLookup experiment. Now anyone can use his bookmarklets to find out whether any books on Amazon, or this site, are available at your local library.

I'd love to find even more examples of people using All Consuming to build other applications. For almost everything that I do, I leave xml files available for use by others. And there's the SOAP Interface which is still pretty limited right now, but which I'll continue to add to if people are using it. Please use them.

I added a list of Top Amazon Referrals of 2002, in case it was interesting to anyone.

Finally, due to an increasing number of requests, I've added a way to have All Consuming manually scrape your weblog instead of having to rely entirely on weblogs.com. You can find the ping form on the homepage at the bottom of the right column--just enter your site's url and it will add any new books that it finds, and update the timestamp on books that it already knows about. Hopefully it will be able to help debug situations where the hourly poll doesn't pick up your books.

First Line Trivia

I got the idea from my favorite neighborhood bookstore, Bailey Coy Books, which has the first line from a book on a street board on display that you can try and guess. Those who tell the cashier the correct answer get 20% off any book in the store. I'm hopelessly addicted to trying to guess the book that it comes from, and have been guessing for about 3 years now (though my percentage of correct answers doesn't seem to be improving). Now visitors to this site can play as well (without the discount). You'll see a new box in the top right corner of the homepage which displays the first line of a book, and a link to that book's detail page if you think you can guess it (or are just curious about the answer). If we start getting more first lines in there (which I'll tell you how to do in a second), I may try to make it more of a challenge by giving you a few books to guess from and seeing what percentage of people can get the answer right. At work we call this an attempt to influence your "frequency of visit", but I just think it's fun.

The first line trivia box is built on top of another new feature which has just been implemented. On each book page, you'll now see a new bucket titled, "Book metadata & other interesting stuff", with a link to edit that data if you're signed in. Here, you can add anything that you think is useful to link to this particular book. It could be the first sentence, or a favorite quote, or a link to a news article, or the number of pages, or something else that I haven't thought of yet. I've left it pretty open for now because I'm curious to see how people use it (if at all). From there we can refine it and make sure it works in the best possible way. Have fun!

Also, I'm sort of in the mood to add some other new features to the site, so let me know (either by email or comments) if there's something that you think is just begging to be done. I've been thinking of adding some kind of discussion area to each book page, but am not sure how useful that would be. So, what do you want?

Most Mentioned Books of 2002

Which books do you think were most talked about this year in the world of weblogs? I had some guesses, but thought I might as well run a quick query to find out for sure. Taking a tally of every book mentioned in weblogs since this site was begun in August, I've created a list of the top 100 books of 2002:

Top 100 Most Frequently Mentioned Books of 2002

A couple observations:
+ There's only 1 fiction book in the top 10: American Gods by Neil Gaiman.
+ 6 of the top ten books' authors have weblogs or active mailing lists.
+ The distribution of the number of weblog mentions and their respective placement on the list appears to be behaving loosely according to the Zipf Power Law (see "Linked" for more about that), not surprisingly.

Here are a few other end-of-year lists that are related to books:

Time's Best and Worst Books of 2002

Amazon.com's Editor Picks from 2002, Customer Picks from 2002, and more.

New York Times 2002 Best Books

Washington Post's Book World Raves

Book Magazine's Best of 2002

Publishers Weekly's The Year In Books

Borders' Best of 2002

Barnes & Noble's Top 50 Staff Favorites

There are millions of lists out there of course, and even quite a few lists of lists, but this should cover most of the primary lists that have been published so far. There are a couple books out there, like Middlesex (not on the All Consuming list) and Atonement (number 76) which were on numerous of the "Best of" lists and which I haven't read yet but look forward to.

I didn't read many books this year that were actually published this year. I'd have to say that my favorite book of 2002 would have to be the book I just recently finished, The Cave.

What was your favorite book in 2002?

Now Accepting ISBNs

Jill and Noah recently called me on my Amazon-centricity, claiming that I should not only collect links from Amazon as it's only one of the many different places that people might use to link to a particular book. So, in a way, I wasn't creating the list of the most talked about books among webloggers, but rather more of a list of the most talked about books among weblogging Amazon customers. Not keen on letting this connotation bring me down, I updated the hourly poll script to also pick up on any link to a book that has a "isbn=0123456789" in the URL. This includes links to BookSense, Barnes & Noble, this site, and a few others. Hopefully now we're a little more rounded, although there's still plenty of room to continue to improve.

I've also updated the links on this site so that if you're using the Javascript Include, I won't count those links as book links since they're already counted. However, if you just want to link to a book page from one of your entries, that will count.

The weekly list of most-mentioned books may be a little inaccurate during the next couple days as we absorb all the isbn links that he had not been picking up prior to today.

Steven Berlin Johnson Emerges

The author of Emergence, which incidentally was one of the primary inspirations behind this site, now has a weblog. I found this out, appropriately enough, by seeing that his book had again gone to the top of the weekly list, and that people were linking to his book when linking to his weblog. I think this is the moment when All Consuming became aware of itself, the moment when it was able to detect, in an indirect manner, its own maker, by using tools that the maker made for it.

In similar news, the number two item today, Google Hacks, will have code samples that I gave to Tara when she wanted to take a deeper look into how this site, and my other book watcher worked.

Automation Rules

I love automated websites (perhaps I'm biased since I'm in the Automated Merchandizing team at work), because I can take a month off of site development and maintenance here to do NaNoWriMo, and nobody notices. Have a good November.

Downloadable RSS Reader

I have created a simple script that will read any RSS feed that includes links to Amazon, and display it as a nice list. See the example on my site.

Files are downloadable by clicking here: bookwatch.tar.gz

To install, follow these instructions:

• Upload the .cgi and the .txt templates to a web-accessible directory.
• Upload the .pm files to a non-web-accessible directory. Both the Amazon and the Google file will require that you have tokens or keys that you have to download from their respective sites. Go here to get one from Google, and here to get one from Amazon, if you don't have them yet. Then, open the two .pm files in your favorite text editor and add them to the appropriate lines (as marked in the file).
• Chmod the bookwatch-plus.cgi file to be 755.
• Open bookwatch-plus.cgi in your favorite text editor and change the line that begins "use lib" to point to the directory that you uploaded the two .pm files to.
• There are further modifications you can make to the script by changing the lines in bookwatch-plus.cgi that are above "The Inner Workings" line. You don't need to though, it'll just use the defaults that I've set up.

If anyone downloads this and has any trouble setting it up, let me know. I've tested it once on my machine, but not on every possible machine in the known universe, so there may be a bug or two. I'm also open to any improvements or suggestions that anyone has to offer.

Some Context

I'm now starting to save (and display) some context information about each book link on a weblog. Inspired by Mark Pilgrim's work, I think the conversation surrounding a particular book can be more easily surfaced if you don't have to go to each individual weblog to read what the person has to say. You can see these snippets on the main page (accompanying the links found in the last hour), in the archive, and on book detail pages. I'm still far from having a perfect context capturer, and there will most likely be some problems that I'll run into as a result of this in the coming weeks.

If there are any Perl experts out there that can think of a way to isolate the perfect snippet of text around a particular link, let me know. Some of the challenges involve finding contexts for multiple books per page (some contexts may overlap each other), searching on hundreds of weblogs per hour without overwhelming the server, and saving the result as valid XML.

In other news, I've added a side nav to the front page in order to surface some of the more interesting features that I was feeling were getting buried in the news archives.

Possible Breakage Alert

I'm trying something new with the poll that checks websites for book links, so it might break once or twice during the day. If all checks out fine though, I'll have something new to show everyone in a day or so.

Fateless and the Art of Deception

I've been waiting to see Fateless, this year's winner of the Nobel Prize for literature, show up on the weekly list, and it finally has. I wonder if it will make it above #14 though... come on, just a few more links.

The other book I've been watching is The Art of Deception by Kevin Mitnick. Hacker guru, sent into solitary confinement for many years for allegedly breaking into NORAD, inspiring the movie War Games, Kevin Mitnick actually came to talk to us at Amazon about social engineering and how to prevent hackers like himself (now reformed, however) from breaking into basically anything they want to. He's still not allowed to use the internet, and is only allowed to use a computer on a limited basis. It was a very interesting talk he gave, the best part of course was all of the stories about various tricks he knows about (but would not admit to actually using). I'm sure the book goes into much more detail about those things, which makes it a tempting purchase.

Blogdex Restructuring

I'm keeping a close eye on progress over at blogdex, as a whole slew of new features are set to launch next Monday. Some of them that I'm particularly interested in (as it may relate to All Consuming) are the "link diffusion" charts (which will display how a particular link was born in the weblog community and then how it spread out), and also the "social weather index". When I was developing this site, I experimented briefly with extracting phrases like "I like", "I love", "I hate", and "I feel" to see if their numbers fluctuated in any set pattern, and if those fluctuation could be mapped to any kind of overall emotional index. It was stopped short due to server problems, but if Cameron's social index looks useful, maybe I'll try a couple more iterations on that idea.

Down and Out in the Magic Kingdom

Cory Doctorow, of boingboing.net has writting a book: Down and Out in the Magic Kingdom. And it has gotten quite a bit of attention from webloggers. What other books have webloggers written, outside of books about blogging? For all of you out there who have a novel in you, disregard this article, and, in fact, if only in sheer defiance of such a silly article, go sign up over at NaNoWriMo and join the thousands who will be attempting to write a 50,000 word novel (~150 pages) during the month of November. I will be.

What are the locals reading?

I don't have anything against books about blogs (that I'll mention here), but I have to admit that I am a little anxious for people to stop talking about them. I feel strange looking at the aggregation of people's book interests and somehow trying to wish for them to change. It just makes me feel like I'm visiting a town obsessed with itself, which probably isn't an inaccurate assessment, or even a bad one necessarily, but in a way books about blogs seem to be for the tourists. I want to know what the locals are reading.

What We're Reading The Day After September 11

Here's the archived list of what we're reading on September 12th, 2002.

It's very reassuring to see the variety of books that people are reading the day after. There are tribute books as well as books that criticize America's involvement. There's some evidence, through this one proxy at least, that we're mourning as well as continuing to think about the complicated new world that was revealed to us (even though it had been there for a long time) around this time last year.

Bug in Books Poll Found and Killed

Up until today I was only checking for updated weblogs for links that looked something like this:

http://www.amazon.com/exec/obidos/ASIN/[ten digit ISBN]/etc

It actually doesn't matter if it's a link to amazon.com or amazon.co.uk, I'm only matching the end of the url from "ASIN/" forward. Anyway, since I happen to work at the company, I noticed that last week we internally started linking to items with a url that looks more like this:

http://www.amazon.com/exec/obidos/tg/detail/-/[ten digit ISBN]/etc

And since most people find their links by searching Amazon first, and then cutting and pasting the URL, the links in weblogs began to look more like the latter than the former. I only started accepting links like that today (and will of course still look for links that look like the former), so the volume of books found each hour may go up for the next couple days while this site catches up to everything people have begun reading over the last week.

All Consuming is becoming popular enough that pretty soon I'm going to have to begin looking for links to this site as well, but we'll deal with that when it's more of an issue. Right now, most people are linking to this site from their javascript includes, so they're items I already know about.

All Consuming Introduces New SOAP API

I took my first baby steps with creating a SOAP interface for All Consuming today. If you have experience with SOAP, it's now pretty easy to get any of the information directly from this new API rather than from xml or rss feeds.

Here's a sample script that uses All Consuming's new SOAP API.

If you're familiar with perl, here's a quick sample of what you need to do: make sure you have SOAP::Lite installed on your server, and then use this code:

use SOAP::Lite +autodispatch =>
uri => 'http://www.allconsuming.net/AllConsumingAPI',
proxy => 'http://www.allconsuming.net/soap.cgi';

my $AllConsumingObject =
AllConsumingAPI->new(
$hour,
$day,
$month,
$year
);

Where $hour, $day, $month, and $year are defined previously, of course. After you have that object, you can then call a number of methods on it (described below) to get whatever information you want.

The methods are pretty limited right now, since they only return hash references with lots of data in them right now. Within the next week I'm going to make more friendly methods for getting the data, but I wanted to release this today because I'm just so happy with it. If you end up playing with this, I'd definitely appreciate a quick note to let me know if it's helpful at all, or if there are certain features that you'd like to see sooner rather than later. I wouldn't consider any features permanent just yet, since I'm so new to this and am bound to discover that I've done a million things incorrectly. It's fun, though, and that's all that matters.

MORE...

Javascript Modifications

Responding to suggestions from Kevan and Bill about the Javascript Include, I've added two new features:

1) You can now specify how many items from your "Currently Reading" list you'd like to have displayed in the Javascript. If you frequently read several books at once (like I do), this will give you the ability to display that fact on your weblog.

2) You'll notice that in the RSS and XML feeds available here, that I've added my own Amazon Associate ID whenever providing links to Amazon detail pages. You can choose to change the Associate IDs to your own, in the RSS and XML feeds for your reading lists by going to the Control Panel > Edit Your Info page, and changing the Associate ID field to whatever you like. Your Associate ID can also be used on any Book pages that you link to from your own weblog by including the "&assoc_id=[your associates id]" at the end of any link to item.cgi. For example, a link to The Death and Life of Great American Cities, with an Associate ID of "kokogiak-20" would look like this: http://allconsuming.net/item.cgi?id=067974195X&assoc_id=kokogiak-20. If you have any questions about that, let me know.

3) Coming Soon: the ability to receive an email whenever a friend of yours lists a new book on their weblog or updates their account here. I've got to test it a bit first, but you can sign up for this email by going to the Control Panel > Edit Your Info page, and checking the "Receive Emails" checkbox.

I've continued development on this largely in response to people who are using it and providing feedback. If you have a good idea, please let me know, I'm in my respond and improve phase for this site at the moment.

Recommendations from Friends

New Feature: Recommendations! You can now add weblogs as friends and get a customized list of what your friends are reading. Friends have always been a valuable source of book recommendations from me, and I've always wondered why there wasn't an easy way to tell at a glance what all your friends are reading, and what they're saying about these books.

It works like this. This site collects books two different ways: by checking recently updated weblogs for links to books at Amazon, and by allowing people who sign up (it's free) to create their own reading lists. Already, this site is keeping track of what thousands of different people are reading.

With this information, I can now let you select certain sites that you trust to read the types of books that you're interested in. As long as they notify weblogs.com that they've updated their weblog, or have an account here, we'll know when your friends are reading something new.

I then take this list of all the books your friends are reading, score them so that more recently read books are favored over books read a while ago, add bonuses to the score when more than one of your friends is reading the same book, and present the list to you. Super simple.

So how can you get recommendations? This is what you need to do.

1. Sign up.

2. Log in.

3. Go to your control panel and add a couple friends (we'll even help you find people we think you might like). Add me if you like.

4. Check your Recommendations page. If any of your friends have books in the database that we know they're reading, you'll get your first list of recommendations. Check back whenever you're looking for a new book to read.

Let me (erik at allconsuming dot net) know if you have any questions.

TrackBack Everywhere

Every book page on this site is now TrackBack-enabled. If you're using Movable Type or another TB-enabled weblog, and you're mentioning a book, this means you can link your entry with the book page on this site. See this book for an example of how this is displayed on the page. If you have the TrackBack bookmarklet, that should work on each page, and the TrackBack url to ping is listed at the bottom of each page. Soon, I plan on encorporating this into the hourly build so that TrackBacked entries will also automatically show up on the front page and be used to help calculate the most popular books per hour and week.

Notable Books

After watching the books that float to the top of the All Consuming lists over the last month, I've noticed that some of the books tend to reflect memes in the weblog community at large; sometimes something related to world news, sometimes something very specific to the world of weblogs. So this category will be about highlighting those books, as they're sort of mini windows into the thoughts of the readers.

Foundation, by Isaac Asimov is the first book in this series. There have been articles floating around lately tying this book to Osama Bin Laden and his Al-Qaeda network (which means foundation), based on some possible translations of this book into Arabic. Silly as it sounds, there are a number of articles and discussions about it.

Track Book

Taking of the advantage of Movable Type's TrackBack feature, I've created a page that lists weblog entries about books. Because I love a horrible pun, I've named it:

Track Book

If you have a weblog using Movable Type (or another TrackBack enabled tool) and you find yourself writing about books every once in a while, you can now ping this page with your book-related entries.

Book Websites

Here's a list of other websites out there that may help encourage your book-worm tendencies.

ISBN.nu: A terrific service that allows you to compare book prices across 9 different online retailers.

Oddbook: Making sense of your reading. A friendly php script to help organize your reading on your own website.

Book Review Repository: An interactive repository of Book-reviews created by bloggers from all over the www.

Singlefile: Organize your book collection.

Banned Books Project: Lively discussions around the topic of banned books.

Blogcritics: A sinister cabal of the web's best writers on music, books and popular culture miscellanea - updated continuously.

BookCrossing: A global book club that crosses time and space. A reading group that knows no geographical boundaries.

Booklend: A lending library by post.

If you know of other book-related websites, please leave a comment here or email me and I'll add it to this list.

Currently Reading javascript include for your weblog

I added a new feature due to popular demand (I think one person asked for it). If you create an account and log in, and add a few books to your collection, a javascript include will be made that you can add to your own weblog (instructions here). It will feature the most recently updated item from your "currently reading" list. In addition, there are rss and xml feeds for your entire "currently reading" list, as well as for your "favorite books" list. Share and enjoy.

For an example of how this looks when working, take a look at the right hand column of my weblog: erikbenson.com.

Acknowledgements

This site is built on top of a whole slew of open-source and collaborative software efforts. It wouldn't be proper to say I built this site (which I did in my spare time during the span of a couple days) without acknowledging the work I'm building on top of.

MORE...

Perma-linking to non-permalinkable things

One of the main challenges with scraping other peoples' sites and storing information about them is that you don't have context. People talk about a book, sure, but what did they say? So you have to have links back to that site. Due to the nature of scraping, and the fact that there's no standardization of how a site's content is stored on its homepage (some people, I know, are thinking of ways around this), the scraper can't know which entry talked about the book... just that it was mentioned on the front page somewhere. Linking to the front page, then, is a largely futile business, since it requires the person (who's only interested in the book) to read through all the entries to find the reference to that item. To make matters even worse, people may not link the book's name, so scanning the page is also futile. I found, when prototyping this last week, that I often had to view source and search for the reference to the string "asin" or the actual ISBN number itself. That sucks.

In typical computer geek fashion, I created a script that does that for me. Now, all the links to actual sites go through this script, which searches for (and, more importantly highlights), the reference to the book. Not only that, but it puts a little anchor tag next to that entry so that when the page loads, the browser scrolls down to that entry. A self-made perma-link, of sorts.

Try it out: Mockerybird's reference to "Godel's Proof".

I think I'll probably be expanding the functionality to a stand-alone perl script that others can use as well. It makes permalinking so much nicer, I think.

Movable Type Turned On

Rather than employ Salieri for this site, I've decided to give Movable Type a try. I'm impressed by their feature set, jealous of their zealousness, and really wish I had some way to claim some of their excellent work. This is really a great application, I love them.

One of the primary reasons I've tried this out is so that I can piggy-back on their TrackBack innovation, which is the first real step I've seen towards the semantic web . This kind of stuff for some reason makes me really excited. My friends and family just don't understand why, though, and that's a shame.

Anyway, lets get a move on. Or as Kharis has started saying, "Bring it!"