I was explaining to someone the other day about the old blogging dichotomy between “thinkers” and “linkers” and the value of both approaches. Personally, I guess I am sometimes one, sometimes the other, but there’s no doubt that linking is easier and it’s what I do by default when I don’t have a more substantial essay or interview or other original content to offer.

As it happens, I am working on an interview for this site, but in the meantime I can’t resist posting links to other people’s stuff when it’s relevant. I actually think that’s important because a site that doesn’t link out much sends a message that it expects its readers to find everything they need in one place, which is presumptuous to the point of hubris. That’s also ignorant about how the web works, in which just about everything of value is distributed across a network of contributors.

Over at Many-to-Many, Clay Shirky’s blogmate Ross Mayfield posts Many-to-Many: Tag This? about Feedster’s new “Tag This” feature:

Feedster is introducing a Tag This widget that blog authors can include in their posts for readers to anonymously tag posts. A volunteer manual way of building a database. After you enter a tag, you get to see the list of tags for the post, but they don’t link anywhere so the reward for the effort is unfulfilling. (Rafer notes: The tags submitted now are “real” and being databased, so give it a shot on your blog or mine. Just due to time constraints, the tags are only displayed once a new tag is submitted. All the tag data will be available via the expected and reasonable mechanisms shortly.) Blog search engines serve readers and with future iterations this hints at a good distributed way to engage them.

See Also: Bookmark This

Great post by Pietro Speroni explaining the difference between a tag set (an un- or arbitrarily ordered list of tags associated with a URL) and a tag cloud (a list of such tags ordered by frequency.)

He notes that tag clouds tend towards power law distribution, and goes on to say

If a tag cloud was a power law, we could express it as an ordered list of tags, and a number. The number representing the steepness of the curve, and the list representing the tags, ordered from the most popular to the least popular.

The fact that tag clouds only approximate power laws, means that if we try to express a tag cloud as a power law, we will be making an error, and although we might expect the error eventually to go to zero, at the beginning it might be quite massive. Also global cultural changes, even small ones, might shift the resulting aim from one approximated power law to another, thus temporarily generating a bigger error, until the change has been integrated in the curve.

Apropos yesterday’s post about the meaning of pages changing over time, he finds another example of same.

For example the paper: Clay Shirky: Power Laws, Weblogs, and Inequality, has by now being bookmarked by 113 person on delicious. When it came out the term ‘long tail’, was not used. […] On the issue of October 2004 the article from Wired: The Long Tail came out. […] This article changed the way people looked at power law. Thus it changed the way people perceived the previous article from Clay Shirky. At the moment 8 people have tagged his paper as ‘longtail’, and 3 as ‘long_tail’.

He builds on this intiution to talk about ways of finding not just similarity or difference between two links based on their tag clouds, but on being able to compute distance in information space, an idea with potentially fantastic value. Read the whole thing.

I think Pietro is really onto something here, with one caveat: I think that the goodness or badness of fit between any given tag cloud and a pure power law distribution is likely to persist over time, and that the misfit is itself informative.

Now I like power law distributions as much as the next person (well, way more, actually), but in this case I’m seeing a kind of variability in the tag clouds that isn’t easily explained away as an artifact of small scale.

For example, take two links on del.icio.us as I write this: one pointing to anti-virus software, which has currently been tagged by 91 people, and one about Amazon image hacks, which is rising fast, and which I measured when it had also been tagged by 91 people.

The virus page tag has 65 unique tags, for a tag/user ratio of 0.71; the Amazon page has 79 unique tags, and a ratio of 0.87. The tail is longer for the Amazon page than for the virus page, and, at the same time, the Amazon head is higher. The tag cloud for the Amazon page begins “42 amazon/18 images/15 hacks/14 web/12 cool”, while the virus page is “20 antivirus/15 virus/10 security/10 free/9 online”, a much flatter distribution.

I agree with Pietro that the curves will become clearer as the scale increases. However, I doubt that they will also converge on neat enough mappings to power law curves to be mapped to one another based on their coefficient of steepness. (Edit made after mail from Pietro explaining his idea further.) I think the tag cloud for the anti-virus page is more constrained because it’s clearer to the users what that page is about. Put another way, I am betting, based on early observations of tag cloud development, that the curves for each of those two pieces will become more like some idealized curve, but I do not think that the pieces have the same ideal curve.

Now there are source and audience issues here, so what the compressed tag cloud of the anti-virus page could be telling us includes both “that page is obviously about one or a few things” and “the audience for that page has a coherent view of what it is about”, and, of course, the Amazon link is telling us both “that page covers a lot of different issues” and “several different audiences have tagged it based on local perspectives.”

It will be hard to isolate those effects, but I am convinced that they are there, and that waiting for every tag cloud to settle into a pure power law is actually giving up on some really valuable information that is better regarded as communication about a particular intersection of content and community than simply being an artifact of small scale.

A persistent theme in the interest in folksonomy, and tagging in particular, is the threat of majority control. When the meaning of a link is determined by the masses, it can be inimical to anyone with a marginal point of view, and, the larger the pool the majority is pulled from, the greater the pressures towards lowest common denominator choices.

You can think of this as the ‘picking a restaurant’ problem. If you and one other person need to agree on a restaurant, you have at least a chance of selling them on the wonders of Uzbeki cuisine, and schlepping out to that hole in the wall in Queens. Once you need agreement among a large enough group, though, it’s Pizzeria Uno for you. In many social situations, scale drives the group to the lowest common denominator.

Last week, I had a chance to look at tagging at scale, related to my Ontology Is Overrated piece, and wrote a little script to examine the development of the consensus view of tags. After OIO came out, it was tagged on del.icio.us fairly frequently over the following week. As of this writing, a little over 700 users have tagged it, with 450+ unique tags, roughly two-thirds of which tags were (of course) used by one and only one user. I used this data set to hack up a python script that shows me the list of unique tags, sorted by popularity, after any given user tagged the post.

The output looks something like:

1. tagging mooch
2. tagging mooch fordrew
3. tagging ontology mooch fordrew folksonomy clayshirky
4. tagging ontology mooch fordrew folksonomy clayshirky
5. tagging ontology folksonomy clayshirky mooch fordrew
…

which is to say, after the first user tagged it, the total tag cloud was ‘tagging mooch’; after the second user tagged it, it became ‘tagging mooch fordrew’; and so on. I then sliced this data two ways — I first truncated the list to the top N tags, and then collapsed lines that were identical, so I could tell when a stable set of those N tags arose, and how long it lasted.

The current top 3 tags for OIO are ‘ontology tags folksonomy’ — these three tags were the top 3 after about 20% of the users tagged the piece. However, it took only 10 users (not quite 1.5% of the current total) before the top 3 tags were ‘tagging ontology folksonomy’, conveying much the same sense, with only the use of ‘tagging’ instead of ‘tags’ making this different from the current set of 3.

The current top 8 tags are: ontology tags folksonomy tagging classification del.icio.us shirky web. Interestingly, below the top 8, the list has never stabilized, with the 9th and 10th terms being, variously, taxonomy, metadata, toread, article, categories, and categorization, even after the top 8 became set.

I’m setting the script loose on some other frequently tagged links, to test the following hypotheses:

Popular tags get set quickly, but not in stone

It only took 10 users for ‘ontology tags folksonomy’ to sort to the top of the tag list, meaning that even a small group of users can pretty quickly create much of the consensus value around a given link. This is in keeping with the idea of lowest common denominator tagging. However, though this consensus was established quickly, it was not frozen, with the positions among those three words varying, and with tags eventually replacing tagging.

I’ve only found one popular link so far that violates this idea, for the original Adaptive Path piece on Ajax. For this link, the tag ‘ajax’ is overwhelmingly #1, with 1171 occurrences from 2352 taggers. (Second place is ‘javascript’, with a mere 644 tags.) Yet over 800 people, more than a third of the total, tagged it before ‘ajax’ hit the #1 spot — it’s as if you can see Ajax becoming a real term as enough people read the article. The Ajax article may be a one off, or there may be some small but instructive number of links whose consensus view changes slowly, documenting the rise of some new concept.

Beneath a fairly high threshold, tags remain in flux

This is not in keeping with fears of the lowest common denominator — the deeper into the tag list you go, the less stable the tags and order are, suggesting that groups, even large groups,have simple consensus views but highly varied overall views of a particular link. More importantly, the larger the group, the larger that variability becomes. As you’d expect with this sort of distribution, the top few tags get ever more popular, even as the tail gets longer. Scale, and even scale with strong consensus on a few tags, are not in fact incompatibile with variety — in fact, in this situation, scale supports variety.

And this is why the ‘majoritarian tyranny’ argument fails — the relevant unit of opinion is not the user, but the tag, and the variety of tags grows with the number of users. Tagging isn’t voting, in other words, with each user committed to one and only one choice, and the views people share widely are less numerous, considered as unique occurrences, than the views they share with only a few other people, or with no one else at all. Tagging isn’t like getting a group to go to a restaurant, in other words, because there’s no requirement for the users to converge on a single opinion.

In the same way cities offer more varied experiences to their citizens as they get larger, the popularity of a link increases the tag variability, as well as increasing the likelihood that any tag you’d use will have been added by someone else already. Provided a tagging system is mainly for personal value, with social value as a seocond-order benefit (as del.icio.us is), then scale increases varibility and reduces the constraints of consensus.

Readers are good at finding implicit themes

A number of readers tagged the piece with tags relating to information architecture or the Semantic Web, which are only noteworthy because I intentionally never used those phrases in the piece. The piece is an argument about information architecture and about the Semantic Web, but only by extension, since the idea of predictive classification is core to both of those efforts, not because I took on (or even mentioned) the particular efforts in either of those fields.

This is another answer to Tim Bray’s question: Taggers are good at characterizing material in ways that search engines are incapable of, and tags are thus good for letting you find material whose characterization does not appear in the text itself.

You can see subcultures cycle through the tag lists

During a period of about 120 users’ additions of OIO, 20 or of them used the tag ‘ia’, putting it between #7 and #10 during that period. Now it is down to #17. This suggests that one or a few IA-oriented sites or mailing lists posted the link, and it got a flurry of attention from those taggers in a narrower window of time. This in turn suggests a conversationally tightly-knit IA community.

Something similar happened with variations on ’socialsoftware’ and my name. Since the first notices that the piece was out were the shirky.com RSS feed and Many-to-Many, people using either social software or my as a memory of the source of the information were disproportinately represented early on. As the piece started to get pointed to fromelsewhere, those effects faded.

All of these are just hunches, of course, based on looking at a few well-trafficked links. Still to come: more data, and looking at tag growth for links tagged just a few times. But already, the time-slice view is exposing a degree of dynamism in tagging behavior not obvious from looking at representations of the current state of any given set of tags.

This spring, I gave a pair of talks on opposite coasts on the subject of categorization and tagging. The first was entitled Ontology Is Overrated, given at the O’Reilly ETech conference in March. Then, in April I gave a talk at IMCExpo called “Folksonomies & Tags: The rise of user-developed classification.”

I’ve just put up an edited concatenation of those two talks, coupled with invaluable editorial suggestions from Alicia Cervini. It’s called Ontology is Overrated — Categories, Links, and Tags. Here’s the intro:

Today I want to talk about categorization, and I want to convince you that a lot of what we think we know about categorization is wrong. In particular, I want to convince you that many of the ways we’re attempting to apply categorization to the electronic world are actually a bad fit, because we’ve adopted habits of mind that are left over from earlier strategies.

I also want to convince you that what we’re seeing when we see the Web is actually a radical break with previous categorization strategies, rather than an extension of them. The second part of the talk is more speculative, because it is often the case that old systems get broken before people know what’s going to take their place. (Anyone watching the music industry can see this at work today.) That’s what I think is happening with categorization.

What I think is coming instead are much more organic ways of organizing information than our current categorization schemes allow, based on two units — the link, which can point to anything, and the tag, which is a way of attaching labels to links. The strategy of tagging — free-form labeling, without regard to categorical constraints — seems like a recipe for disaster, but as the Web has shown us, you can extract a surprising amount of value from big messy data sets.

More at Ontology is Overrated — Categories, Links, and Tags.

Ken Norton nailed something critical in The web is full of tags

[T]agging isn’t new; the web is full of tags. But they’re not in meta keywords, they’re in the links. The text of the links pointing to other web pages are simply the web publisher’s best effort to describe the page she’s linking to. And it turns out those links are some of the most valuable metadata we have to work with in search. And you know what? They’re subject to all of the flaws people say will doom tagging. Spammers lie. The spelling is atrocious. And there’s ambiguity everywhere. But given a huge population of links, you can begin to make sense of the madness. Why? Well, there are humans on both ends of the search rope. There’s a person searching, and there’s a person who’s written some content. The job of the search engine is to simply connect the two. Traditional software engineers, in their endless pursuit of the elimination of ambiguity, sometimes forget this. Search engineers embraced it.

There are humans at both ends of the rope. It seems so simple, but technologies that can rely on this fact have a huge advantage, since the human brain is terrific at signal extraction in environments that consistenly defeat machine strategies. This is one of the reasons that semweb-flavored approaches to metadata attempt to express data in an unambiguous format — if there is a machine at the other end of the rope, even slight ambiguities defeat the recipient’s interpretive capabilities. As the man said, time flies like an arrow, but fruit flies like a banana.

Once you have humans at both ends of the rope, though, even purely contextual tags that are unextractable from the tagged content itself, tags like cool and toread, become valuable. This is why attempts to improve tagging by making it less ambiguous are missing the point — the ambiguity allows for a huge reduction of both markup cost and conceptual brittleness, by involving human brains as the final endpoints.

Jeffrey Zeldman’s recent post on tag clouds has gotten some play elsewhere, but not here so I thought I’d mention it. It’s not quite as spot-on as his usual commentary, but it raises an interesting question about tag clouds: are they making good use of Fitts’ Law or just reinforcing herd behaviour?

The idea behind tag clouds is that users know best. Their actions determine how other users navigate. Their choices leave a trail. Typically, though not always, the “important” topics get big while those considered less important (which in this case only means less popular) get small. Once they get small enough, they disappear.

In Flickr and Technorati, users create their own tags (“design,” “cats,” “California”). When enough people have used the same tag, it begins to show up in the cloud. Once a lot of people have used it, it becomes a visually dominant element, encouraging others to click it — and subtly discouraging them from creating their own tags.

Beyond the cloud there are many interesting ways we could visualize the, um, tag-geist if you will. What about tag abandonment, or clustering by social group, or unique users, or changes in frequency? Obviously there’s lots of room for innovation here.

Aside: the del.icio.us shades-of-red technique gets my vote as the best popularity interface. It’s subtle, effective and actually useful.

Ken Norton addresses in some detail a common UI problem with tags - how to handle delimiters, i.e., how to allow multiple tags without forcing people to mash terms together like “sanfrancisco”.

There’s a bit of an irony here. The openness of interfaces like del.icio.us (where you’re just given a text input box and the unclear clue of “space separated) has lead directly to tag uptake. But that uptake is still limited, and, I would argue, limited by that openness–there’s a large part of the user population that, when faced with an empty text box and little help, are daunted, and enter nothing.

There are many problems to be addressed in tagging UIs. The challenge is not getting in the way (tagging works because of its low “cost”), while providing enough cues and structure so that people feel confident in using it.

My name’s Gene Smith. Like Peter, I have one foot in the world of information architecture.

Aside from helping spread the folksonomy meme, I coordinated and moderated the folksonomies panel at this year’s Information Architecture Summit. Lately I’ve been frustrated by the unbridled enthusiasm surrounding tagging, so I wrote a piece comparing it to the New Economy enthusiasm of the not-too-distant past.

But my main interest in tagging these days is practical. I helped a client build a folksonomy application for their intranet, which generated a flood of useful metadata along with a lot of metacrap. Even though they found tags useful on the whole, they also found some aspects of it confusing (the lack of accuracy and authority, in particular, were a concern). So the big issues for me are joining tags with other metadata systems and deepening our knowledge of how to make sense of tags and tagging behaviour. On that last point, I wonder if everyone’s seen Ericka Menchen survey of del.icio.us users?

Peter Merholz here. I am an “information architect,” which, I think, makes me part of the “well-designed metadata crowd,” though if Clay saw the metadata we produce, he might think again before saying “well-designed.”

I’ve published various and sundry on folksonomies/tags.

For my job-job, I wrote “Metadata for the Masses,” an essay on the value of tagging, and how it can be intelligently integrated with more formal approaches to classification and categorization. I think perhaps the most overlooked, yet potentially rewarding, opportunity in tagging is to meld them with the good, formal work that people have done before. There’s no reason they can’t exist together.

Later, I wrote, “Tag Inversion - When Metadata Isn’t,” which discusses how while tags originated as metadata, to describe something after the fact, we’re now getting tags first and the “data” later.

In terms of what others have written, I find Adam Mathes’ Folksonomies - Cooperative Classification and Communication Through Shared Metadata to be among the best. Go read it.

The real zinger for me was realizing that tagging or folksonomy is yet another manifestation of our evolution from hierarchical systems to more later, emergent, and empowering network/grassroots approaches. Here we’re talking about a populist approach to taxonomy: rather than fit our thinking into authoritative closed classification schemes, we can create our own through tagging, and in social tagging environments we can negotiate new, more nuanced ways to map meaning and relationship through shared, emergent classification systems.

Louis Rosenfeld makes an analysis about folksonomies which, according to Clay, lacks economic sense.

The advantage of folksonomies isn’t that they’re better than controlled vocabularies, it’s that they’re better than nothing, because controlled vocabularies are not extensible to the majority of cases where tagging is needed. Building, maintaining, and enforcing a controlled vocabulary is, relative to folksonomies, enormously expensive, both in the development time, and in the cost to the user, especially the amateur user, in using the system.

Indeed. And the Dewey Decimal system and other established hierarchies for organizing information (or reality) won’t be replaced by tags, but through tagging we’re finding new ways to think about classifications and new applications for organizing and sharing knowledge.

It’s odd to be so excited about these little chunks of metadata. The concept of tagging, and the way the concept’s been applied so far, are deceptively simple. On the one hand, I can’t believe we weren’t doing this years ago; on the other hand, I have to admit that I didn’t get the value of tags when I first used del.icio.us. What’s the value of an ephemeral label, I wondered, a category I’ve dreamed up for my own use? I was misunderestimating my ability to build systems of organization that are simple and effective, and I wasn’t thinking about the value of “gardening,” as we do with wikis where architecture is not enforced by the technology. I had an aha moment when I realized I could garden my tags - if I create a category that doesn’t ultimately work, I can just edit the items with that tag (usually just one or two, if the problem is applicability), replacing it with something better (by whatever criteria for “better” by tagsonomic thinking du jour defines), and when all references to that tag are removed, the tag disappears from my list.

Tags don’t scale if you’re looking for specific entries, but they’re not really supposed to. I don’t look at a Technorati’s page for a specific tag to find comprehensive knowledge about that category. I’m just looking for the latest blog posts. If I want something more specific, I can use Google or some other flavor of search, there’s jillions of ways I can search for and extract data from the vast universe of Internet resources. If you’re using those tools effectively, and tagging, and blogging, blogrolling, using wikis, posting to forums, using and logging chat spaces, etc. - you can be pretty damn effective. To me social software is about combining tools and approaches and orchestrating their use to fit your methods and quirks.