Tag and flatten

I am attending the Web 2.0 Conference this week, so I'm preparing for it by thinking about how it can apply to Puppet. I'm guessing that I'm ignoring a significant portion of what's considered to be a crucial aspect of Web 2.0, because I'm focusing on the immense value-add of tagging objects and then providing simple interfaces for sharing and browsing tags, which I think of as the 'tag and flatten' method. That is, rather than building up specific, static heirarchies based on whatever specific categories or criteria, just tag every object in the heirarchy with whatever details you think matter and then flatten the heirarchy, letting the tags themselves draw patterns out.

A good example of self-discovered patterns are Flickr's tag clusters, which are algorithmically determined groups of related tags -- that is, they are tags that are determined to be related by 1) individuals tagging individual items with whatever they want, and then 2) computers assessing those tags and finding sets of tags that seem "related", via whatever definition the algorithm is using.

How can Puppet take advantage of this? I mentioned it briefly in my letter to the O'Reilly Radar, but it's pretty simple. One way to talk about Puppet's goals is that it is empowering sysadmins to normalize object specifications across an entire network. Given any configurable element on any machine on your network -- file, user, package, cron job, IP address -- that individual element should only be mentioned one time in your configuration, and then every host or host class that needs that element (e.g., to install a package or provision a user) just imports that portion of the specification.

The top-down way of looking at this importing is that it presents a bit of a heirarchy, from server, to server class, to service, to element, except that it's not one heirarchy, it's a myriad of heirarchies, one for every node and every element. Try to draw a map of these relationships in anything resembling a real heirarchy and you will soon need more dimensions than string theory. Tag that same set of heirarchies, though, and tag-and-flatten them, so that each configurable element is tagged with the host names, services, and server classes that mention it, and you are no longer forced into one way of seeing the data. You can draw out whatever structures you think are appropriate, and sufficient tools should be able to draw them out for you, just like Flickr's tag clusters.

If you do this for your whole network, you could go to this tagged-and-flattened list of elements and quickly determine which hosts have a given user on them, or which server classes require Apache. And because you have these tags on the elements themselves, you could subsequently tag those elements' data with the same tags. Centralized Apache logs are great, but they lose a lot of implicit information. Imagine a log centralizer that also tagged each log message with all of the Apache element's tags (localized to the specific host, of course -- that is, the tag list would not include every host name with Apache in it or all of the server classes that use Apache, just the tags related to Apache running locally) -- go to your log database and see all Apache logs related to a specific server class, or a specific server, or network, or whatever you want.

If course, those add-on tags require quite a bit of additional infrastructure -- standard means of referring to an individually configurable element (hah! just try to standardize the definition of an element, I dare you!), plus standard ways of retrieving the tags on them, and then tools that actually do so. I'm guessing that this is not worth it for many purposes or many organizations, but I think it is worth it for some, and I also think it adds a helluvalot more value than is necessarily obvious.

Puppet is not quite ready to support this level of pervasive tagging, at least partially because I've been stupidly focused on making it actually do work, but... maybe this is what I will do in the 36 hours or so I have between now and when the conference starts.