The new Metadata open frontier
Apple has certainly worked hard in recent OS revisions to enable smarter searching via leveraging metadata. It appears that our reliance on folder hierarchy is going to wane as more efficient and custom methods for tagging.
It's heartening to see developers like Ironic Software , and Gravity Apps embrace tagging. What is needed is a system wide architecture for sharing metadata. I want to tag my files and know that my tags are searchable in other apps. Thankfully Ironic Software is spearheading Open Meta which attempts to leverage the Extended Attributes native to Leopard Even the developer of MailTags is interested(Scott Morrison).
I'm happy that developers are taking steps to get on a common ground for handling tagging via Leopard's xattr support. I have a sneaking suspicion that Apple will probably deliver an Apple solution but you cannot bet on that and it's nice to have Open Meta as a fallback in case Apple fails to deliver. Even if Apple creates a solution I figure Open Meta will easily support the Apple tool preserving developer investment.
Is anyone else here interested in tagging and being able to access that tag information in various applications? I think it could be the basis for a some killer features.
It's heartening to see developers like Ironic Software , and Gravity Apps embrace tagging. What is needed is a system wide architecture for sharing metadata. I want to tag my files and know that my tags are searchable in other apps. Thankfully Ironic Software is spearheading Open Meta which attempts to leverage the Extended Attributes native to Leopard Even the developer of MailTags is interested(Scott Morrison).
I'm happy that developers are taking steps to get on a common ground for handling tagging via Leopard's xattr support. I have a sneaking suspicion that Apple will probably deliver an Apple solution but you cannot bet on that and it's nice to have Open Meta as a fallback in case Apple fails to deliver. Even if Apple creates a solution I figure Open Meta will easily support the Apple tool preserving developer investment.
Is anyone else here interested in tagging and being able to access that tag information in various applications? I think it could be the basis for a some killer features.
Comments
Tagging should not be necessary, as most users (me certainly), just can't be bothered to apply tags by hand. That costs time and effort and renders the whole point of metadata moot (IMHO). Worse, even if users do start to tag (say, their pictures), they often don't finish it/update it, making the results of any process using those tags incomplete and unreliable. I have plenty of files I might tag, my Aperture library (thousands of pics), my Music (70+ Gb), my gmail (thousands of emails), but tagging al that, do it completely and accurately, would take me a month or more of full-time dedication....
I think the future here lies in smarter automatic meta-data gathering (e.g. faces in iPhoto), and, where necessary, making tagging as easy and fun as possible (Faces, Places). Obvious developments that come to mind: a "Faces" feature with movies, improved automatic meta-data completion in iTunes (composer, year, genre, albumart), Voice recognition in iPhone.
In general I believe that enhanced meta-data and context sensitivity/recognition is an essential step in the development of smarter, pattern-recognizing software, which will ultimately to true artificial intelligence. I mean, consider the amount of meta-data any human needs to gather, learn and connect before even considered remotely intelligent
I have plenty of files I might tag, my Aperture library (thousands of pics), my Music (70+ Gb), my gmail (thousands of emails), but tagging al that, do it completely and accurately, would take me a month or more of full-time dedication....
For some of that data, you've already done the tagging by sorting it into folders. It's another form of tagging, just not as flexible. If a file is in ~/Documents/Personal/Financial/CitiBank/Checking/2007, then it is essentially tagged with (Documents) (Personal) (Financial) (CitiBank) (Checking) (2007) already. Either have the folder structure replicated into the tags, or have the tag search also consider the path to the file. That's lot of user-supplied metadata right there.
While I like the possibilities enabled by meta-data, metadata should be a background-thing, automatic where possible, easy and fun when user intervention is necessary. The idea is to have software that is aware of all meta-data in your files, and automatically generate it. e.g. creation date, who sent you this file as an attachment, who did you forward it to 2 days later, where was this photo taken, etc..
Tagging should not be necessary, as most users (me certainly), just can't be bothered to apply tags by hand. That costs time and effort and renders the whole point of metadata moot (IMHO). Worse, even if users do start to tag (say, their pictures), they often don't finish it/update it, making the results of any process using those tags incomplete and unreliable. I have plenty of files I might tag, my Aperture library (thousands of pics), my Music (70+ Gb), my gmail (thousands of emails), but tagging al that, do it completely and accurately, would take me a month or more of full-time dedication....
I think the future here lies in smarter automatic meta-data gathering (e.g. faces in iPhoto), and, where necessary, making tagging as easy and fun as possible (Faces, Places). Obvious developments that come to mind: a "Faces" feature with movies, improved automatic meta-data completion in iTunes (composer, year, genre, albumart), Voice recognition in iPhone.
Agreed on that level. Automatic metadata is certainly something I expect Apple and many others to continue to improve.
However it is impossible for Apple or anyone else to aggregate disparate pieces of data (like all the files in a typical project) under the umbrella of a custom piece of metadata. For instance my mother needs to piece together so many bits of data in her firm to support her case there is no way this can be done automatically without a system being setup in advance. I want the basics handled automatically with a nice interface for encapsulating many differing pieces of data under some tailored tagging.
Although I have the slight vestiges of OCD I personally have refrained from going "tagging mad" in Leopard. There are some items that tagging becomes paramount and other items that don't really matter.
The real power is when your custom tags are exposed system wide. When you're in a 3rd party app yet you can recall every custom tag you've used in Aperture or any other app. It is here where we'll finally see the folder structure (hard coded metadata ) that Kickaha mentions begin to dissolve. In a future where physical objects are becoming digital and gigabyte drives are becoming terabyte it becomes imperative for computer users to begin to think of how they will create a schema for managing such a huge amount of data.
I hope Apple is working feverishly on creating a mechanism for system wide extended attributes...ok maybe not feverishly.
Open Meta coming to Default Folder
I?m very excited about Ironic Software?s establishment of OpenMeta, a new standard for storage of tag metadata on OS X. Storing spotlight keywords in the Finder/Spotlight comments of files has always been problematic, but up until now, it was the best solution available if you wanted general-purpose access to the tags via Spotlight.
Now OpenMeta uses the metadata capabilities in HFS+ to uniformly store tag information - and provides open source code to make it easy for developers get on board. Ironic?s Deep application uses it, and Gravity Applications? new Tags app is doing it too - you can assign tags to files, email messages, photos - it?s very slick and oh-so-much-better on a technical level - we just have to get more people to adopt it! As always, one of the missing pieces is being able to tag documents as you?re saving them - Default Folder X already supports this using the traditional Spotlight comments, so it makes all the sense in the world for DFX to support OpenMeta.
So in answer to all the emails I?ve been getting - YES, Default Folder X will adopt the OpenMeta standard (while still supporting Spotlight comments too, for those of you that aren?t ready to switch).
I don't have DF yet but when they had the Spotlight Comments features I was today thinking "wouldn't it be cool if they were going to support Open Meta?" and voila!
While I like the possibilities enabled by meta-data, metadata should be a background-thing, automatic where possible, easy and fun when user intervention is necessary. The idea is to have software that is aware of all meta-data in your files, and automatically generate it. e.g. creation date, who sent you this file as an attachment, who did you forward it to 2 days later, where was this photo taken, etc..
Tagging should not be necessary, as most users (me certainly), just can't be bothered to apply tags by hand. That costs time and effort and renders the whole point of metadata moot (IMHO). Worse, even if users do start to tag (say, their pictures), they often don't finish it/update it, making the results of any process using those tags incomplete and unreliable. I have plenty of files I might tag, my Aperture library (thousands of pics), my Music (70+ Gb), my gmail (thousands of emails), but tagging al that, do it completely and accurately, would take me a month or more of full-time dedication....
I think the future here lies in smarter automatic meta-data gathering (e.g. faces in iPhoto), and, where necessary, making tagging as easy and fun as possible (Faces, Places). Obvious developments that come to mind: a "Faces" feature with movies, improved automatic meta-data completion in iTunes (composer, year, genre, albumart), Voice recognition in iPhone.
In general I believe that enhanced meta-data and context sensitivity/recognition is an essential step in the development of smarter, pattern-recognizing software, which will ultimately to true artificial intelligence. I mean, consider the amount of meta-data any human needs to gather, learn and connect before even considered remotely intelligent
Full-text searching is a type of "automatic metadata" but it's far from the final answer. I might save a funny article that nowhere contains the word "humor", which I would like to tag it with. I might have a legal contract that doesn't contain the word "contract", another obvious tag.
Tagging is generally supposed to be "free form," but there are benefits to adding some structure to it, such as by picking from a list of tags. If I forget, and tag half of my contracts with "contract" and the other half with "agreement," my search results won't be complete.
A hierarchical file system in 2009 is unnecessary if legacy systems don't need to connect to it. Even with the moderate tagging being done by the OS, I can find most of my files using Spotlight. Of course, better tagging and a Finder that doesn't have Spotlight tacked onto it as an afterthought would be great. But even the Finder 'rewrite' in Snow Leopard seems to show that Apple is incapable moving file management past the late 80s.
The Finder should make full use of metadata for file management. Hierarchical viewing of the file system should be what is tacked on to it as an afterthought. Not the other way around.
The "Search for" options in the Finder sidebar is a good start. Creating Smart Folders should be easier though. Stop fuckin' around Apple.
I understand that a lot of people still love micromanaging their files -- creating folders for them, moving them around (and asking Apple to implement cut and paste), giving their files a descriptive name -- but that is not what a computer is about. Computers are about removing micromanagement. I don't give my text documents (.txt, PDF, Word, Pages) descriptive names because I know I'll find them by typing a few keywords into Spotlight.
I fully agree with the idea that tagging should predominantly be the OS/Apps' job. The option to let the user tag files should exist but should be, for the most part, unnecessary.
iPhoto 09 is a step in the right direction for easier tagging of photos (places and faces)...however, the new tags don't even show up in Spotlight. I can't search for a face or a place and have the photos show up in Spotlight.
What is it going to take? A brand new OS built from the ground up? Why isn't Apple on the ball with this?
Full-text searching is a type of "automatic metadata" but it's far from the final answer. I might save a funny article that nowhere contains the word "humor", which I would like to tag it with. I might have a legal contract that doesn't contain the word "contract", another obvious tag.
Tagging is generally supposed to be "free form," but there are benefits to adding some structure to it, such as by picking from a list of tags. If I forget, and tag half of my contracts with "contract" and the other half with "agreement," my search results won't be complete.
Yes currently there's a laundry list of basic metadata info that may or may not be in the file depending on if it's a photo, PDF or other type of document. We certainly need a way appending our own unique metadata that Spotlight can search and we're almost there with a 3rd party solution. Though to really exercise the benefit we really need Apple to add their touch and expertise to this area in the form of API support in a future version of OS X.
I don't know what's taking Apple so long to fully leverage metadata. I don't know if it's the inept Finder team holding Mac OS X back or if it's just Apple and Steve Jobs that are still stuck with a 80s vision of file management.
A hierarchical file system in 2009 is unnecessary if legacy systems don't need to connect to it. Even with the moderate tagging being done by the OS, I can find most of my files using Spotlight. Of course, better tagging and a Finder that doesn't have Spotlight tacked onto it as an afterthought would be great. But even the Finder 'rewrite' in Snow Leopard seems to show that Apple is incapable moving file management past the late 80s.
The Finder should make full use of metadata for file management. Hierarchical viewing of the file system should be what is tacked on to it as an afterthought. Not the other way around.
The "Search for" options in the Finder sidebar is a good start. Creating Smart Folders should be easier though. Stop fuckin' around Apple.
kim kap sol. I have to be thinking that Apple is onto this and is just playing its cards close to the vest. Tom Andersen from Ironic Software has contacted a Matt Drance from Apple regarding custom Metadata and he's filed a case with the ADC to inquire about Apple's thoughts on OpenMeta hitching a ride on the xattr (extended attributes) express. My hope is that Apple sees a viable market and use for this and creates something that is usable and powerful. I do like the tagging in iPhoto because it easily highlights to you what tags are enabled for a particular photo and what tags are not.
I understand that a lot of people still love micromanaging their files -- creating folders for them, moving them around (and asking Apple to implement cut and paste), giving their files a descriptive name -- but that is not what a computer is about. Computers are about removing micromanagement. I don't give my text documents (.txt, PDF, Word, Pages) descriptive names because I know I'll find them by typing a few keywords into Spotlight.
I fully agree with the idea that tagging should predominantly be the OS/Apps' job. The option to let the user tag files should exist but should be, for the most part, unnecessary.
iPhoto 09 is a step in the right direction for easier tagging of photos (places and faces)...however, the new tags don't even show up in Spotlight. I can't search for a face or a place and have the photos show up in Spotlight.
What is it going to take? A brand new OS built from the ground up? Why isn't Apple on the ball with this?
Bulleye! I myself thought with Leopard that I'd be able to go to a more flat file structure and easly use smart folders to pull out the data I want but the problem that I found was that the automatically generated metadata gets me close. It puts me within a few feet of the needle in that haystack but I'm finding that with a decent tagging system my Spotlight searches now put me within a few inches of the needle. Smart folders become a lot more accurate when searching for custom metadata keywords.
Well Jon Gotow from St Clair Software is in and he wrote the most well reasoned argument for taking risk to implement OpenMeta.
Jon's reasons for supporting OpenMeta
Conclusion:
I initially had reservations about the way OpenMeta stored tag data. However, given Apple's prevalent use of the com.apple.metadata labelling to store searchable data, the fact that xattrs provide a well-defined and atomic storage mechanism, and the existence of a backup system built into OpenMeta, I think it's a reasonable and relatively safe mechanism.
Note that I'm saying "relatively safe." At it's core OpenMeta is still storing data using a name that's reserved for Apple, and is counting on Spotlight indexing that data, a capability that Apple could change or remove in the future.
While OpenMeta is a practical solution to a problem we have now, this should be something that evolves with additional support from Apple. To that end, Tom has filed a bug report requesting support for tags in the future, has contacted Apple via email to start a dialog, and has opened an incident with Apple DTS (Developer Technical Support) to get clarification from Apple on the best way to proceed.
OpenMeta will continue to evolve based on what Apple does and says in the future, but it's a good choice for the present, and something that I feel we can continue to work with in the long run. I plan on completing Default Folder X's support for OpenMeta and releasing it shortly.
My emphasis added. Well that's a biggie. I've been waiting to buy Default Folder X and now I'll be buying a license pretty soon. The ability to tag your files right from the open/save dialog box is going to be great. Getting DFX support is going to go a long way towards getting some other developers on board.
I just may be able to go to that flatter folder structure like I dreamt when Apple delivered Leopard. My wish is that 10.7 is the OS that brings OpenMeta like support with that Apple polish. I'll never go back to micromanaging hierarchical files and fretting over "should I group all data types or via some other method" .
HM