HFSX aims to support "inline attributes", which certainly seems like another way of saying "storing metadata without using full-blown forks." More info can be found in this Ars thread, along with my pessimistic predictions about the chances of seeing anything really neat from Apple on this front. Summary: HFSX looks to be about it for the foreseeable future, and even that's not going to blossom this year in any meaningful way. (Feel free to prove me wrong, Apple ;-)
As for the general question, I think BFS has done it the best so far. I think real-time indexing and notifications are essential. BFS is actually pretty primitive when you get down to it, which is why I have such a hard time getting excited about any new filesystem that doesn't do something that BFS did. I also have a hard time appreciating any file system that supports arbitrarily extensible metadata, but doesn't support indexing. Gee, thanks.
I'd solve scoping problems with namespaces. Just slap on a nice API and it'll do just fine. Scopes would cascade, with more specific scopes overriding more general ones. Permissions get a bit thorny, but there are plenty of workable simplifications (e.g. file permissions determine metadata scope permissions.)
The problem is and has always been one of inertia, power, and will--not technology.
I'm all for more metadata, extensible and indexable as possible. At the same time I'm curious... What are some things that ex-BeOS users used BFS and the BeOS Tracker for -- that the current OS X Finder can't provide ? What specific solutions (to what problems) can an OS w/ a FS with smart, powerful metadata offer, for both users and developers? And would such a system have a consumer-oriented selling point, or will it remain a behind-the-scenes internal overhaul for geeks only?
(Personally speaking, my own need for metadata -- AFIAK -- is fairly modest. I'd like to know which user worked on a file and when, have the ability to add comments that don't disappear across operating systems, and have more access in the Finder to file details, depending on what kind of data it is.)
That question is easy to answer. the things that you have to do in OS X with an app could be done in beos with the tracker ( aka finder ). Email info was all stored as metadata. The id3 tags on mp3's were metadata. People in your address book were metadata.
Cool things like, when you installed an app ( copied its directory to applications ) there was typically a 'person' file in there as well. That automatically ( no importing ) became available to programs that used people files ( like email ).
Where it gets really interesting is that because all that info is stored in metadata, your people file can be something else ( like an application ), or an mp3, and the info about the person moves around with that, it doesnt need to be a seperate file.
A normal method for handling email in BeOS is to have some save searches ( which look like directories ), they pull up unread mail, or mail from a specific person. I never actually organised my email, it just went into my inbox, and I used searches to get different views on that data. You never wanted to open the inbox directory, because displaying 10000 files was slow, but the searches were really fast ( thanks to indexing of meta data ).
One of the underlying concepts of UNIX is having small, specialised programs that you can tie together to do more complex things. Well, metadata lets you do this for gui applications. Many people have talked about have a way to construct gui pipelines, just you do with cli pipelines in UNIX. Nothing ever works. Metadata does.
You dont need a complex music player which can do all sorts of stuff ( like itunes does ), you just need a simple music player, and a playlist generator.
The biggest reason that metadata is slow to take off is backwards compatibility. How do you transfer metadata to and across file system that dont support it? When that data is important ( not redundant ) you cant afford to ignore it.
There are lots of solutions, and none of them are very elegant. When that problem is solved then metadata will take off big time.
The description of named forks makes no mention of hfsx, and it seems to be a feature of hfs+. It is not clear, but previously it was stated that the minimum size for a fork was 1 block ( 512bytes ), which is entirely unsuitable for general use for metadata ( which might only be a few bytes ).
I think this is why we havent seen Apple use HFS+ to support metadata.
Interestingly, there is mention of HFSX later on, stating that the only version so far was to support case sensitive file names. What that section also says is that HFSX has an associated version number, and that any low level tools which encounter an HFSX version number they dont recognise must _not_ perform any operations on the volume. This leaves the gate wide open for Apple to implement a completely new file system that looks a lot like an HFS volume to old software ( eg: might repurpose the named fork capabilities to do better metadata ).
The description of named forks makes no mention of hfsx, and it seems to be a feature of hfs+. It is not clear, but previously it was stated that the minimum size for a fork was 1 block ( 512bytes ), which is entirely unsuitable for general use for metadata ( which might only be a few bytes ).
I think this is why we havent seen Apple use HFS+ to support metadata.
Interestingly, there is mention of HFSX later on, stating that the only version so far was to support case sensitive file names. What that section also says is that HFSX has an associated version number, and that any low level tools which encounter an HFSX version number they dont recognise must _not_ perform any operations on the volume. This leaves the gate wide open for Apple to implement a completely new file system that looks a lot like an HFS volume to old software ( eg: might repurpose the named fork capabilities to do better metadata ).
Is it? 512 bytes is only 256 characters. That's the same length as a file name (one form of metadata).
Where it gets really interesting is that because all that info is stored in metadata, your people file can be something else ( like an application ), or an mp3, and the info about the person moves around with that, it doesnt need to be a seperate file.
That does sound interesting, and quite cool. But wouldn't such technology actually work a bit against Apple's strategy of creating digital hub applications like iLife?
When Apple promotes iLife as "it's like Microsoft Office for the rest of your life," there's something of a sly double meaning connected with that... They'd like to lock users, with their personal data, into iTunes and iPhoto. All the ID3 and EXIF tags help get data into these apps, but once you move the data in, it's not so easy to move out.
It seems like a FS w/ BeOS-like metadata would (a) make their stable of iApps less valuable, and (b) make it easier for users to move away.
I'm not saying that Apple's moves here are necessarily best for the consumer and free choice, BTW, but it is clearly a strategy. So again, I'm curious: is there a commercial plus to developing a such a metadata-savvy FS? What marketable features could it lead to, and how would it connect with Apple's current strategy (or, even potentially, vision)?
Hobbes, I don't agree with that at all. Apple uses proprietary means to store metadata because there is no open means - which is essentially what this thread attempts to address. Apple has been doing everything possible in recent years to make the Mac as open and standards compliant as possible.
Hobbes, I don't agree with that at all. Apple uses proprietary means to store metadata because there is no open means - which is essentially what this thread attempts to address. Apple has been doing everything possible in recent years to make the Mac as open and standards compliant as possible.
I tend to see more of a complicated mix. Mostly, Apple has embraced open and standards-compliant (OS X's underpinnings, ics, mbox, etc.) but reserves the right to lace the OS with proprietary components (Aqua, FairPlay). The flag for open and standard seems to duck down for a moment, when a competitive advantage beckons.
So, what's the advantage for Apple to take the metadata from inside their (heavily promoted) suite of apps and make it systemwide?
And what -- aside from the flexible and liberating ability to move all of one's information from an app's database to a system database -- is the advantage for the consumer? What could such a solution accomplish that Apple isn't offering today with, for example, the Address Book API?
Well, I like the ability for cross talk between different types of files, but i think Apple's current solution is pretty good also. I like being able to quickly make a few edits of my photos, and then post them to the web or make a book just by clicking a button. I wouldn't want to do this with other types of files, however. If apple got rid of iTunes and iPhoto, they would have to implement that stuff, along with iTMS, iPod syncing, and all that other stuff in the finder or system level or something. That would be a very inelegant solution, because some features would be incompatible with certain files, and you wouldn't know off-hand which features work with what
Thus, i give it as my fixed opinion that metadata is very useful (esp. for mail, calender, addressbook), but it cannot replace iTunes and iPhoto.
Thus, i give it as my fixed opinion that metadata is very useful (esp. for mail, calender, addressbook), but it cannot replace iTunes and iPhoto.
You miss the point. itunes and iphoto dont have to dissappear, their functionality doesnt have to get subsumed by the finder ( god forbid, the finder is bad enough already ). Rather, the information that they use becomes available to other applications, in a generic way which isnt bound to itunes or iphoto, or any other passing preference.
The biggest advantage to metadata is that it allows you to get access to your information in a very database-like manner. I'll stand on the backs of giants to explain some of the useful points of metadata for those unfamiliar with it.
For instance: let's say I have a bunch of artwork on my hard drive. Paintings, photographs, sketches, what have you, all in various file formats (PNG, JPEG, GIF, etc.).
Under the normal file system, you have to create a structure to store things in. You could just toss it all in one folder, but that's going to result in a mess of 1000's of random files. So, most folks come up with a heirarchy like:
Type->Genre->Style->Artist->Files
(Which gives you 'Photos->Landscapes->Desert->Ansel Adams->files')
or
Artist->Type->Genre->Style->Files
(Which gives you 'Ansel Adams->Photos->Landscapes->Desert->files')
The first one is good if you primarily look for things by a broad category, and are less interested in who created it. The latter is better for seeing collections of a particular artist, broken down by category.
The problem is that the former results in lots of different 'Ansel Adams' folders on your hard drive, while the latter results in lots of 'Desert' folders on your hard drive. There's overlap that gets divided in a most user-unfriendly manner: you're stuck with one or the other, unless you duplicate files.
With a metadata system, however, those folder groupings become less important. Each of my files would have information like:
Filename: Saguaro.png
File Creation Date: 4/26/2004
Art Date: 1956
Type: Photograph
Genre: Landscape
Style: Desert
Subject: Saguaro cactus with sunset
Artist: Ansel Adams
From there, I can create a search, similar to an iTunes Smart Playlist. This would look like a folder (probably with a spyglass tag icon), and when double-clicked it would go through your files (or a cache of the metadata) to get the results.
In this case, let's say I had a Smart Folder that shows me all my graphics files (anything with PNG, JPEG, etc. file extensions or marked as such in the metadata) which feature a Type of "photograph" a subject of "cactus" by the artist "Ansel Adams". With this Smart Folder, it doesn't really matter where on the hard drive I've put this particular file: any file that matches those criteria will display in the Smart Folder, regardless of their 'physical' location on the computer.
It's much faster to create a quick Smart Folder in this setup than it is to take a bunch of files and try to sort out what kind of folder heirarchy would be best for your work-style. The drawback being that metadata needs two things to be effective: set metadata types for certain files (Word documents don't need "Bitrate" information, for example), with user-created metadata for unique files (a Quake map might have a "Environment" flag so you know if it's a CtF map vs. a Deathmatch map). There's a bit of conflict between the two: who decides what the 'official' metadata for a text file is? And how do you pass unique metadata when the file is copied?
Overall, though, i'd love to see something like this come about.
Comments
As for the general question, I think BFS has done it the best so far. I think real-time indexing and notifications are essential. BFS is actually pretty primitive when you get down to it, which is why I have such a hard time getting excited about any new filesystem that doesn't do something that BFS did. I also have a hard time appreciating any file system that supports arbitrarily extensible metadata, but doesn't support indexing. Gee, thanks.
I'd solve scoping problems with namespaces. Just slap on a nice API and it'll do just fine. Scopes would cascade, with more specific scopes overriding more general ones. Permissions get a bit thorny, but there are plenty of workable simplifications (e.g. file permissions determine metadata scope permissions.)
The problem is and has always been one of inertia, power, and will--not technology.
(Personally speaking, my own need for metadata -- AFIAK -- is fairly modest. I'd like to know which user worked on a file and when, have the ability to add comments that don't disappear across operating systems, and have more access in the Finder to file details, depending on what kind of data it is.)
Cool things like, when you installed an app ( copied its directory to applications ) there was typically a 'person' file in there as well. That automatically ( no importing ) became available to programs that used people files ( like email ).
Where it gets really interesting is that because all that info is stored in metadata, your people file can be something else ( like an application ), or an mp3, and the info about the person moves around with that, it doesnt need to be a seperate file.
A normal method for handling email in BeOS is to have some save searches ( which look like directories ), they pull up unread mail, or mail from a specific person. I never actually organised my email, it just went into my inbox, and I used searches to get different views on that data. You never wanted to open the inbox directory, because displaying 10000 files was slow, but the searches were really fast ( thanks to indexing of meta data ).
One of the underlying concepts of UNIX is having small, specialised programs that you can tie together to do more complex things. Well, metadata lets you do this for gui applications. Many people have talked about have a way to construct gui pipelines, just you do with cli pipelines in UNIX. Nothing ever works. Metadata does.
You dont need a complex music player which can do all sorts of stuff ( like itunes does ), you just need a simple music player, and a playlist generator.
The biggest reason that metadata is slow to take off is backwards compatibility. How do you transfer metadata to and across file system that dont support it? When that data is important ( not redundant ) you cant afford to ignore it.
There are lots of solutions, and none of them are very elegant. When that problem is solved then metadata will take off big time.
http://developer.apple.com/technotes/tn/tn1150.html
Just after the contents it says:
"future support for named forks, and"
The description of named forks makes no mention of hfsx, and it seems to be a feature of hfs+. It is not clear, but previously it was stated that the minimum size for a fork was 1 block ( 512bytes ), which is entirely unsuitable for general use for metadata ( which might only be a few bytes ).
I think this is why we havent seen Apple use HFS+ to support metadata.
Interestingly, there is mention of HFSX later on, stating that the only version so far was to support case sensitive file names. What that section also says is that HFSX has an associated version number, and that any low level tools which encounter an HFSX version number they dont recognise must _not_ perform any operations on the volume. This leaves the gate wide open for Apple to implement a completely new file system that looks a lot like an HFS volume to old software ( eg: might repurpose the named fork capabilities to do better metadata ).
Originally posted by mmmpie
This is what I read about HFS+
http://developer.apple.com/technotes/tn/tn1150.html
Just after the contents it says:
"future support for named forks, and"
The description of named forks makes no mention of hfsx, and it seems to be a feature of hfs+. It is not clear, but previously it was stated that the minimum size for a fork was 1 block ( 512bytes ), which is entirely unsuitable for general use for metadata ( which might only be a few bytes ).
I think this is why we havent seen Apple use HFS+ to support metadata.
Interestingly, there is mention of HFSX later on, stating that the only version so far was to support case sensitive file names. What that section also says is that HFSX has an associated version number, and that any low level tools which encounter an HFSX version number they dont recognise must _not_ perform any operations on the volume. This leaves the gate wide open for Apple to implement a completely new file system that looks a lot like an HFS volume to old software ( eg: might repurpose the named fork capabilities to do better metadata ).
Is it? 512 bytes is only 256 characters. That's the same length as a file name (one form of metadata).
Barto
Originally posted by mmmpie
Where it gets really interesting is that because all that info is stored in metadata, your people file can be something else ( like an application ), or an mp3, and the info about the person moves around with that, it doesnt need to be a seperate file.
That does sound interesting, and quite cool. But wouldn't such technology actually work a bit against Apple's strategy of creating digital hub applications like iLife?
When Apple promotes iLife as "it's like Microsoft Office for the rest of your life," there's something of a sly double meaning connected with that... They'd like to lock users, with their personal data, into iTunes and iPhoto. All the ID3 and EXIF tags help get data into these apps, but once you move the data in, it's not so easy to move out.
It seems like a FS w/ BeOS-like metadata would (a) make their stable of iApps less valuable, and (b) make it easier for users to move away.
I'm not saying that Apple's moves here are necessarily best for the consumer and free choice, BTW, but it is clearly a strategy. So again, I'm curious: is there a commercial plus to developing a such a metadata-savvy FS? What marketable features could it lead to, and how would it connect with Apple's current strategy (or, even potentially, vision)?
Barto
Originally posted by Barto
Hobbes, I don't agree with that at all. Apple uses proprietary means to store metadata because there is no open means - which is essentially what this thread attempts to address. Apple has been doing everything possible in recent years to make the Mac as open and standards compliant as possible.
I tend to see more of a complicated mix. Mostly, Apple has embraced open and standards-compliant (OS X's underpinnings, ics, mbox, etc.) but reserves the right to lace the OS with proprietary components (Aqua, FairPlay). The flag for open and standard seems to duck down for a moment, when a competitive advantage beckons.
So, what's the advantage for Apple to take the metadata from inside their (heavily promoted) suite of apps and make it systemwide?
And what -- aside from the flexible and liberating ability to move all of one's information from an app's database to a system database -- is the advantage for the consumer? What could such a solution accomplish that Apple isn't offering today with, for example, the Address Book API?
Thus, i give it as my fixed opinion that metadata is very useful (esp. for mail, calender, addressbook), but it cannot replace iTunes and iPhoto.
Originally posted by Imergingenious
Thus, i give it as my fixed opinion that metadata is very useful (esp. for mail, calender, addressbook), but it cannot replace iTunes and iPhoto.
You miss the point. itunes and iphoto dont have to dissappear, their functionality doesnt have to get subsumed by the finder ( god forbid, the finder is bad enough already ). Rather, the information that they use becomes available to other applications, in a generic way which isnt bound to itunes or iphoto, or any other passing preference.
For instance: let's say I have a bunch of artwork on my hard drive. Paintings, photographs, sketches, what have you, all in various file formats (PNG, JPEG, GIF, etc.).
Under the normal file system, you have to create a structure to store things in. You could just toss it all in one folder, but that's going to result in a mess of 1000's of random files. So, most folks come up with a heirarchy like:
Type->Genre->Style->Artist->Files
(Which gives you 'Photos->Landscapes->Desert->Ansel Adams->files')
or
Artist->Type->Genre->Style->Files
(Which gives you 'Ansel Adams->Photos->Landscapes->Desert->files')
The first one is good if you primarily look for things by a broad category, and are less interested in who created it. The latter is better for seeing collections of a particular artist, broken down by category.
The problem is that the former results in lots of different 'Ansel Adams' folders on your hard drive, while the latter results in lots of 'Desert' folders on your hard drive. There's overlap that gets divided in a most user-unfriendly manner: you're stuck with one or the other, unless you duplicate files.
With a metadata system, however, those folder groupings become less important. Each of my files would have information like:
Filename: Saguaro.png
File Creation Date: 4/26/2004
Art Date: 1956
Type: Photograph
Genre: Landscape
Style: Desert
Subject: Saguaro cactus with sunset
Artist: Ansel Adams
From there, I can create a search, similar to an iTunes Smart Playlist. This would look like a folder (probably with a spyglass tag icon), and when double-clicked it would go through your files (or a cache of the metadata) to get the results.
In this case, let's say I had a Smart Folder that shows me all my graphics files (anything with PNG, JPEG, etc. file extensions or marked as such in the metadata) which feature a Type of "photograph" a subject of "cactus" by the artist "Ansel Adams". With this Smart Folder, it doesn't really matter where on the hard drive I've put this particular file: any file that matches those criteria will display in the Smart Folder, regardless of their 'physical' location on the computer.
It's much faster to create a quick Smart Folder in this setup than it is to take a bunch of files and try to sort out what kind of folder heirarchy would be best for your work-style. The drawback being that metadata needs two things to be effective: set metadata types for certain files (Word documents don't need "Bitrate" information, for example), with user-created metadata for unique files (a Quake map might have a "Environment" flag so you know if it's a CtF map vs. a Deathmatch map). There's a bit of conflict between the two: who decides what the 'official' metadata for a text file is? And how do you pass unique metadata when the file is copied?
Overall, though, i'd love to see something like this come about.