Transparent Distributed Processing
I thought the demo of Xcode's ability to transparently (and *efficiently*) distrbute the task of compiling over a network might have been the most astounding announcement of the keynote. Right up there with the G5. But it didn't seem like the audience really grasped the potential implications. Maybe I'm the slow one, but...
This is the arrival of "the network is the computer". If Apple can write code to efficiently distribute compiling, why not other tasks? Why not build the code to allow it into the compiler (like VE calls) so all apps have at least a rudimentary ability to do it? Why not suggest all developers built it into their products? OSX itself already takes advantage of multiple processors even if none of the individual apps do, (right?) by allowing app threads to be distributed to both processors. Why not do the same thing across the network? It wouldn't improve the snappiness of folder openings, but for any task that requires heavy crunching of relatively small amounts of data "the network is the computer" could give a huge speed boost, especially when running multiple processes. I salivate at the thought of being able to run mySQL on my iBook but offloading the really complicated queries to the two dual-G4's next door.
Or is this all a pipe dream, and there's something special about the task of compiling that makes it uniquely able to be efficiently distributed?
This is the arrival of "the network is the computer". If Apple can write code to efficiently distribute compiling, why not other tasks? Why not build the code to allow it into the compiler (like VE calls) so all apps have at least a rudimentary ability to do it? Why not suggest all developers built it into their products? OSX itself already takes advantage of multiple processors even if none of the individual apps do, (right?) by allowing app threads to be distributed to both processors. Why not do the same thing across the network? It wouldn't improve the snappiness of folder openings, but for any task that requires heavy crunching of relatively small amounts of data "the network is the computer" could give a huge speed boost, especially when running multiple processes. I salivate at the thought of being able to run mySQL on my iBook but offloading the really complicated queries to the two dual-G4's next door.
Or is this all a pipe dream, and there's something special about the task of compiling that makes it uniquely able to be efficiently distributed?
Comments
BTW, there's another thread w/ similar topic:
http://forums.appleinsider.com/showt...threadid=26552
AFA compiling, you can set the (max) number of concurrent threads of a compile job. It is generally wise to set the number of jobs to <number of CPUs> + 1. So for my TiBook, make -j2 is best.
For distcc, <number of CPUs> x2 +1 seems to give the best returns. So for three computers, make -j7 is best.
W/ compiling, you have small files being transferred (code, aka text) over the network; the 'real' work is the mathematically intense act of compiling that code. Then, after compilation, a small binary is then passed back to the computer that sent out the job.