AppleInsider › Forums › Software › Mac Software › XGrid?
New Posts  All Forums:Forum Nav:

XGrid?

post #1 of 69
1/5/04 at 3:17pm
Thread Starter 
Everyone seems to have forgotten the XGrid developer mailing list that was leaked. I just now thought of it after noticing the folding in my mail account for the 21 messages I have from or regarding the list. And looking at the dates I signed up 10/12 and my last message was on 10/29. Which isn't quite as long as it seems, but the information was there. Just a thought.
Reply
Reply
post #2 of 69
1/6/04 at 11:43am
Well, it's CONFIRMED.

http://www.apple.com/xserve/cluster

"Xgrid, new software from Apples Advanced Computation Group, is one of many solutions for Mac OS X that make managing a supercomputer as easy as Macintosh."

It's linked on Apple's cluster node page, but the link goes nowhere.

Barto
Self Indulgent Experiments keep me occupied.

rotate zmze pe vizspygmsr minus four
Reply
Self Indulgent Experiments keep me occupied.

rotate zmze pe vizspygmsr minus four
Reply
post #3 of 69
1/6/04 at 11:49am
http://www.apple.com/pr/library/2004/jan/06xgrid.html

Apple® today previewed Xgrid, a computational clustering technology from Apples Advanced Computation Group (ACG). Xgrid helps scientists and others working in compute intensive environments to fully utilize all IT resources, including desktops and servers, by creating a grid enabled virtual IT environment that takes advantage of unused computing capacity to run batch and workload processing. Available as a free beta download today from www.apple.com, Xgrid brings Apples legendary ease-of-use to computational clustering by providing the easiest way to run compute intensive applications, such as the popular gene-sequencing application BLAST, on multiple Macs using Apples Rendezvous networking technology.

Barto
Self Indulgent Experiments keep me occupied.

rotate zmze pe vizspygmsr minus four
Reply
Self Indulgent Experiments keep me occupied.

rotate zmze pe vizspygmsr minus four
Reply
post #4 of 69
1/6/04 at 1:02pm
http://www.apple.com/acg/xgrid/
Reply
Reply
post #5 of 69
1/6/04 at 2:02pm
If it does half of what they say it does then it is the most important thing they released today.

Tie your existing mac computer labs into a supercomputer with a plug and play solution. . .. kick ass. Gotta use that built in Gigabit ethernet for something.

Reply
Reply
post #6 of 69
1/6/04 at 2:37pm
Quote:
Originally posted by Curufinwe
If it does half of what they say it does then it is the most important thing they released today.

Tie your existing mac computer labs into a supercomputer with a plug and play solution. . .. kick ass. Gotta use that built in Gigabit ethernet for something.


I agree. And with standard office apps (of the sort people are likely to be using in labs) leaving all those Velocity Engines just sitting around with nothing to do... spare cycles anyone?
"Time goes, you say? Ah no! Alas, Time stays, we go." -H. A. Dobson
Reply
"Time goes, you say? Ah no! Alas, Time stays, we go." -H. A. Dobson
Reply
post #7 of 69
1/6/04 at 2:50pm
  • Kickaha
  • Really Fast Typing Member
*drool*

We have Macs scattered hither and yon in my dept.

Their cycles will be mine. Oh yes. They will be mine.
My brain is hung like a HORSE!
Reply
My brain is hung like a HORSE!
Reply
post #8 of 69
1/6/04 at 4:10pm
this is huge...
"I'm learning how to meditate, so far so good."
Donald Fagen and Walter Becker
Reply
"I'm learning how to meditate, so far so good."
Donald Fagen and Walter Becker
Reply
post #9 of 69
1/6/04 at 4:24pm
Quote:
Originally posted by blue2kdave
this is huge...

I totally agree.

As I've said before, clustered computing is the way of the future.

Now Apple is pushing it quite hard, (ie: investing a fair amount in it), and it should bear friut soon. m.
Killing me isn't going to help you. It isn't going to do me much good either, is it? -- The Doctor
Reply
Killing me isn't going to help you. It isn't going to do me much good either, is it? -- The Doctor
Reply
post #10 of 69
1/6/04 at 4:38pm
  • Kickaha
  • Really Fast Typing Member
Grabbed the dev kit for it.

*drool*^2.

Oh my.
My brain is hung like a HORSE!
Reply
My brain is hung like a HORSE!
Reply
post #11 of 69
1/6/04 at 5:20pm
Thread Starter 
Xgrid + Xserve = LOTS of Xserve sales to come. Once Xgrid goes final anyway.

I can see Apple machines filling in the rest of the Top 10 fastest (minus EarthSim and Los Alamos) with something like this.

You say you have the dev kit. Do you, or anyone else, know if this is MVAPICH (what was ported for VA Tech)? Or anything else about what kind of standard they are using for this.
Reply
Reply
post #12 of 69
1/6/04 at 6:56pm
  • Kickaha
  • Really Fast Typing Member
XML layered over BEEP, apparently.
My brain is hung like a HORSE!
Reply
My brain is hung like a HORSE!
Reply
post #13 of 69
1/6/04 at 6:57pm
What all can be clustering used for. I mean I know it isn't normally used for this, and people probably won't program this in, but could encoding DVD's be helped by clustering? I wish they made terragen capable for clustering with this. I am just curious what limits things from being used with clustering. I know like safari and other apps wouldn't even be helped because everything is instant, but can anything that takes several minutes to process be helped by clustering?
0 People Found This Reply Helpful
Reply
0 People Found This Reply Helpful
Reply
post #14 of 69
1/6/04 at 7:05pm
Forgive my simplistic question but can any app utilize this?
Reply
Reply
post #15 of 69
1/6/04 at 7:13pm
  • Kickaha
  • Really Fast Typing Member
Apps have to have jobs that are: decomposable, require little communication between the chunks being processed. Additionally, the apps need to be able to communicate back results.

If you download the developer kit, you'll find that you write an Xgrid plugin that tells interfaces between Xgrid and your app. You then launch the batch jobs from within Xgrid, and it handles the communications between the copies of the apps on various machines.

So something like BLAST is *perfect*, (and it's gonna make my research SCREAM ) but other apps (like iDVD encoding) will have to be rewritten to take advantage of it.

Of course, if an app was designed to be externally automatable (*cough* AppleScript *cough*), then it should be a lot easier.
My brain is hung like a HORSE!
Reply
My brain is hung like a HORSE!
Reply
post #16 of 69
1/6/04 at 8:00pm
Video rendering will benefit greatly from clustering. That is one common application that I can think of. Also, compiling those bloated applications from openoffice.org should be a cinch.

The success of the xgrid technology really depends on what developer tools are going to be available to the end user. Apple would do well to release the developer tool source to GNU. gcc 4.0 with a --cluster switch would be sweet (I can dream, can't I?)

P>s>: All your CPU cycles belong to us!
Most of us employ the Internet not to seek the best information, but rather to select information that confirms our prejudices. - Nicholas D. Kristof
Reply
Most of us employ the Internet not to seek the best information, but rather to select information that confirms our prejudices. - Nicholas D. Kristof
Reply
post #17 of 69
1/6/04 at 8:03pm
  • Kickaha
  • Really Fast Typing Member
Xcode already does this now.
My brain is hung like a HORSE!
Reply
My brain is hung like a HORSE!
Reply
post #18 of 69
1/6/04 at 8:21pm
Ok so I want XGrid to be part of the OS and automatically check if the processing can be clustered and do it for you without any setup.

Would this be possible or aren't apps that dependent on the OS that the OS could do that?
0 People Found This Reply Helpful
Reply
0 People Found This Reply Helpful
Reply
post #19 of 69
1/6/04 at 8:24pm
  • Kickaha
  • Really Fast Typing Member
Simple. Each app in it's description of itself to the OS includes a bit about Xgriddable services and jobs, and how to break them up. Then, when the app hits a job it can break up into pieces, it asks Xgrid.framework (or whatever) to do so. The rest is taken care of.

As to what it can do *now*... I'm looking into it.
My brain is hung like a HORSE!
Reply
My brain is hung like a HORSE!
Reply
post #20 of 69
1/6/04 at 10:13pm
Here's what I find a little odd: Why the screensaver approach?

OS X seems to handle 'nice' processes with little performance lag. When I used to run the distributed.net clients, I never noticed a substantial impact on everyday, non-demanding performance. Especially since anyone who is running Xgrid of random, widely distributed desktop machines had better be doing something with low input/output needs. I would think this 'always on' approach would offer better cluster performance, because you would never waste processor time waiting 15 minutes for the screensaver to come on (that the default).

Xgrid also has a 'dedicated' mode, which is always on (doesn't wait for the screensaver). But now I wonder what the performance impact of that would be on the local user (again doing non-demanding tasks, with Xgrid running a low input/output calculation)? Is Xgrid not "niced" in this mode?
"Time goes, you say? Ah no! Alas, Time stays, we go." -H. A. Dobson
Reply
"Time goes, you say? Ah no! Alas, Time stays, we go." -H. A. Dobson
Reply
post #21 of 69
1/7/04 at 7:18am
Thread Starter 
From the look of the System Preferences Panel, you have 2 options currently:[list=1][*]Only when the computer is idle[*]Always[/list=1] Then you can choose to "Suspend tasks when computer is not idle"

The way it is described and at least how it should work, it will use unused processor cycles, just like Distributed.net does.

This isn't exactly what I had pictured. This is a fast and easy cluster setup, and not really a grid. I think of a grid as completely dynamic, without dedicated Xgrid Controller. Though it does seem as if you can be both client, controller, and agent.

Clients submit jobs, controllers dispatch jobs, and agents execute jobs. Have only been able to meddle on my PowerBook, though I'm going to try to get into the lab and put it on the two machines that meet the system requirements.

I think it'll take a little while for some more information to trickle down about what exactly is going on. I printed out the header files for the BEEP and GridPlugs frameworks, and am going to look through the Xgrid app bundle. And of course read all the included documentation.

Oh yeah, Zilla, the NeXT name for this was almost exactly like this (screen saver mode, and options). I wonder why it took them as long as it did? Though I guess they did all minds of G5 and AltiVec stuff, and added Rendezvous. But still seems like a long time for something that is almost exactly the same as Zilla.
Reply
Reply
post #22 of 69
1/7/04 at 11:25pm
Huh. This software would be the most important thing that came out of Apple if it was integrated into the OS at a core level. 10.4 maybe?

However, what could this be used as far as NOW is concerned? I'm the IT guy for a video production company. Could this be used to offload projects that require heavy processing?
Reply
Reply
post #23 of 69
1/7/04 at 11:55pm
  • Kickaha
  • Really Fast Typing Member
Oh.

My.

God.



G4 PowerBook 1.2GHz, 512MB RAM. Nothing to sneeze at. First stage of Mandelbrot demo in local mode (local CPU only): 43.074 sec.

Add on a Dual 1.4GHz G4 and a Dual 2.0GHz G5, 1GB RAM each.

9.9533 sec.

Holy.

Crap.

~4.4x faster.

I'm in geek heaven.
My brain is hung like a HORSE!
Reply
My brain is hung like a HORSE!
Reply
post #24 of 69
1/8/04 at 12:08am
Quote:
Originally posted by Kickaha

Holy.

Crap.

~4.4x faster.

I'm in geek heaven.

The computer Chess and GO geeks are going to be all over this. Just you wait and see.
"Time goes, you say? Ah no! Alas, Time stays, we go." -H. A. Dobson
Reply
"Time goes, you say? Ah no! Alas, Time stays, we go." -H. A. Dobson
Reply
post #25 of 69
1/8/04 at 11:21am
Quote:
Originally posted by jginsbu
The computer Chess and GO geeks are going to be all over this. Just you wait and see.

Yep, you could put each legal move on a node. You'd prolly see linear scaling with this. Good call.
Electric Gypsy
Reply
Electric Gypsy
Reply
post #26 of 69
1/8/04 at 11:49am
Thread Starter 
Does anyone have more info?

Like:
How do you write the plug-ins?
What is coming? (By the amazingly poor administration features there's lots of stuff missing) You can't do anything.

I assume Final or last beta will be available for WWDC so the Devs can work and get seminars. Then, come September, we'll be swimming in 10.4 Xgrid goodness.

As of yet, I have been unable to read any of the archives for the Xgrid list or re-sign up.
Reply
Reply
post #27 of 69
1/8/04 at 12:03pm
  • Kickaha
  • Really Fast Typing Member
Quote:
Originally posted by macserverX
Does anyone have more info?

Like:
How do you write the plug-ins?
What is coming? (By the amazingly poor administration features there's lots of stuff missing) You can't do anything.

Huh? What do you mean by that?

Read the documentation that comes with Xgrid, as well as the sample plug-in Xcode project. It's all in there.
My brain is hung like a HORSE!
Reply
My brain is hung like a HORSE!
Reply
post #28 of 69
1/8/04 at 12:31pm
  • HOM
  • Awesome Member
It comes with a MHz tachometer *drool*
CARTHAGO DELENDA EST
Reply
CARTHAGO DELENDA EST
Reply
post #29 of 69
1/8/04 at 7:01pm
Thread Starter 
You can't even run a controller without being logged in. Which is useless when you have user switching and security concerns. I did read the docs. I just need information, and I'm just venting at the lack of it. I'm getting into the Sample Plug-In doc tomorrow.

It is admittedly a Preview product, but it would be nice to be able to fine tune the system. Like all the Docs say that a properly configured cluster of M machines will get the power of the combined Mhz of M machines. I'm Rendezvous is nice, but I can't imagine that it optimizes anything. So how do you optimize it?

One thing I did note while testing, was that the machine I was using as an Agent, with the screen saver activated, would consistently not exist. Since the graphic updated every 10 sec or something it would sometimes be at 0 for extended periods of time, while on my computer right beside it, the needle would continually jump from 1.25 to 2.25. I guess that's something to submit to Apple about.

To HOM, if you look in the Resources for Xgrid, there are additional "10x", "100x", "1000x" graphics as multipliers for the shown speed. I cannot imagine. So Big Mac would be at 2200 * 2 = 4.4 1000x.
Reply
Reply
post #30 of 69
1/8/04 at 7:07pm
  • Kickaha
  • Really Fast Typing Member
Quote:
Originally posted by macserverX
You can't even run a controller without being logged in. Which is useless when you have user switching and security concerns. I did read the docs. I just need information, and I'm just venting at the lack of it. I'm getting into the Sample Plug-In doc tomorrow.

Are you *sure* about that? I have two towers sitting in the lab, running their own controllers and agents, and popping up Xgrid here in my office shows all three as possible controllers to connect to... and those two lab towers are showing the login panel.

Quote:
It is admittedly a Preview product, but it would be nice to be able to fine tune the system. Like all the Docs say that a properly configured cluster of M machines will get the power of the combined Mhz of M machines. I'm Rendezvous is nice, but I can't imagine that it optimizes anything. So how do you optimize it?

Er... unsure how you would want to optimize it beyond what Xgrid already does, finding services, figuring compute power, load levels, etc, and then dispatching jobs appropriately. What did you have in mind? Rendevouz only finds the machines in the first place, and has nothing to do with it after that.

Quote:
One thing I did note while testing, was that the machine I was using as an Agent, with the screen saver activated, would consistently not exist. Since the graphic updated every 10 sec or something it would sometimes be at 0 for extended periods of time, while on my computer right beside it, the needle would continually jump from 1.25 to 2.25. I guess that's something to submit to Apple about.

I gave up on getting anything meaningful from the gauge. On my three-node system, the tach consistently was barely bobbing above 0 since it was sending out jobs as fast as it was getting results. The smoooooooooth needle movement was actually too slow. Digital readout might be more useful for burst-style batch jobs. The tach may be great for massive ongoing computations though. Definitely give feedback however.
My brain is hung like a HORSE!
Reply
My brain is hung like a HORSE!
Reply
post #31 of 69
1/9/04 at 1:43pm
Thread Starter 
The Xgrid mailing list is now running.

I was able to get Xgrid installed on the other compatible system, though I missed something with the password so it was show as offline until I could get back to it. As I remember, there was no one logged into the Controller computer, so maybe it just wasn't happy before, cuz I haven't changed anything on the system.

The plug-in architecture is simple, which makes it great and for command line scientific tools, it's more than enough.

There are other applications, that would be better suited for a naming scheme different than what is specified "command_args...."

Actually, how would this work outside of a plug-in, let's say, in Photoshop for a large render. Or even better, Final Cut Pro/Express or Compressor.
Reply
Reply
post #32 of 69
1/12/04 at 5:40pm
Just curious...how easy is this to implement into programs? I mean is it as easy as tying in rendezvous? It sure would be nice next when when I'm doing rendering or working on big apps if I could have my powebook connected to my G5 and get that extra processing power easily.
0 People Found This Reply Helpful
Reply
0 People Found This Reply Helpful
Reply
post #33 of 69
1/12/04 at 5:42pm
  • Kickaha
  • Really Fast Typing Member
Check the dev kit. There's a sample plugin that tells Xgrid how to hook into your app. Not too difficult looking, really.
My brain is hung like a HORSE!
Reply
My brain is hung like a HORSE!
Reply
post #34 of 69
1/12/04 at 6:21pm
Quote:
Originally posted by Kickaha
Check the dev kit. There's a sample plugin that tells Xgrid how to hook into your app. Not too difficult looking, really.

This == Extent of my Cocoa Knowledge


But I'll download and try to check it out.
0 People Found This Reply Helpful
Reply
0 People Found This Reply Helpful
Reply
post #35 of 69
1/12/04 at 6:55pm
Ok well I'm sure wireless isn't the best thing to do it over, but the 733Mhz QS I was running the controller on, it's tachometer didn't go above the 1.25Ghz mark my machine runs at.
0 People Found This Reply Helpful
Reply
0 People Found This Reply Helpful
Reply
post #36 of 69
1/12/04 at 7:04pm
  • Kickaha
  • Really Fast Typing Member
One thing to keep in mind - that tach is a smooth animation, so it takes time to 'get up to speed' then back down when jobs go out, and then get completed. Mine was bouncing around 1.4GHz on a 1.25GHz machine, but the speed was almost 5x faster. On the *right* task, it pegged at 5GHz easily. The problem was that the tach would get the 'go' signal, then get the 'done' signal so quickly that it couldn't get the needle up in time. :P Watch the clock, not the tach.
My brain is hung like a HORSE!
Reply
My brain is hung like a HORSE!
Reply
post #37 of 69
1/12/04 at 8:01pm
Quote:
Originally posted by ast3r3x
This == Extent of my Cocoa Knowledge

Hey, then you know C too! (and PERL, and...)
"I'm learning how to meditate, so far so good."
Donald Fagen and Walter Becker
Reply
"I'm learning how to meditate, so far so good."
Donald Fagen and Walter Becker
Reply
post #38 of 69
1/12/04 at 8:21pm
Quote:
Originally posted by blue2kdave
Hey, then you know C too! (and PERL, and...)

haha c++ and PHP
0 People Found This Reply Helpful
Reply
0 People Found This Reply Helpful
Reply
post #39 of 69
1/14/04 at 6:52am
So is everyone in agreement that this could eventually find it's way into the system level of the OS, and therefore utilize multiple machines for any and all processor intensive tasks? Think about the possibilities, and think about the longevity this could give to old machines that are considered too slow.
I too, have the ability to kill a yak at 200 yards... with mind bullets!
Reply
I too, have the ability to kill a yak at 200 yards... with mind bullets!
Reply
post #40 of 69
1/14/04 at 7:14am
  • Kickaha
  • Really Fast Typing Member
'any and all'? Nope. That's what a couple of us have been trying to explain here.

Imagine you have a system of n gravitational bodies to solve. Each body interacts with all the other bodies. The calculations aren't involved, and pretty simple to conduct on each body once you have all the data.

Good candidate for Xgrid, right?

Wrong.

The calculation is quick and easy, but since all entities need to know about all the other entities, you have to transmit each result to *all* the other machines on *every* step.

Your quick calculation is now waiting on the network. D'oh. Very slow.

And that's for a relatively modest amount of data being transmitted. Imagine the problems with those that need a *lot* of data being passed around. :/

No, this is a technology for specific situations, but in those situations, it rocks.
My brain is hung like a HORSE!
Reply
My brain is hung like a HORSE!
Reply
New Posts  All Forums:Forum Nav:
  Return Home
  Back to Forum: Mac Software
AppleInsider › Forums › Software › Mac Software › XGrid?