or Connect
AppleInsider › Forums › Software › Mac Software › XGrid?
New Posts  All Forums:Forum Nav:

XGrid? - Page 2

post #41 of 69
This. Is. Shweet. Finally I can steal my Dad's processing power for my own use.

(Does PS support it yet?)
post #42 of 69
What is a good size of files to tranmit? I would love to be able to do this for encoding DVD's and photoshop, anything that requires more then a min or so of processing. I understand at some point it won't be good because the network is too slow or there is no need for other computers, so what is the size range for data to have usefullness.
0 People Found This Reply Helpful
Reply
0 People Found This Reply Helpful
Reply
post #43 of 69
Quote:
Originally posted by Placebo
This. Is. Shweet. Finally I can steal my Dad's processing power for my own use.

(Does PS support it yet?)

Nope, and not likely to for a while.

Xgrid is a strange beast as it currently stands - you don't fire off an Xgrid job from your application, you fire off your application from Xgrid.

I can see how one *could* do the former, but that's now how it is now.
My brain is hung like a HORSE!
Reply
My brain is hung like a HORSE!
Reply
post #44 of 69
Quote:
Originally posted by ast3r3x
What is a good size of files to tranmit? I would love to be able to do this for encoding DVD's and photoshop, anything that requires more then a min or so of processing. I understand at some point it won't be good because the network is too slow or there is no need for other computers, so what is the size range for data to have usefullness.

*COMPLETELY* depends on your calculation time, file size, and network speed.

I have files that are going to be massive XML dumps, maybe .1-1GB... but they are a one-time transmit, and multiple runs can be made on each file, and the calculation time swamps the network wait. So my system will work with Xgrid, even with big files.
My brain is hung like a HORSE!
Reply
My brain is hung like a HORSE!
Reply
post #45 of 69
Thread Starter 
There is still lots of potential with Xgrid. Apple has made it extendible at least enough so that they think that MPI (or others) capabilities could be added. But you do have Pooch for that.

I'd really like to the see the Mandelbrot code. I can tell what the logic behind it working is, i'd just like to actually see it. There are the specified parameters, X, Y, Width and the plug-in take these values, splits them up into X1, Y1; X2, Y2; etc then sends theses new values to the nodes to be processed. That's what I would think is being done anyway. It's relatively simple for the Client or Controller (not sure which) to calculate the right values for many computers to compute the image very quickly. Little overhead for fast results.

This could be extended to a few apps. Final Cut Pro/Express might get faster renders for a few things, but since it is real-time visualization, anything you want to see is done on your computer, but things like deinterlacing of large files would see massive improvement, on the right network.

Compressor could benefit along with everything that will require large files transfers, only if the performance of splitting the work up and sending it across the network is fast.

Photoshop could quite possibly do this is certain filters, where the results of operations on one pixel don't effect another pixel, or where they can be combined in such a way as to null any effect the splitting would do. Although looking at PS now, I don't see any that would qualify.

Apple has not completely, or ever really, disqualified the possibility of adding MPI or such support into Xgrid. And with proper encapsulation, it should become relatively trivial to submit jobs to the grid, with the framework handling communication, and your Controller (class) handling the I/O. I think anyway, I haven't figured it all out yet.
post #46 of 69
I just think it would be nice to have an xgrid option in the sharings prefpane that allows your spare cycles to be used by other computers. Then when an applications could use more it can just look quickly to see if it has any other computers that could help.

Is this unrealistic?
0 People Found This Reply Helpful
Reply
0 People Found This Reply Helpful
Reply
post #47 of 69
Thread Starter 
No, it'd definently be a lot simpler than having to go into Xgrid Preferences and set up that for novices who just want to do what you say.

And the overhead for administration would be much less I think, just make sure the password is available to use, so that Sys Admins can set it up on all machines and only for their machines to use.
post #48 of 69
You get precisely that when you install Xgrid, a SysPref pane that lets you set all sorts of items.

And yes, sharing your computer can be as simple as a checkbox.


Eg: the simplest is Rendezvous enabled auto-connect no-password on the same LAN. If that's what you want, then one checkbox will do it for you. Someone else sets up a controller, and voila. Done.
My brain is hung like a HORSE!
Reply
My brain is hung like a HORSE!
Reply
post #49 of 69
Thread Starter 
I know you get the Xgrid Pref Pane, but an option within Sharing would be logical to regular users. You could almost justify putting the entire pane inside Sharing. "I'm Sharing my cycles with them. They're Sharing their cycles with me."

I'm not sure whether it was the mailing list of this thread (I browsed quickly for it), but Xgrid needs a lot more META data for each Agent.
  • CPU Type/Speed (AltiVec?)-AltiVec could be determined by CPU, but can still be displayed separately. This tells Controller what machines are really fast, slow, in-between and sends jobs out accordingly. Code that contains a switch saying "I'm not AltiVec Optimized" can be justifiable sent to faster G3s and get the same speed as fast G4s.
  • Machine Model-same as above
  • System Memory-more of above, SPEED
  • Network Port(s) - Perhaps even Ping times, more speed factors
  • Current Job/Priority/Status(Pending/XX% Complete/Finishing)-just what it says, everything you'd ever want to know about the current job
I'm sure there's some other things which I can't think of immediately.

This is where the customizability will come in. When you can prioritize and submit jobs with a great amount of control over what is done, and how fast it's done.
post #50 of 69
Those are good ideas.

Hop on the Xgrid mailing list and let them know you want them. Xgrid's still in prerelease, so there's lots left to do.
"...within intervention's distance of the embassy." - CvB

Original music:
The Mayflies - Black earth Americana. Now on iTMS!
Becca Sutlive - Iowa Fried Rock 'n Roll - now on iTMS!
Reply
"...within intervention's distance of the embassy." - CvB

Original music:
The Mayflies - Black earth Americana. Now on iTMS!
Becca Sutlive - Iowa Fried Rock 'n Roll - now on iTMS!
Reply
post #51 of 69
Jeez mon! It's a tech *preview*!

I fully expect a lot more detail can be exposed to power users, (the info about CPUs, etc, has to be buried in there already for the CPU tach, etc) but you have to admit that the idea of a simple checkbox for such a complex technology is appealing.
My brain is hung like a HORSE!
Reply
My brain is hung like a HORSE!
Reply
post #52 of 69
Excellent rundown can be found here.

POVray on Xgrid! Woot!
My brain is hung like a HORSE!
Reply
My brain is hung like a HORSE!
Reply
post #53 of 69
Quote:
POVray on Xgrid! Woot!

NOW yer talkin' !!! 8)
-=:[T]:=-
Reply
-=:[T]:=-
Reply
post #54 of 69
Quote:
Xcode already does this now.

Does Xcode use distcc (distributed c compiler, farms out compiles) or something else ?
Stoo
Reply
Stoo
Reply
post #55 of 69
Quote:
Originally posted by jginsbu
Here's what I find a little odd: Why the screensaver approach?


I believe that this is one of Xgrid's strengths. Remember, it is designed to enable laboratories, universities, rendering shops etc make use of spare cycles in an already existing network of Macs. Thus, it will 'take over' secretaries' or accountants' computers once they have gone home. Having a screensaver means these workers don't need to remember to set anything, and, in the event that they need their computers, will retain control. My school has several hundred Macs that I'm certain are doing nothing between about 7PM and 8AM - think of the wasted cycles that will become available. Thus, you get high power distributed computing without any significant investment. As things stand, I believe Apple views this sort of system as at least as important as using Xgrid in a specially bought system like Big Mac (for which it is probably far from ready anyway).

Looking into the future, I'm sure we'll see this built in at OS level. In fact, MOSR has it coming in 10.4, though you can probably take that with a large pinch of salt. One thing I think would be an especial advantage of this would be that, if enabled, Xgrid could seek out ANY available cycles on the network, and then ask the user if s/he would mind 'donating cyles' in the background. Of course, it would have to be configured to avoid slowing the donor's system, and there would doubtless be security issues to sort out. The preview won't do this right now; you have to know the cluster's name and password to join.

In the even further future, isn't this one time where we can really say that a shift in the most basic way we design computers is possible? I'm well aware that the OS and apps and so on would require a lot of work to be clusterable, but nevertheless what about this: at some point, processor speed becomes irrelevent. As soon as you fire up Doom 6 or whatever, the OS locates all the cycles you need and allocates them to you. Same goes for your big render, your statistics project, your bioinformatics stuff, your compiles. Is this a fair way off in the future and does it require a lot of development, not least in networking? Yes to all. Is it feasible? I certainly believe so.

It is an exciting prospect.
post #56 of 69
Thread Starter 
That's not going to happen. That's far too complicated and doesn't provide any significant benefit. Playing Doom 6 with everyone else's computers is outright stupid. They use massive supercomputers on highly optimized and near-zero latency interconnects to do visualization. Admittedly, 3D-games aren't THAT intensive, but the return for such work would never prompt anyone to do it. Plus processor speed doesn't become irrelevant, SOMEONE needs to have those cycles.

Also, you completely contradict yourself in your first and second paragraphs. You begin with using accountants computer when they're not working to, finding every unused clock-cycle and taking advantage of it. So what's the screensaver for?

As the tech evolves and people put together plug-ins and frameworks to build off of, pretty amazing stuff will come out of this. Already on the mailing list someone figured out some of the BEEP syntax, and wrote a custom plug-in, not building one from Apple's Custom Plug-In.

I'm gonna being waiting for the Emerging Technologies Conference to see what comes out then.
post #57 of 69
Quote:
Originally posted by macserverX
That's not going to happen. That's far too complicated and doesn't provide any significant benefit. Playing Doom 6 with everyone else's computers is outright stupid. They use massive supercomputers on highly optimized and near-zero latency interconnects to do visualization. Admittedly, 3D-games aren't THAT intensive, but the return for such work would never prompt anyone to do it. Plus processor speed doesn't become irrelevant, SOMEONE needs to have those cycles.

You'd be surprised. One of the projects right now in my dept is a dedicated uber-graphics-machine that consists of a graphics rendering node on everyone's desk connected through a high-speed backbone. It's used to drive your own desktop, but if someone else wants to use your spare cycles, they're available. Also, you can share desktops (or individual windows, or individual sections of windows, your choice) and stitch them together into one big projected layout... it's quite unreal.

Granted, Xgrid is quite different, but the type of shift being talked about isn't as far off as you may think.
My brain is hung like a HORSE!
Reply
My brain is hung like a HORSE!
Reply
post #58 of 69
Quote:
Originally posted by macserverX
That's not going to happen. That's far too complicated and doesn't provide any significant benefit. Playing Doom 6 with everyone else's computers is outright stupid. They use massive supercomputers on highly optimized and near-zero latency interconnects to do visualization. Admittedly, 3D-games aren't THAT intensive, but the return for such work would never prompt anyone to do it. Plus processor speed doesn't become irrelevant, SOMEONE needs to have those cycles.

Sorry, but I can't buy those objections. Sure, it's not going to happen for a while, but never? These clusters are generating several hundred Ghz already, but you think there's no significant benefit? You won't always need 'highly optimized, near zero latency' and all the rest to do effective grid computing. The return is already worth it to rendering shops, so it isn't that much of a jump to gaming. Heck, I'm not claiming that this is inevitable, but I certainly wouldn't dismiss it on the grounds you chose. The history of computing is full of those who said 'couldn't be done' or 'isn't worth it' and so on. They were wrong a lot of the time.

Quote:
Also, you completely contradict yourself in your first and second paragraphs. You begin with using accountants computer when they're not working to, finding every unused clock-cycle and taking advantage of it. So what's the screensaver for?



The point of that is simply to ensure that the grid will automatically start up. Thus, clerical workers can forget it's even there, while Sys admins can go home knowing it'll kick in. Sorry, it may seem that I meant to include unused cycles while the computer is in use; this isn't possible with Xgrid AFAIK, though it might be a goal for the future.

Quote:
As the tech evolves and people put together plug-ins and frameworks to build off of, pretty amazing stuff will come out of this. Already on the mailing list someone figured out some of the BEEP syntax, and wrote a custom plug-in, not building one from Apple's Custom Plug-In.

I'm gonna being waiting for the Emerging Technologies Conference to see what comes out then.

I couldn't agree more. But I don't think these things will be limited to traditional supercomputing apps (as Kickaha suggests). I'm sure that this will evolve in ways we can't begin to predict. Of course, some will always need the fastest processor, but nevertheless wouldn't it be benficial if one could make use of others' cycles?

I will be attending an Xgrid presentation, given by Apple, on Feb 3. I'll see if I can ask them about some of the things on this thread.
post #59 of 69
Thread Starter 
I'm trying to make the point that what you are talking about (I'm not even sure what it's called at the moment but I've read up on it) I'll call it Distributed Imaging is a whole different ball game than rendering frames of a movie in Renderman or Shake or Xgrid. It's called x-GRID for a reason. They all distribute jobs with specific input and parameters and let it run, and if you build the capabilities for these jobs to communicate then they can.

Kickaha, you used the phrase "high speed backbone" now if you did this with switch Gigabit Ethernet then, I guess I fold. But that isn't a cheap or common solution anyway. And the office types you're talking about don't transfer very many files or large ones, so using 10Base-T switches in those areas may be a company cost cutter (doubtful) in any case a lot of equipment would need to be upgraded at huge expense.

Xgrid is able to use cycles while not in screensaver mode when the "Always" radio button is active.

jouster, could you indicate what you might ask about, and maybe we'll think of some other things or clarify before you ask? I'm hoping they put out a bunch more docs on the stuff.
post #60 of 69
Thread Starter 
I was just looking at some old docs I have from my previous contributions to the High Performance Computing (HPC) from Apple rumors. It's kinda scary how accurate I was.

Project Wolf/XGrid
Xgrid not vaporware

Actually, a few Replies into the first thread, they talk about radiosity rendering, which would completely eliminate the speed advantage. Also, you can't predict the next frame of a game, so every 'n' seconds it'd have to send AND receive an image.
post #61 of 69
Kickaha and Amorph couldn't moderate themselves out of a paper bag. Abdicate responsibility and succumb to idiocy. Two years of letting a member make personal attacks against others, then stepping aside when someone won't put up with it. Not only that but go ahead and shut down my posting priviledges but not the one making the attacks. Not even the common decency to abide by their warning (afer three days of absorbing personal attacks with no mods in sight), just shut my posting down and then say it might happen later if a certian line is crossed. Bullshit flag is flying, I won't abide by lying and coddling of liars who go off-site, create accounts differing in a single letter from my handle with the express purpose to decieve and then claim here that I did it. Everyone be warned, kim kap sol is a lying, deceitful poster.

Now I guess they should have banned me rather than just shut off posting priviledges, because kickaha and Amorph definitely aren't going to like being called to task when they thought they had it all ignored *cough* *cough* I mean under control. Just a couple o' tools.

Don't worry, as soon as my work resetting my posts is done I'll disappear forever.
post #62 of 69
If you want to see where something like this could contribute to gaming, look at where corners are currently being cut, or possibilities discarded, in the name of preserving framerates and playability.

For instance: Computers on a grid could be handed the task of pre-rendering upcoming areas, so that the transition was seamless regardless of the size of the level. Or, cycles could be spent on compute-intensive things like natural language parsers, freeing designers from fixed dialog trees at least to some extent. Or, game designers could borrow a page from systems designers and use short-, medium- and long-range AI. Right now all AI is short-range (tactical). Any medium or long term planning is calcified into scripting. Spare cycles could be used to have NPCs react "off-screen" to what the player is doing, so that the player would not necessarily have to be restricted to whatever options the designer decided to allow. Even things like the contents of containers and storerooms could be offloaded, since the access latency could be hidden behind animations of doors opening and such. This would free up precious low-latency RAM on the main machine for things that the game really needed to access now without sacrificing depth of detail.

Those are just a few options off the top of my head. I can see RPGs and character-driven games and MMORPGs benefitting especially, but a clever designer could doubtless find ways to use that extra power.
"...within intervention's distance of the embassy." - CvB

Original music:
The Mayflies - Black earth Americana. Now on iTMS!
Becca Sutlive - Iowa Fried Rock 'n Roll - now on iTMS!
Reply
"...within intervention's distance of the embassy." - CvB

Original music:
The Mayflies - Black earth Americana. Now on iTMS!
Becca Sutlive - Iowa Fried Rock 'n Roll - now on iTMS!
Reply
post #63 of 69
Thread Starter 
I'll hand you that. It is a very good idea, the likelyhood of it happening is questionable, but still a very good idea. Kinda takes a hint from Myst, pre-render everything and it looks really really good. And then modify the pre-rendered object/room/character as things happen, on the local machine. Sounds better and better the more I think about it. If you shoot the lights out, changes the radiosity quick and stores what everything should look like, then adds characters and shadows and such. Ok, I'll give you this, it's cool to think about but, the RIO for developers might not be great. People will buy their stuff no matter what, and how many people have extra computer lying around to render rooms in Doom 6? That's my main argument for it not happening, there really isn't a market for it, so devs won't do it (unless it's stupidly easy).
post #64 of 69
Well, Amorph and the MacserverX have by now thought the gaming thing through much more thoroughly than did I. Though I never doubted the possibility of using grid computing for gaming, at least at some point in the future, I mentioned it merely because it seemed an appropriately intensive task that might benefit. I certainly didn't think of those ideas. The prerendering and AI ideas are particularly intriguing.

I think it all goes to show that there are any number of uses that we are currently only vaguely aware of. I can only assume that the hardware costs MacserverX mentions will decrease as time goes by.

The presentation will probably be dominated by scientists, since it is taking place in a research university. I'm sure things like bioinformatics and nuclear physics will be the popular topics, but I will try to ask about rendering/gaming if I get a chance. Hopefully I'll get an idea of Apple's general plans for Xgrid.

Edit: macserverX, I was thinking that people wouldn't need extra comps at home, since they could 'find' others by rendezvous or some similar method. Of course, you would be able to block others from using your cycles if you wanted to. Maybe Xgrid will continually enlarge the 'area' from which it could efficiently accept/disperse tasks as bandwith and network speeds increase?
post #65 of 69
Thread Starter 
Well, in the original post years ago by "allenmcjones", "allen" mentioned maybe a dozen code names for Apple research. Then he had a list of things that would be part of Project Wolf ("Wolves work well alone, but are awesome in packs"). One thing I remember is a little story about, starting a Job in New York, with computer all over the country working for you. Getting on an airplane and flying to...Cupertino let's say. You connect to the Net and your completed job is delivered. There were also all kinds of random words something like "scalar-vector components" is what the jobs are broken into. Strange.

That would be pretty awesome if every Mac was connected to an Apple Grid. High speed Internet required (or not depending on job type). So that every Mac user could contributed to Protein folding, DNA matching, and whatever else people do, without thinking about it. Helping all man-kind. See Macs are good for everyone.

Now I'm justing going off the edge, but it's still fun.
post #66 of 69
Quote:
Originally posted by macserverX
Well, in the original post years ago by "allenmcjones", "allen" mentioned maybe a dozen code names for Apple research. Then he had a list of things that would be part of Project Wolf ("Wolves work well alone, but are awesome in packs"). One thing I remember is a little story about, starting a Job in New York, with computer all over the country working for you. Getting on an airplane and flying to...Cupertino let's say. You connect to the Net and your completed job is delivered.

That's exactly the sort of thing I was thinking of (at least as much as my gaming example), though I don't for a moment pretend to know the ins and outs of implementing it.
Quote:
There were also all kinds of random words something like "scalar-vector components" is what the jobs are broken into. Strange.

Heh that sounds like some pretty specialized stuff......
Quote:
That would be pretty awesome if every Mac was connected to an Apple Grid. High speed Internet required (or not depending on job type). So that every Mac user could contributed to Protein folding, DNA matching, and whatever else people do, without thinking about it. Helping all man-kind. See Macs are good for everyone.Now I'm justing going off the edge, but it's still fun.

I'll try to ask the Apple guy about that sort of thing. It sure is fun!
post #67 of 69
Thread Starter 
I know on the mailing list or the Technology Overview that was posted there, Apple says to do those things from the computer, and not to use Xgrid to run distributed.net or SETI. But as with all things Xgrid, this could change, which would make it a heck of lot easier to get these things going on lots of computers.

If the government awarded tax incentive or whatever for businesses that spend money on electricity and maintenance for running these kinds of clients on their computers, we would get a lot of stuff done as a country research wise. And then the added incentives of some extra cash if your organization finds something good. The drug to cure cancer perhaps, or how many licks to the center of a Tootsie Pop?, important scientific discoveries.

I think that post from "allen" might still be around deep in the archives. Cuz I apparently deleted it from my HD and I doubt I would have had I not had access somewhere else. He had some other pretty interesting stuff. The search is on.
post #68 of 69
Thread Starter 
That was quick:

This is what "allenmcjones" said in this thread.
Quote:
Wolf has been a project for over 3 years and will be shipped later this year. The summer of 99, Apple acquired elite kernel and assembler programmers from some friends in Mountain View. This team is less than 6 and has been working on the Wolf project and my understanding is they are flipping brilliant.

Wolf is Apple Advanced Clustering Network Foundation. I dont know all the details but I felt it was time to share it for what its worth. Heres what I know and what Ive been able to piece together.

- Wolf is a code name for Apple Advanced Clustering Network Foundation
- Once you install Wolf (upgrade, patch, software?), you are able to use parallel processing at the kernel level
- Processing runs over TCP/IP
- ALL applications will have access to Wolf
- The network is similar to Peer to Peer
- You can process jobs on any Wolf enabled Macintosh
- You can share your computer resources to the network
- It utilizes ZeroConf as a service
- A demo of a 8 Mac Wolf cluster ripped an entire 120 minute movie in under 15 seconds.
- This is extremely advanced technology
- Team was acquired from elite people from Mountain View, CA
- Team is less than 6 and all have PHDs and are considered to have written the book on clustering.
- You can start a Wolf job on the east coast, turn off your book, fly to the west coast, get on the net and your job is completed and delivered.
- (tough to explain because I dont know what this means) Data is sent back and forth through vectorization mode. Binary data is converted in real-time to vector data, converted to ascii, parsed-mapping and then sent back. ( I have no idea)
- Metro cities will see higher responses than rural areas based upon number of Macs in regions.
- Ive been told this has also been called Davids Stone like Glove
- You can close off any local Wolf cluster similar to an Airport network.
- Its yet another reason to switch to a Mac.
- I have more but info, but this is a start for now.

Actually Amorph made the last post in that thread. The information and "jones" are heavily debated throughout the thread, though a quick browse.

Not quite the same things as Xgrid actually, but good read no the less.
post #69 of 69
Actually, I remember the post you just dug up. True, It's not exactly Xgrid - but not a million miles away. Perhaps Wolf had some more advanced stuff that hasn't yet made it to Xgrid's feature set.
New Posts  All Forums:Forum Nav:
  Return Home
  Back to Forum: Mac Software
AppleInsider › Forums › Software › Mac Software › XGrid?