PC3200 and 970s?

[Ompus]

Am I correct that only PC-3200 would come close to having enough bandwidth to feed a SINGLE 1.8ghz 970? What options are on the horizon?

[Amorph]

Yes. Furthermore, if Apple wants to extend their current architecture then they'd have to go dual channel, or quad for dual 970s.

Why do that? Because for all that's called a hack, the current architecture has some real advantages. With QE, the graphics card needs to be fed as well as the CPU, so memory should be able to saturate both at once (at least insofar as the busses allow). Currently, it can. The fact that there's memory bandwidth enough for all the other I/O channels at once is a bonus. It means that the machine as a whole doesn't start choking when you really push it hard. I'd like to see Apple continue down this road, and their OS seems to indicate that they will.

[Ompus]

If PC 3200 is the fastest memory we can expect in 2003, and if it can barely keep up with a SINGLE 970, then isn't it highly unlikely that we'll see DUAL 970s in 2003?

[Amorph]

That depends on the architecture Apple settles on. Dual channel SDRAM (two banks running in parallel) is already deployed on Athlon-based motherboards. They could both feed into one (really fast) memory controller that in turn fed the processor bus (and whoever else needed to be fed). That's one option.

The other is to give each CPU its own bank of RAM, and hook the CPU/RAM units together on a high-speed fabric. I wouldn't be surprised if Apple goes this way. This is the approach used in SGI workstations. It wasn't cost-effective in PCs until HyperTransport and RapidIO appeared, and even with those it won't be easy (because the system has to keep all those banks of RAM synchronized, and enable one CPU to grab a value from another's RAM). There would probably have to be another bank of RAM to sate AGP and PCI and I/O channels as well, if that's feasible.

Caveat: I'm getting in over my head, here, so I fully expect one of the more hardware-minded folks here to correct something I've said. Don't tattoo this post on your forehead or anything.

[BRussell]

[quote]Originally posted by Amorph:
<strong> Don't tattoo this post on your forehead or anything. </strong><hr></blockquote>Damn, you could have made that your first sentence instead of your last. These things are permanent you know.

[Pilmour Boy]

I don't think we'll see the RAM-proc-"North Bridge" combo with IBM Power-PCs until the Power5 variant (PPC 980?) hits the streets. The Moto 7457-RM will also be of this design, and it really is the future, despite the interesting implications of a NUMA design for cache coherency and DMA access for peripherals.
The 6.4 Gb/s of the 970 @ 1.8 GHz is actually 2*3.2 Gb/s, so won't be as good as other solutions for some algorithms.
According to IBM, the 970 can work in a shared FSB mode, but this seems rather strange considering the design (2*32 bit one-way busses), and has caused some speculation over at Ars (the perpetual future Apple CPU thread).

[wmf]

[quote]Originally posted by Amorph:
<strong>The other is to give each CPU its own bank of RAM, and hook the CPU/RAM units together on a high-speed fabric.</strong><hr></blockquote>

That sounds like it would involve a northbridge chip for each CPU (expensive) instead of one northbridge for the whole system as we have today. If the RAM attached directly to the CPU it would be different, but that's not the case for the 970.

[Programmer]

There is actually good reason to have a dual northbridge system, and while it might be a bit pricey we might still see it appear.

For starters, Apple will only want to design one chip so it either has 2 970-FSB ports, or they use two of them and interconnect (perhaps with RapidIO, and/or ApplePI). A single chip with two ports would be much larger, have many pins, and thus more expensive, whereas building two simpler ones gives you additional volume to help drive costs down.

These chips communicate with the 970 via a very high speed bus -- 900 MHz in the case of the 1.8 GHz chip. This will require a high quality circuit board and the shortest traces they can manage. One way to do this and minimize costs is to mount the high speed parts on a daughtercard. The rest of the motherboard is much like the current ones, and the interconnect is via something like RapidIO or HyperTransport. With dual northbridges they have two options: put both processors & northbridges on the same daughtercard, or on seperate daughtercards. If they are together they can use a high speed interconnect (ApplePI), if they are seperate they can use something like RapidIO. 2 single processor cards has the advantage that single and dual processor machines use the same card... but its slower.

The third question is where the memory lives and is divided. Motherboard? Daughtercard? Managed seperately by each northbridge (if there are two), or somehow shared? If seperate a NUMA architecture is needed... and again this is something RapidIO is designed for. If shared then somehow both northbridges need to talk to the controller, but HyperTransport can do this (with multiple ports on the memory controller).

Lots of options.

[Nevyn]

[quote]Originally posted by Programmer:
<strong>Lots of options.</strong><hr></blockquote>

Can you (or anyone) definatively say the ppc 970's 2x one-way buses aren't a varient of RapidIO or HT? As far as I could tell (which isn't saying much) the description of the 970's FSB uses pretty much the same terminology, and could pretty easily be an understatement of that one aspect. Or are they possibly set up the way the Power4's bus lines were such that 4 of the chips could be assembled into a larger module by just rotating each 90 degrees relative to the previous chip?

It just seems odd that Apple would end up transitioning to an whole new set of northbridges for just a year or so until their suppliers switch over to RIO. And once Apple had a dual RIO CPU setup, it should be possible to step to more processors with off the shelf RIO switches...

I don't know - just wild-assed guessing. The whole RIO thing screams "Supercomputer's switched fabric backplane in very thin disguise" to me.

[Powerdoc]

I doubt that Apple will ever use PC 3200 memory.
First of all this type of memory is not officialy recocnize by the consortium of producers.
PC 3200 memory are not officialy supported in the PC world by the producers of mobo.
PC 3200 lack of stability and performance. CL2 PC 2700 are more performant than CL2,5 PC 3200.

In the future it will be better to use DDR 2 memory, or double channel PC 2700.

[wmf]

[quote]Originally posted by Programmer:
<strong>There is actually good reason to have a dual northbridge system, and while it might be a bit pricey we might still see it appear.

For starters, Apple will only want to design one chip so it either has 2 970-FSB ports, or they use two of them and interconnect (perhaps with RapidIO, and/or ApplePI). A single chip with two ports would be much larger, have many pins, and thus more expensive, whereas building two simpler ones gives you additional volume to help drive costs down.</strong><hr></blockquote>

My only concern with that approach is that a northbridge with one 970 FSB and one other high-speed port doesn't sound any simpler than a northbridge with two 970 FSBs.

And who says Apple's suppliers are switching to RapidIO? Once you have the 970 FSB, why switch?

[Barto]

QDR! Go QDR! In production now, cheaper than DDR-II and backwards compatible with DDR! Go QDR! Woohoo!

Barto

PS I hope the Power Mac G5 uses quad channel QDR400.

[Bacon]

Regardless of the ultimate solution, a dual 970 system would necessitate some pricey components. Such a system would as expensive as it would be fast.

[Pilmour Boy]

Programmer, these diagrams were made by "I Want X" on the Ars forums, and show what you're saying wrt to having a "North Bridge" on the daughter board, and then RIO/HT links to other daughter boards.
For a single proc system:

and for a dual proc system
.
Now of course with these, the North Bridges could be the same, and it could even be possible to put all of the processors and "companion" chips on to daughter boards, and then have a RIO/HT link to the mother, meaning that the same motherboard could be used for all the systems.

[Programmer]

[quote]Originally posted by wmf:
<strong>My only concern with that approach is that a northbridge with one 970 FSB and one other high-speed port doesn't sound any simpler than a northbridge with two 970 FSBs.

And who says Apple's suppliers are switching to RapidIO? Once you have the 970 FSB, why switch?</strong><hr></blockquote>

Different busses are designed for very different purposes. the 970 FSB is designed for very short path, point-to-point, high speed communication. This is similar to HyperTransport, but it isn't HT.

RapidIO, on the other hand, is being pushed by Motorola, IBM and others. It is designed to connect together many chips in kind of a on-board packet network. It is slower (currently maxing out at around 2 GB/sec in the fastest implementation, IIRC). It uses fewer, slower traces (well, slower than the 970 FSB anyhow).

The reason to use both is to separate the high speed components from the lower-speed components, which allows costs to be minimized. The "companion chip" has to run at the FSB's speed, but the I/O system doesn't. If you integrate them all into one chip, however, then the whole thing has to be able to run fast and yields will be lower.

I notice that my terminology doesn't match those diagrams -- above I was using "northbridge" to mean the "companion chip", whereas they are called out separately on the diagrams from Ars. I'll switch to using "companion chip".

[ 01-11-2003: Message edited by: Programmer ]</p>

[Pilmour Boy]

No, you're right. Bad Andy pointed out that traditionally, the North Bridge is just whatever is connected to the processor and contains the memory controller. It is just that in recent times we in the Macintosh community have become used to a kitchen sink style NB, which contains all high speed peripherals. What we may be seeing come back in a NB which just has a memory controller and a RIO/HT output, up from the traditional PCI connection.

[mmicist]

[quote]Originally posted by Programmer:
<strong>

Different busses are designed for very different purposes. the 970 FSB is designed for very short path, point-to-point, high speed communication. This is similar to HyperTransport, but it isn't HT.

RapidIO, on the other hand, is being pushed by Motorola, IBM and others. It is designed to connect together many chips in kind of a on-board packet network. It is slower (currently maxing out at around 2 GB/sec in the fastest implementation, IIRC). It uses fewer, slower traces (well, slower than the 970 FSB anyhow).
</strong><hr></blockquote>

Up to a point, lord Copper.
Actually current RapidIO runs at a base frequency of 500MHz, with DDR to give an effetive 1GHz signal rate, so it is clocked faster than the 970's bus. It is, however, 16 bit each way instead of 32 bit each way, so fewer traces, but gets to 4GB/s bidirectional bandwidth, 2GB/s unidirectional. RapidIO and HyperTransport are also both point to point.
This confusion arises because of marketing peoples' preference for stating the highest number when quoting speeds, DDR266 is, of course, a 133MHz bus, and P4's 533MHz, is actually also a 133MHz bus using QDR signalling. The 970's bus will actually clock at twice the speed of the P4's upcoming (so called 800MHz) bus.
I cannot be certain that the 970 bus uses low voltage differential signalling, but strongly suspect it does, and if that is is added to IBM's wave pipelining techniques and deskewing circuitry, it may be that the 970 bus is actually easier to route than RapidIO or HyperTransport. If the bus is not differential then very short traces will be required.

michael

[Outsider]

Another thing to consider is the negineering involved in making a 900MHz 32bit (times 2) bus travel through the physical connection between the daughterboard and motherboard. So putting the main memory controller and IO connector on the daughter card connecting to the main peripheral controller on the motherboard using a bus designed for this like HT or RIO would be the ideal solution. But riddle me this: Unless you plan on putting the memory slots on the daughter card (unlikely but conceivable), you would still need a 64bit interface to the slots on the motherboard coming out of the physical connector between processor card and motherboard. That's not too bad but with a NUMA style architecture you'll need a second set of traces to the motherboard via the connector. So either slots on the daughter-card so you can design the mobo to accept dual or single processor modules, or the mobo needs to be designed specifically for dual machines and only for duals. Unless the controller can auto-sense the number of processors and assign all slots or just half to each processor. Interesting to see how they will work this out.

[Programmer]

[quote]Originally posted by mmicist:
<strong>
Actually current RapidIO runs at a base frequency of 500MHz, with DDR to give an effetive 1GHz signal rate, so it is clocked faster than the 970's bus. It is, however, 16 bit each way instead of 32 bit each way, so fewer traces, but gets to 4GB/s bidirectional bandwidth, 2GB/s unidirectional. RapidIO and HyperTransport are also both point to point.
</strong><hr></blockquote>

Well I don't want to get into an argument about the details of bus design (I'll lose), but I'll point out some differences...

• RapidIO is packet based and thus has only data lines. Addresses are sent as part of the packet. The system is designed to be part of a fabric switched network.
• The 970's FSB has an address bus (46 lines or so?) and operates as a more traditional pipelined, split transaction bus.
• RIO will eventually scale to almost 8 GB/sec raw bandwidth (according to the RIO FAQs). At the moment, however, they are well short of that. The packet-oriented nature of the bus exacts a 5-25% overhead from its theoretical transmission speed.
• The 970 FSB is synchronous, locked at half the processor's clock rate. This means its highest initial rate is 900 MHz, but as the 970's speed increases (and I suspect we'll see it reach at least 2.5 GHz) the bus speed will increase (to 1.25 GHz if I'm right about the 2.5 GHz). That rate applies to both the data and address busses.
• The 970 FSB will have a much lower potential latency than the packet oriented RIO bus.
• RIO is designed to be scalable from slow implementations to very fast ones, whereas the 970 is pretty much just always fast.

[Amorph]

I'm also hoping - perhaps foolishly - that they'll find a way to ensure that there's enough memory bandwidth left over to keep everything else on the motherboard happy. That will help them squeeze every last bit of performance out of their hardware, which is especially useful considering that they're beginning to rely on that (Quartz Extreme).

Given the complexity that's already involved in implementing a traditional NUMA architecture with the technology the 970's built around, I'm not sure how they'd go about doing that. But some part of me is sure that they will, and I wouldn't be surprised if the board looks nothing like anything that's ever come out of a PC vendor as a result.

[Programmer]

[quote]Originally posted by Amorph:
<strong>I'm also hoping - perhaps foolishly - that they'll find a way to ensure that there's enough memory bandwidth left over to keep everything else on the motherboard happy. That will help them squeeze every last bit of performance out of their hardware, which is especially useful considering that they're beginning to rely on that (Quartz Extreme).

Given the complexity that's already involved in implementing a traditional NUMA architecture with the technology the 970's built around, I'm not sure how they'd go about doing that. But some part of me is sure that they will, and I wouldn't be surprised if the board looks nothing like anything that's ever come out of a PC vendor as a result.</strong><hr></blockquote>

Oh, I'm pretty sure that'll be true (the "it'll look like nothing from a PC vendor before" part).

I'm doubtful that all the PowerMacs will have enough memory bandwidth to feed the 970 (or two of them) and all the I/O ports, and the GPU. At least not when all of them are going full tilt. That really doesn't matter, however, because it gives Apple room to maneuver & improve their designs. Right now they've maxed out the MPX bus and don't really have any options. In the new scheme the busses have legs so Apple has options. And we know they like options.

[Pilmour Boy]

Programmer, are you sure about the separate address bus? Page 12 of the mpf .pdf says that address data and control is multiplexed in. Also, the 6.4 GB/s is of usable bandwidth, brought down from 7.2GB/s because of address and protocol overhead. The pdf says that there is a separate side bus for cache snooping and ACK (acknowledge?).
I do not think that the 970 will reach 2.5GHz on the .13 process, especially considering that the same pdf is only saying for 1.4-1.8 GHz, not for initial speeds.

[mmicist]

[quote]Originally posted by Programmer:
<strong>

Well I don't want to get into an argument about the details of bus design (I'll lose), but I'll point out some differences...

• RapidIO is packet based and thus has only data lines. Addresses are sent as part of the packet. The system is designed to be part of a fabric switched network.
• The 970's FSB has an address bus (46 lines or so?) and operates as a more traditional pipelined, split transaction bus.
• RIO will eventually scale to almost 8 GB/sec raw bandwidth (according to the RIO FAQs). At the moment, however, they are well short of that. The packet-oriented nature of the bus exacts a 5-25% overhead from its theoretical transmission speed.
• The 970 FSB is synchronous, locked at half the processor's clock rate. This means its highest initial rate is 900 MHz, but as the 970's speed increases (and I suspect we'll see it reach at least 2.5 GHz) the bus speed will increase (to 1.25 GHz if I'm right about the 2.5 GHz). That rate applies to both the data and address busses.
• The 970 FSB will have a much lower potential latency than the packet oriented RIO bus.
• RIO is designed to be scalable from slow implementations to very fast ones, whereas the 970 is pretty much just always fast.

</strong><hr></blockquote>

Actually, I believe the 970's bus is also packet based, which explains the 12% overhead, reducing the useful bandwidth from 7.2 to 6.4 GB/s for the 900MHz version. The 970 has 42 bit addressing but not any address lines.
At higher frequencies the 970 may drop down to a lower divisor for the bus (eg: 2.4GHz 970 using 2400 / 6 * 2 = 800MHz bus), but I'd be dissapointed by that. Of course I haven't seen any proof of this yet either, but there follows a quote from <a href="http:////www.realworldtech.com/page.cfm?AID=RWT101502203725" target="_blank">David Wang</a> who was at the presentation.

[quote] David Wang
<strong>Packet Based System Interconnect

One of the more interesting aspects of the PowerPC 970 processor is the system interconnect. Unlike the bi-directional processor busses seen on Intel IA-32 and IA-64 processor, or even the bi-directional point to point interconnects used on Alpha EV6 and AMD Athlon processors, the system interconnect of the PowerPC 970 processor are uni-directional, point to point, source synchronous interconnects that do not have to worry about bus loading factors or bus turn around times, and the interconnect can wave-pipeline multiple number of bits of data on the wires concurrently. The most difficult part of such high frequency system interconnect may be the deskewing circuitry that would be required. In this case, the PowerPC 970 appears to have benefitted well from the POWER4 lineage, where the deskewing circuitry for a wavepipelined interconnect was previously disclosed by IBM.

The system interconnect on the PowerPC 970 has been designed to operate at an integer fraction of the CPU core frequency. At a CPU core frequency of 1.8 GHz, the system interconnect will operate at a frequency of 900 MHz. With two unidirectional 32 bit wide interconnects, one from the CPU to the companion system controller chip, the other from the companion system controller chip back to the CPU, the system interconnects can provide 3.6 GB/s of raw system bandwidth on each direction for an aggregate bandwidth of 7.2 GB/s. However, the unidirectional links must multiplex address and control information onto the same interconnects, and when these overheads are taken into considerations, IBM claims an effective peak data bandwidth of 6.4 GB per second.
</strong><hr></blockquote>

michael

[ast3r3x]

...


who knows who cares, 970 is the future, it will be blazing fast


...

[Programmer]

[quote]Originally posted by Pilmour Boy:
<strong>Programmer, are you sure about the separate address bus? Page 12 of the mpf .pdf says that address data and control is multiplexed in. Also, the 6.4 GB/s is of usable bandwidth, brought down from 7.2GB/s because of address and protocol overhead. The pdf says that there is a separate side bus for cache snooping and ACK (acknowledge?).
I do not think that the 970 will reach 2.5GHz on the .13 process, especially considering that the same pdf is only saying for 1.4-1.8 GHz, not for initial speeds.</strong><hr></blockquote>

Heh, well I was right about one thing -- I would lose. I went back and looked again, and you guys are quite right. that David Wang quote is particuarly interesting. I had looked at the diagram and mis-interpreted the control path as an address bus. Woops.

I didn't mean that the 970 would reach 2.5 GHz on the 0.13 micron process. The 0.09 micron version will likely have the same bus interface though.

Hmmm... so the 970 FSB is much like a paired uni-directional double-wide RIO bus. It doesn't look like it would be scalable downward, however, so I'd still contend that they'd likely be using RIO or HT to the motherboard I/O chip, and leave the FSB on a daughtercard for communicating between processor(s) and the memory controller(s). Or perhaps that's just all wrong an rather than a motherboard / daughterboard setup they'll so something radically different. I/O on a daughtercard?

[quote]
who knows who cares, 970 is the future, it will be blazing fast<hr></blockquote>

I do -- the details are important, and I'd rather be corrected than spewing incorrect information. Thanks to the guys for correcting me.

[ 01-12-2003: Message edited by: Programmer ]</p>

[Outsider]

I don't know, given the details about the 970 2.5GHz seems like it's approaching the higher limits of the 130nm process. I think it may scale to that and like what the Wang comment that mmicist showed and what I saw heard about in an internal IBM doc that the bus of the 970 is not as simple as 2:1 ratio of the processor speed, that there is an internal multiplier like (1800/4)*2=900MHz and the internal multiplier can change although the final number is always at a 2:1 ratio. So something like (2250/5)*2=900MHz is also conceivable although for 2GHz the bus should be able to hit 1GHz. We'll see soon.

[Programmer]

[quote]Originally posted by Outsider:
<strong>I don't know, given the details about the 970 2.5GHz seems like it's approaching the higher limits of the 130nm process. I think it may scale to that and like what the Wang comment that mmicist showed and what I saw heard about in an internal IBM doc that the bus of the 970 is not as simple as 2:1 ratio of the processor speed, that there is an internal multiplier like (1800/4)*2=900MHz and the internal multiplier can change although the final number is always at a 2:1 ratio. So something like (2250/5)*2=900MHz is also conceivable although for 2GHz the bus should be able to hit 1GHz. We'll see soon.</strong><hr></blockquote>

I don't think anybody would object to a 2.5 GHz 0.13 implementation, but I wonder if IBM would even bother with the 0.09 implementation so close behind? A fast 0.13 would run very hot.

I hope they don't turn down the frequency multiplier of the bus, as that seems to be a key feature of the 970. Using the FSB as a system wide interconnect doesn't seem (to me) to be as compelling as using RapidIO because of Motorola's support of RapidIO. The 970 is an attractive high end part, but I won't rule out Motorola pushing the G4 line to the 7457-RM (integral memory controller, RapidIO, and ~1.8 GHz -- i.e. a very compelling low-end part). If that happens then having both processor subsystems using RapidIO simplifies Apple's system design, and RapidIO's scalability fits well with their need to design a whole line of portable and desktop (and other?) computers.

[mmicist]

[quote]Originally posted by Programmer:
<strong>

Hmmm... so the 970 FSB is much like a paired uni-directional double-wide RIO bus. It doesn't look like it would be scalable downward, however, so I'd still contend that they'd likely be using RIO or HT to the motherboard I/O chip, and leave the FSB on a daughtercard for communicating between processor(s) and the memory controller(s). Or perhaps that's just all wrong an rather than a motherboard / daughterboard setup they'll so something radically different. I/O on a daughtercard?
</strong><hr></blockquote>

I quite agree. I think that we need to know exactly what the protocol is going to be on the 970 bus, but it may be very simple in order to minimise latency of memory accesses, leaving a more capable, higher latency, bus for the I/O (and remote memory accesses if using NUMA).

Interesting, isn't it?

michael

[T'hain Esh Kelch]

[quote]Originally posted by ast3r3x:
<strong>...


who knows who cares, 970 is the future, it will be blazing fast


... </strong><hr></blockquote>
I wouldnt be so sure about that.. Everybody thinks that this chip is going to be THE messias, that will save Apple and give us 10% marketshare..

Back to the real world guys!

[Programmer]

[quote]Originally posted by T'hain Esh Kelch:
<strong>
I wouldnt be so sure about that.. Everybody thinks that this chip is going to be THE messias, that will save Apple and give us 10% marketshare..

Back to the real world guys! </strong><hr></blockquote>

No, we think this chip will free us from the chains that hold the G4 back from competing well with the leading x86 processors. It may increase marketshare by virtue of realizing the pent up demand for a fast PowerMac, but I don't think its the answer to give flight to Apple's market volumes.

Having 2 different processors does suddenly open a much larger gap between Apple's low and high end machines, however. Suddenly Apple would be able to ship low end machines with leading edge G4's without fear of cannabilizing their PowerMac market. Buy a single 1.4 GHz G4 tower/slab/cube cheap (i.e. sub $1K), or buy a dual 1.8 GHz 970 monster tower for their usual Pro price point.