Does anyone know of any software that can show me the load on my Neural Engines? Does anyone know why Apple's Activity Monitor refrains from reporting that data?
Open a terminal to a large enough size, drag in the binary and hit enter.
That worked. I'm very grateful for your advice. Now I will check the load on my Neural Engine, and report it here.
In my preliminary test, my Neural Engine usage peaks at 7% when I do my OCR task, which is still higher than 0% when I'm not running my app, but you did say the percentage wasn't accurate. At least now I have something to examine, thanks.
Hope this is true. I’m desperate to update my 10year old iMac.
So why not just get what's available now? It's already waayyy faster than what you have and will easily last another 10yrs.
Horrid advice. Buy a two year old iMac when a new one is just around the corner? You’re kidding surely.
More like just wait with what you have until the iMac is update. An m3 will smoke the m1.
People thought the A17 Pro would smoke the A16 because of 3nm but that's not the case.
It’s… in a phone.
And the GPU does smoke the a16.
I didn’t say m3 smokes m2, btw. I said it smokes m1 as the actual context of my comments pertained to the 24vinch imac which is only available with m1. Which the m2 is already a nice upgrade from.
So if you were paying attention, you’d have compared the a17 pro to the A15. But you didn’t, because… you know. Buy hey, thanks for participating.
What I want is better NE (Neural Engine) core speeds, because my software seems to be bound by the speed of the NEs. (It's hard to be sure, since the Apple Activity Monitor in macOS doesn't tell you about the NE load.) Here's what Apple's processors claim to have:
11 trillion ops/sec -- M1 15.8 trillion ops/sec --M2 Pro, M2 Max, M2, A15 17 trillion ops/sec -- A16 22 trillion ops/sec -- M1 Ultra (apparently 2x the M1 speed because the M1 Ultra is 2 M1 cores tied together. I'm not sure if real world performance scales perfectly.) 31.6 trillion ops/sec -- M2 Ultra (apparently 2x the M2 speed because the M2 Ultra is 2 M2 cores tied together. I'm not sure if real world performance scales perfectly.)
Looking at this data, there was a 40% speed bump from M1 to M2, which is definitely okay, but the other increases were caused only from doubling the number of Mx/NE cores. I'm hoping the M3 chip has a true speed bump, in which case I'm likely to replace my M1 Mac with an M3 Mac.
Does anyone know of any software that can show me the load on my Neural Engines? Does anyone know why Apple's Activity Monitor refrains from reporting that data?
Out of interest, what do you use that requires heavy use of the neural engine?
I do two things that (as far as I can tell) are using the Neural Engine. One of them is the macOS builtin optical character recognition which I access using the macOS Shortcuts app. Essentially, I'm taking screenshots of videos using this app and I gather all the text that occurs inside videos. Essentially I'm running OCR in an infinite loop. As far as I know, the OCR code in macOS is using the Neural Engine. One of the reasons I think it's running in the Neural Engine is that I don't see any load on the CPU. So where would that OCR be occurring? Probably in the Neural Engine. That's what I'm doing. As to which videos I'm doing that on, I won't tell you. Trade secret.
Most (not all) of this kind of thing works on Intel Macs still so maybe they're emulating the Neural Engine on Intel. Or it's not actually using the neural engine at all (as you point out it's only 7% used, apparently).
What I want is better NE (Neural Engine) core speeds, because my software seems to be bound by the speed of the NEs. (It's hard to be sure, since the Apple Activity Monitor in macOS doesn't tell you about the NE load.) Here's what Apple's processors claim to have:
11 trillion ops/sec -- M1 15.8 trillion ops/sec --M2 Pro, M2 Max, M2, A15 17 trillion ops/sec -- A16 22 trillion ops/sec -- M1 Ultra (apparently 2x the M1 speed because the M1 Ultra is 2 M1 cores tied together. I'm not sure if real world performance scales perfectly.) 31.6 trillion ops/sec -- M2 Ultra (apparently 2x the M2 speed because the M2 Ultra is 2 M2 cores tied together. I'm not sure if real world performance scales perfectly.)
Looking at this data, there was a 40% speed bump from M1 to M2, which is definitely okay, but the other increases were caused only from doubling the number of Mx/NE cores. I'm hoping the M3 chip has a true speed bump, in which case I'm likely to replace my M1 Mac with an M3 Mac.
Does anyone know of any software that can show me the load on my Neural Engines? Does anyone know why Apple's Activity Monitor refrains from reporting that data?
Out of interest, what do you use that requires heavy use of the neural engine?
I do two things that (as far as I can tell) are using the Neural Engine. One of them is the macOS builtin optical character recognition which I access using the macOS Shortcuts app. Essentially, I'm taking screenshots of videos using this app and I gather all the text that occurs inside videos. Essentially I'm running OCR in an infinite loop. As far as I know, the OCR code in macOS is using the Neural Engine. One of the reasons I think it's running in the Neural Engine is that I don't see any load on the CPU. So where would that OCR be occurring? Probably in the Neural Engine. That's what I'm doing. As to which videos I'm doing that on, I won't tell you. Trade secret.
Most (not all) of this kind of thing works on Intel Macs still so maybe they're emulating the Neural Engine on Intel. Or it's not actually using the neural engine at all (as you point out it's only 7% used, apparently).
Both of your points are smartly made and viable theories. I can't disprove your ideas. They might be right. It's nice to have a thread where people aren't in rude disagreement with each other. Too many people this week have called me an idiot.
What I want is better NE (Neural Engine) core speeds, because my software seems to be bound by the speed of the NEs. (It's hard to be sure, since the Apple Activity Monitor in macOS doesn't tell you about the NE load.) Here's what Apple's processors claim to have:
11 trillion ops/sec -- M1 15.8 trillion ops/sec --M2 Pro, M2 Max, M2, A15 17 trillion ops/sec -- A16 22 trillion ops/sec -- M1 Ultra (apparently 2x the M1 speed because the M1 Ultra is 2 M1 cores tied together. I'm not sure if real world performance scales perfectly.) 31.6 trillion ops/sec -- M2 Ultra (apparently 2x the M2 speed because the M2 Ultra is 2 M2 cores tied together. I'm not sure if real world performance scales perfectly.)
Looking at this data, there was a 40% speed bump from M1 to M2, which is definitely okay, but the other increases were caused only from doubling the number of Mx/NE cores. I'm hoping the M3 chip has a true speed bump, in which case I'm likely to replace my M1 Mac with an M3 Mac.
Does anyone know of any software that can show me the load on my Neural Engines? Does anyone know why Apple's Activity Monitor refrains from reporting that data?
Out of interest, what do you use that requires heavy use of the neural engine?
I do two things that (as far as I can tell) are using the Neural Engine. One of them is the macOS builtin optical character recognition which I access using the macOS Shortcuts app. Essentially, I'm taking screenshots of videos using this app and I gather all the text that occurs inside videos. Essentially I'm running OCR in an infinite loop. As far as I know, the OCR code in macOS is using the Neural Engine. One of the reasons I think it's running in the Neural Engine is that I don't see any load on the CPU. So where would that OCR be occurring? Probably in the Neural Engine. That's what I'm doing. As to which videos I'm doing that on, I won't tell you. Trade secret.
Most (not all) of this kind of thing works on Intel Macs still so maybe they're emulating the Neural Engine on Intel. Or it's not actually using the neural engine at all (as you point out it's only 7% used, apparently).
CoreML can run on the CPU/GPU/NPU and picks the fastest. Apple says the peak FP16 throughput of the ANE on A15 is 15.8TFLOPs. Max or Ultra chips could potentially outperform the neural engine on some tasks:
The NPU percentage usage is measured from the power vs 8W. For OCR across multiple framebuffers, processing/generating the uncompressed buffers will be taking some of the processing time.
The neural engine is 16-core, 1-core = 6.2%. It could be the model used isn't able to run in parallel or it just finishes so quickly it doesn't need to use more of the NPU. Processing frames in batches could potentially make it run faster.
The newer Mac systems do OCR as part of the system now. In the above graph image, dragging over the text will highlight it after a short delay. Even though it's a static image, it runs OCR and lets you copy/paste text from an image. It happens fairly quickly and doesn't register much processor usage on CPU/GPU/NPU.
Comments
In my preliminary test, my Neural Engine usage peaks at 7% when I do my OCR task, which is still higher than 0% when I'm not running my app, but you did say the percentage wasn't accurate. At least now I have something to examine, thanks.
https://machinelearning.apple.com/research/neural-engine-transformers
https://www.photoroom.com/inside-photoroom/core-ml-performance-2022
The NPU percentage usage is measured from the power vs 8W. For OCR across multiple framebuffers, processing/generating the uncompressed buffers will be taking some of the processing time.
The neural engine is 16-core, 1-core = 6.2%. It could be the model used isn't able to run in parallel or it just finishes so quickly it doesn't need to use more of the NPU. Processing frames in batches could potentially make it run faster.
The newer Mac systems do OCR as part of the system now. In the above graph image, dragging over the text will highlight it after a short delay. Even though it's a static image, it runs OCR and lets you copy/paste text from an image. It happens fairly quickly and doesn't register much processor usage on CPU/GPU/NPU.