blastdoor

The news of expanded office space follows a report that Apple TV+ budgets are being cutbecause the service is losing Apple money

tiredskills said:

blastdoor said:

tiredskills said:

blastdoor said:

tiredskills said:

jeff fields said:

tiredskills said:

Why on earth would I want to run an AI model?  Locally or otherwise?

If you don't already know, perhaps you should pause on commenting in public until you've spent 15 seconds figuring it out.

Suffice it to say that many people have very good reasons to do this.

Such as?

I am obviously not one of those people, so am asking why.  Your answer is not illuminating.

Are you retired? If so then don’t worry about it. 

I am not.  I work in enterprise software development.  I have seen nothing significantly useful from this AI revolution so far, just a lot of fakery, deception and disruption of trust.  These are unequivocally bad things to my mind.

So I ask again, why would I want to run an AI model?  Locally or otherwise?

It's probably best that you don't use them -- leave it to others. Just stick to your comfort zone. 

If you can’t think of any answers it’s perfectly fine to just say so.

tiredskills said:

blastdoor said:

tiredskills said:

jeff fields said:

tiredskills said:

Why on earth would I want to run an AI model?  Locally or otherwise?

If you don't already know, perhaps you should pause on commenting in public until you've spent 15 seconds figuring it out.

Suffice it to say that many people have very good reasons to do this.

Such as?

I am obviously not one of those people, so am asking why.  Your answer is not illuminating.

Are you retired? If so then don’t worry about it. 

I am not.  I work in enterprise software development.  I have seen nothing significantly useful from this AI revolution so far, just a lot of fakery, deception and disruption of trust.  These are unequivocally bad things to my mind.

So I ask again, why would I want to run an AI model?  Locally or otherwise?

CheeseFreeze said:

As an AI-developer myself, a M3 Ultra would be an incredibly stupid purchase. The machine would only be good for a very limited set of AI-models.

You'd be better off purchasing Digits for $3K, yes with 25% of the memory (128gb), and offload work to the cloud when needed,
or chain two these machines for $6K. https://www.wired.com/story/nvidia-personal-supercomputer-ces/
It would perform much better. Not only memory should be taken into account, but also the entire ecosystem around AI development 
and performance across as well as internal storage and the type of chip and how it performs across models other than LLMs.

The M3 Ultra is a best for video, 3D and post-production.

Marvin said:

brianus said:

tiredskills said:

Why on earth would I want to run an AI model?  Locally or otherwise?

I’m sure this was meant to be snarky, but for me it’s a genuine question: what are the envisioned real world use cases? What might a business (even a home one) use a local LLM for?

The article mentions a hospital in the context of patient privacy, but what would that model actually be *doing*?

In hospitals, AI models are reviewing patient scans to detect cancer:

https://www.youtube.com/watch?v=Mur70YjInmI

This is image analysis rather than text but text models can be used for medicine. There's an online AI for free here:

https://duckduckgo.com/chat

It can be asked about medical issues like if there's a pain somewhere, what could it be and what treatments are available e.g 'What medicine is typically used to treat acid reflux?'.

In a clinical setting, a doctor would review the recommendations.

In business, they'd be better off using a custom AI model that is trained on high quality data. A legal company might train a model on past cases and they can quickly find similar cases to use as references.

Local models are usually more responsive (if the hardware is fast enough), don't get timeouts and you can save past prompts more easily. They would likely still be cloud-based so that all employees can access them from lightweight clients, just a company cloud server.

blastdoor said:

At about 5 minutes and 30 seconds he says that building this with consumer PC hardware would be "quite expensive." I was looking for a fair bit more precision than that. 

Specs are listed here:

https://geekbacon.com/2025/02/20/running-deepseek-r1-671b-locally-a-comprehensive-look/

It needs multiple 3090 or higher GPUs + 512GB RAM. There's a video here showing a $2000 setup but it only runs at 3 tokens/s:

https://www.youtube.com/watch?v=Tq_cmN4j2yY&t=2822s

Another uses an Nvidia 6000 that costs around $7k for the GPU:

https://www.youtube.com/watch?v=e-EG3B5Uj78&t=560s
https://www.newegg.com/pny-vcnrtx6000ada-pb/p/N82E16814133886

Performance is 4 tokens/s. The video in the article mentioned the M3 Ultra was around 17 tokens/s.

This is one area where Nvidia and AMD are worse value and they do it on purpose because they make a lot of their revenue from this where they lower the memory in the consumer GPUs and charge a lot for enterprise GPUs with more memory that is needed for AI.

This video tests Nvidia H100 GPUs x8 ($28k each - https://www.newegg.com/p/N82E16888892002 ), which gets 25 tokens/s:

https://www.youtube.com/watch?v=bOp9ggH4ztE&t=433s

If Nvidia sold a model of the H100 with 512GB of memory, it could probably compete with M3 Ultra but would cost more than $30k just for the GPU.

Applications that need lots of unified memory is where Apple's hardware design is very competitive and they knew this when designing it.