Apple has built a new generative AI tool for animating images
Researchers at Apple have created Keyframer, a test generative AI app that lets users describe an image and how they want it to animate.
Two images generated by AI and then animated -- such as the rocket taking off -- following the user's prompts
It's not been long since Apple was being described as behind the rest of the technology industry over its adoption of AI. That was always nonsense because Apple's Machine Learning has been key to iOS for years, but then researchers at the company published academic papers including one on AI avatars.
Now another research paper has been published, and this time a trio of Apple researchers have been investigating and testing an app for "empowering animation design using Large Language Models." Called Keyframer, the AI app lets users describe an animation, and it then generates CSS animation code for websites.
Keyframer has not been released publicly, and its testing appears to have been quite limited. The three researchers, Tiffany Tseng, Ruijia Cheng, and Jeffrey Nichols, write that their study was based chiefly on 13 participants.
Those participants began by writing a plain-English description of what image they wanted. So far this is how Adobe Firefly AI works, too.
However, with Firefly and similar existing apps, once an image has been generated, the user can only use the app's manual controls to adjust or enhance it. What Apple's Keyframer was designed to do was let the users iterate through designs by continuing to describe what they need, or what they want removed.
Specifically, the paper describes previous attempts at generative AI image work as "one-shot prompting interfaces." In comparison, Keyframer was built so that a user could continue prompting multiple times on the same image.
Detail from the research paper showing code generated automatically after a user's description
"This is just so magical because I have no hope of doing such animations manually...," one novice participant said after using Keyframer. "I would find [it] very difficult to even know where to start with getting it to do this without this tool."
"Part of me is kind of worried about these tools replacing jobs, because the potential is so high," a professional animator told the researchers. "But I think learning about them and using them as an animator -- it's just another tool in our toolbox."
"It's only going to improve our skills," he or she continued. "It's really exciting stuff."
While the research paper -- a 31 page, 16,000 word document -- has been published, Keyframer itself has not been released and is solely an in-house testing app.
Its existence, however, backs up claims that Apple has been extensively testing generative AI. It's rumored that Apple will unveil significant AI-related improvements to the likes of iOS and Siri, at WWDC 2024.
Read on AppleInsider
Comments
This one hasn't really been tested yet, just 13 people have given it a try, but Apple is already announcing it? I've always seen them as keeping their internal development quiet and under-the-radar and competitors guessing, before announcing to the world what they're up to with a consumer-ready feature all but complete. Rather than these PR pieces intended to "delight the customer" it seems more likely this is meant to delight the market.
Machine learning analyzes existing data to find patterns and make predictions based on it. Generative AI is focused on generating new data.
I like this article that explains it.
https://medium.com/@sandesh.shinde/machine-learning-vs-generative-ai-whats-the-deal-159e690b8acf#:~:text=Data vs.,create stuff without needing them.
Obviously that includes Apple.
Yes, the arrival of NPUs on phones was important. Things got accelerated. They got accelerated for everyone with an interest.
When people say Apple is behind, it's because they haven't brought what others have brought to market.
What does saying 'Apple has been using ML for years' mean?
That others haven't?
Of course not. Was Apple doing more than others with ML? I very much doubt it.
Apple was not even performing well with its initial ML efforts (especially in image recognition which was where most of the onus was at the start).
Let me ask you directly:
Where was the DaVinci equivalent?
Where was the Ascend equivalent?
Where was the MindSpore equivalent?
Where was the CANN equivalent?
Where was the inference hardware?
Where was the GOD network equivalent?
That all dates back to 2018/19 and from just one company.
Then add the progress of Nvidia, Google, Microsoft...
Or Amazon and Meta.
How did Nvidia get where it is today? By releasing solutions.
So now 'AI' is the 'buzzword' (for better or worse) and it is on everybody's lips.
That's tech for you.
We can't ignore that. We can't ignore that solutions (both consumer and industry facing) have come to market and been developed at incredible speed.
And when I say 'industry' I'm including health, science, education etc.
What has Apple actually delivered when compared to what has been on the market for a while now?
And when will an equivalent Apple release actually become reality?
Reveal at WWDC? Yes!
Deployment at the end of the year? Probably.
Beta for months? Possibly.
That is not now. That is not today. That is why people are saying Apple is behind - now.
Research? Great! But who isn't involved in research?
Open source? The same answer.
I'm open to everything changing quickly. That is technology for you, but right now, and looking at what's available, trying to argue Apple isn't behind is not corresponding to what is actually on the market. To reality.
IMO, it is precisely why, after ignoring the term deliberately during WWDC, Apple now finds it necessary to talk about it, finally using the dreaded term, even before a usable consumer facing product is here.
That's not very Apple. The VP reveal wasn't either. But there you go.
That article goes to great lengths to position ML and GenAI as two separate things, but it's wrong.
GenAI is built on top of ML. Think of it as a subset of AI, but a superset of ML.
GenAI draws on its learned corpus of knowledge to generate new data. That knowledge is acquired via ML. Otherwise, how would it know what a "cat" was?
Apple sponsoring conferences, working on ML since 2017, using AI in products etc has nothing to do with the claims of it being behind. That's because no one knows what is happening behind the scenes.
Not at Apple or anywhere else.
How could we know that?
Couldn't that be the strawman here?
What is unhinged is the notion that Apple working on something is justification that it is isn't behind.
How exactly is that supposed to work?
How are you supposed to compare things if you can't see them?
Would you accept Siri as an example?
Did they abandon work on Siri? No. We know that much. We know they continue to work on that.
Those AI conferences around the world (around 40 since 2019?) were largely focused on language and image recognition.
But how has Siri done as a result of all that investigation? For starters the only way for us to evaluate that is from a product that is actually released. So how is it doing today? All I see is most people saying it isn't up to the task. So much for research being the key!
The research - for us - is worthless because everyone researches. Apple wasn't alone at those conferences was it?
The research is only relevant when something comes of it. Only then can we - the consumers or industry watchers - evaluate it.
So people can actually formulate an opinion on Siri and compare it to other offerings.
That is why people say Apple is behind. They look at what's being offered as solutions and compare but they find there is nothing to compare to because Apple hasn't delivered anything yet in the areas that have been getting all the attention for the last couple of years.
Saying it's 'working on it' doesn't count. Sponsoring conferences doesn't count. Research without a final product doesn't count.
But then you slip in Apple already uses AI but people are only now realising it?
And I'm making strawman arguments?
Are people only now realising that other companies also use AI? Because, after all, they have been using it for years too.
That is why I gave you the 'equivalents' list.
How many years do you think Apple needs, because in all those years, competitors have been delivering solutions.
I slipped in the GOD network because it is AI in the automobile realm and is 'self' learning. It's shipping and clocking up millions of km of real world data.
The Apple equivalent here is a complete mystery.
All I've said is that the people making those claims are right. That is because Apple isn't, let's say, where the puck is!
I'm looking at what there is to see and not seeing the Apple equivalent. That is it.
They might turn that on its head tonight, tomorrow, next year or in ten years. The point was, is and will remain the same until it changes. Apple hasn't been providing that much in where things are now and that is why Tim Cook cannot point to an Apple solution. It's why he basically said 'watch this space'.
All I know is that this year's WWDC is likely to an AI lovefest, just like the 5G modem presentation was a few years back.
When everyone was distracted by the iPhone 6s having a 64 bit processor, they missed the fact that that same phone was taking 100 photos, analysing them pixel by pixel, then stitching the best photos together to make a great photo. All that in the time it took to co “click”. That required a lot of machine learning which is why we needed 64 bit processors on a phone.
Before that Apple had Facial Recognition which scanned our images and learned to recognise faces and build an album for those people. Yes, back then we had to train it, even today, but it got better the more photos we took.
That facial recognition led to being able to identify other objects which led to the fix of one of tech’s biggest problems so far… OCR (Optical Character Recognition). Now you can select text from an image and copy and paste it. Hell, I use it to translate the data on my wife’s car which is a Japanese import.
My point is that this is ALL AI and it’s all been done by Apple for at least 10 years in people’s hands since then.
You might not see that as advancement in AI because it’s all back end stuff hidden in a great user interface. You don’t have to interact with the AI but you’ve got it right there and have so for a long time.
This is what people are saying. Apple isn’t behind in AI, it’s more than not ahead of the game, we just don’t recognise it.
My comments can pick up on what you say but remain within the broader topic and nested comments (along with the comment of the article).
I was working from memory when I said 'ML' instead of 'AI' so sorry for that.
Technically that would be more accurate from a quoting perspective but would not change anything in the opinion I was giving.
That opinion tackles your point and I expanded on why I think that is the case. That is not a 'rant'. That also includes the other points in the article, like Apple being behind.
Your answer was not solely about previous research efforts by Apple. It was about shipping technologies. You added:
"The reality is Apple has been including AI features in their products for almost a decade. They have built processors specifically to handle AI. They have published research on AI. And they have bought more AI companies than anyone else.
Largely correct and no different to others (but how do you know they've bought more AI companies than anyone else?) but my point, and this is key, is that that is irrelevant and here (as per the article thread you are posting in) and I explained why.
IMO, it's also incorrect that people are just starting to pay attention to what Apple already has.
You say people are 'jumping to conclusions that Apple is behind and is publishing research in an attempt to get in on the action'
I say the reality is that Apple is behind at the moment and I haven't noted people claiming that research papers are an attempt to get in on the action.
And that wasn't my interpretation of what Gatorguy said either, BTW.
Whenever the recent claims that 'Apple is behind in AI' are banded about, all I've seen is people referring to Apple not having the latest buzzword AI features when compared to its competitors.
Are those claims correct? Because when people say Apple is behind, that is what they mean.
What Apple is shipping is irrelevant. The 'behind' claims are concerned with what Apple is not shipping and hasn't shipped in all these years while competitors have.
I'm not saying that's a 'good' or 'bad' thing in itself. I'm stating a factual situation and when you look at the bigger AI picture there is a lot Apple still isn't doing.