Apple acquires "dark data" specialist Lattice Data for $200M

AppleInsider

In its latest buy, Apple has paid around $200 million to acquire Lattice Data Inc., a specialist in using machine learning to process "dark data" to efficiently build structured data sets that can be analyzed.

The buy was reported by Tech Crunch, and Apple responded with its usual boilerplate confirmation that it "buys smaller technology companies from time to time and we generally do not discuss our purpose or plans."


Lattice, based in Menlo Park, California, was commercializing a Stanford University research project known as "DeepDive," acting as a framework for statistical inference. The firm's website says "our mission is to unlock the value in dark data for critical real-world problems."

"Dark Data" pertains to the mountains of raw information collected by in various ways (such as logs or photos) but which remains difficult to analyze.

Lattice built a platform of proprietary software on top of Stanford's open source DeepDive technology, which had been developed over a six year period with $20 million of DARPA funding, aimed at rapidly make sense out of unstructured data.

DeepDive is aimed at "extracting value" from various "dark data" sources, serving, as the firm's site explains, as a "programming and execution framework for statistical inference, which allows us to solve data cleaning, extraction, and integration problems jointly. We model the known as features and the unknown as random variables connected in a factor graph."

Lattice chief scientist Christopher R won a MacArthur Genius Grantfor his work on DeepDive, and cofounded the firm with Michael Cafarella (who also co-founded Hadoop), Raphael Hoffmann andFeng Niu.

The company contrasts its approach to traditional machine learning, explaining "we do not require laborious manual annotations. Rather, we take advantage of domain knowledge and existing structured data to bootstrap learning via distant supervision. We solve data problems with data."

The firm also emphasizes the "machine scale" of its platform in being able to "push the envelope on machine learning speed and scale," resulting in "systems and applications that involve billions of webpages, thousands of machines, and terabytes of data" at what it describes as "human-level quality."

While Apple doesn't usually comment on why it buys up talent and technology, Lattice was reportedly shopping itself around as a solution to enhancing voice-based assistance, and had been in talks with Amazon's Alexa team as well as Samsung.

The applications that Lattice outlines on its website also suggest potential use in analyzing data for use in Maps and self driving vehicles; HealthKit and ResearchKit; camera logic and processing as well as Internet data document search.

hmurchison

Apple is really serious about this AI stuff. 


Dark Data though?   Isn't that kind of racist?

wonkothesane

hmurchison said:

Apple is really serious about this AI stuff. 


Dark Data though?   Isn't that kind of racist?

According to Gartner, which originally coined the term, dark data is defined as, "the information assets organizations collect, process and store during regular business activities, but generally fail to use for other purposes."Dec 8, 2014

firelock

hmurchison said:

Apple is really serious about this AI stuff. 


Dark Data though?   Isn't that kind of racist?

Average words like "dark" or "white" can only be racist if used in a racist context. If I say, "It's really getting dark outside," and I am referring to the fact that night is coming, there is no racist context. Obviously there are some words that are racist by nature but "dark" is not one of them.

tallest skil

wonkothesane said:
dark data is defined as, "the information assets organizations collect, process and store during regular business activities, but generally fail to use for other purposes."

So what does Apple want with otherwise worthless data? If it was valuable metadata, it wouldn't sit unused.

randominternetperson

tallest skil said:

wonkothesane said:
dark data is defined as, "the information assets organizations collect, process and store during regular business activities, but generally fail to use for other purposes."

So what does Apple want with otherwise worthless data? If it was valuable metadata, it wouldn't sit unused.

And you're joking about the first part (I hope). 

tallest skil

randominternetperson said:

And you're joking about the first part (I hope). 

How is 'regular business activities' defined, then? Maybe I misunderstand.

cornchip

Seems like this could come in handy for Apple in many ways.

foggyhill

Dark data, taking an extreme example is like a company collecting skin samples for decades (incidentally) and suddenly finding extra worth in it because tech/ai/processing speed/whatever now enables you to get more value out of it than the cost it takes to process this..

Lots of what's counted as business analytics these days would have been considered "dark" decades ago.

AI can find correlations, cause and effects, in groups, or solo data to other groups or solo bits of data that would be very hard to surmize from just looking at it (probable that the effort just to find what to look for would be too large to even be profitable before).

kevin kee

I supposed there are a lot of data floating around out there, and normally an AI pick up the one that they can use and ignore the one that they "deemed" worthless. Apple is interested in the later part because as Foggy said, they found its worth to get more value out of it, for whatever purpose. One example is medical data. Another is image recognition. Apple is very selective in acquiring other business, so I expect this to be another good one for their R&D investment in a long term especially in AI.

randominternetperson

tallest skil said:

randominternetperson said:

And you're joking about the first part (I hope). 

How is 'regular business activities' defined, then? Maybe I misunderstand.

Another example: a business records every transaction it conducts, but if it's only used to conduct that transaction, it's "dark data."  If you use it to determine what product to recommend to which customer or which products sell best at what time of day/week/year, etc. you're putting dark data to use.  Data is only "worthless" if you can't put it to use.  

tallest skil

randominternetperson said:

If you use it to determine what product to recommend to which customer or which products sell best at what time of day/week/year, etc. you're putting dark data to use.

Oh! They’re not doing that already? I thought that’d be SOP. Thanks for the clarification.

Fatman

Gartner is obviously not a marketer. The alliteration works but they used the wrong word. 

melgross

tallest skil said:

wonkothesane said:
dark data is defined as, "the information assets organizations collect, process and store during regular business activities, but generally fail to use for other purposes."

So what does Apple want with otherwise worthless data? If it was valuable metadata, it wouldn't sit unused.

Data is only worthless if you can't find a use for it. No data is inherently useless. All data has some use. It's just that it may not be known, at the time, how to extract that which would make it useful.

bells

firelock said:

hmurchison said:

Apple is really serious about this AI stuff. 


Dark Data though?   Isn't that kind of racist?

Average words like "dark" or "white" can only be racist if used in a racist context. If I say, "It's really getting dark outside," and I am referring to the fact that night is coming, there is no racist context. Obviously there are some words that are racist by nature but "dark" is not one of them.

Intent is what is important. 

Soli

firelock said:
 Obviously there are some words that are racist by nature but "dark" is not one of them.

I'd argue that no word is racist by nature. A string of letters or series of phonemes have no innate meaning. We have to give all words meaning which means we can choose to be offended (or not) by words. We see language evolve many times over in our lifetimes.

Can you think of any terms that weren't considered derogatory by society when you were a kid that are now demonized? I can think of plenty. It also works the other way, but to a lesser extent in the short term. For example, calling someone a scalawag isn't likely to offend anyone today, but not that long ago in American history that was one of the most offensive terms you could call certain groups.

We're wired poorly to accept things as being absolute when it's, at best, based on fundamental limitations of our species, or, at worst, actually random, but we all still perceive things about language that aren't actually real. Ideasthesia covers this phemonomen.

• https://en.wikipedia.org/wiki/Ideasthesia

One popular psychological example are the names of these shapes:

• https://en.wikipedia.org/wiki/Bouba/kiki_effect

Rayz2016

randominternetperson said:

tallest skil said:

randominternetperson said:

And you're joking about the first part (I hope). 

How is 'regular business activities' defined, then? Maybe I misunderstand.

Another example: a business records every transaction it conducts, but if it's only used to conduct that transaction, it's "dark data."  If you use it to determine what product to recommend to which customer or which products sell best at what time of day/week/year, etc. you're putting dark data to use.  Data is only "worthless" if you can't put it to use.  

An excellent example, and probably the most common use. 

Lots of sales related data is stored during the day, but not much of it is analysed in different contexts. A retail chain in the U.K. spent a lot money on one of these analysis package, and as a result the began to move items around the store depending on the time of day. For example, they moved beer and nappies to the front of the shop after 8pm and increased sales on both items. 

These packages can can also show historical data across many regions that they can also link to events that may be missed by the marketing analysts. 

Analyst: I need to know more about the sales spikes in condoms and toilet paper we are seeing in different regions at different times of the year. What's the connection?

Software: Music festivals. 

mattinoz

So could bootstrap learning be used to say create a private search engine per user?

improve Siri and search in general by deep diving the users data into a private encrypted database. Similar to photos image recognition system but using iCloud syncing to open that data and more across devices and app silos. 

Soli

mattinoz said:

So could bootstrap learning be used to say create a private search engine per user?

improve Siri and search in general by deep diving the users data into a private encrypted database. Similar to photos image recognition system but using iCloud syncing to open that data and more across devices and app silos. 

I'd assume so, but it already does that.

Rayz2016

mattinoz said:

So could bootstrap learning be used to say create a private search engine per user?

improve Siri and search in general by deep diving the users data into a private encrypted database. Similar to photos image recognition system but using iCloud syncing to open that data and more across devices and app silos. 

Here's something I've never quite understood about Apple's strategy.

All the processing is done on the device (which is why the photo library on my phone is out of step with the photo library on my iPad), but if Apple doesn't want to put this data in the cloud, then what happens to all this learned stuff when I get a new iPhone? Does it have to learn it all again?

mattinoz

Soli said:

mattinoz said:

So could bootstrap learning be used to say create a private search engine per user?

improve Siri and search in general by deep diving the users data into a private encrypted database. Similar to photos image recognition system but using iCloud syncing to open that data and more across devices and app silos. 

I'd assume so, but it already does that.

I thought they still relied on Developers surfacing useful information via spotlight plugins.

Rayz2016 said:

mattinoz said:

So could bootstrap learning be used to say create a private search engine per user?

improve Siri and search in general by deep diving the users data into a private encrypted database. Similar to photos image recognition system but using iCloud syncing to open that data and more across devices and app silos. 

Here's something I've never quite understood about Apple's strategy.

All the processing is done on the device (which is why the photo library on my phone is out of step with the photo library on my iPad), but if Apple doesn't want to put this data in the cloud, then what happens to all this learned stuff when I get a new iPhone? Does it have to learn it all again?

I assumed it was due to new file system coming soon. As in Multi-key encryption would allow them share this sort of data and still cut off devices as needed for security reason.