Evidence shows Apple operating a mysterious Web crawling bot
A developer has discovered that Apple appears to be crawling HTML websites with its own automated bot, but the exact purpose of the mysterious homegrown software remains unknown.
Evidence of Apple's Web crawler, written in the Go programming language, was first discovered by developer Jan Moesen, who has found hits from the software dating back to Oct. 15. Others have discovered significant traffic to their websites from Apple's mysterious crawler, which appears to be shut off from time to time.
Requests from Apple's bot are only for HTML pages, and do not access the accompanying CSS, JavaScript or image files from a website. Moesen found that the Apple Web crawler identifies itself as "Mozilla/5.0 (compatible; Fetcher/0.1)".
The crawler was found originating from an IP address starting with "17.", which would indicate that the bot is emanating from Apple's servers.
It's unknown whether the Apple bot is an official project for the company, or if an employee built and ran the system in an unofficial capacity from a company machine.
It's possible that Apple's efforts could be related to Spotlight enhancements in both OS X Yosemite and iOS 8. Both were updated this fall with improvements that allow users to access Web search content directly, without the need to access a third-party search tool such as Google or Bing.
Though Apple doesn't compete directly with the likes of Google or Microsoft Bing, the company has been working to provide specific information directly within Siri search results. Doing so makes it unnecessary to provide potential links through the official Siri Web search partner, Bing.
Evidence of Apple's Web crawler, written in the Go programming language, was first discovered by developer Jan Moesen, who has found hits from the software dating back to Oct. 15. Others have discovered significant traffic to their websites from Apple's mysterious crawler, which appears to be shut off from time to time.
Requests from Apple's bot are only for HTML pages, and do not access the accompanying CSS, JavaScript or image files from a website. Moesen found that the Apple Web crawler identifies itself as "Mozilla/5.0 (compatible; Fetcher/0.1)".
The crawler was found originating from an IP address starting with "17.", which would indicate that the bot is emanating from Apple's servers.
It's unknown whether the Apple bot is an official project for the company, or if an employee built and ran the system in an unofficial capacity from a company machine.
Does anyone know why Apple is running a web crawler (written in @golang, no less)? http://t.co/5JDTZuplQp
-- Jan Moesen (@janmoesen) It's possible that Apple's efforts could be related to Spotlight enhancements in both OS X Yosemite and iOS 8. Both were updated this fall with improvements that allow users to access Web search content directly, without the need to access a third-party search tool such as Google or Bing.
Though Apple doesn't compete directly with the likes of Google or Microsoft Bing, the company has been working to provide specific information directly within Siri search results. Doing so makes it unnecessary to provide potential links through the official Siri Web search partner, Bing.
Comments
Competition for Google, obviously.
Also, I always think its cool that Apple owns a whole Class A IP block. Odd that MS never requested one.
Apple launching a search engine would be Steve's ultimate revenge.
John C Dvorak's prediction coming true?
How Long Before Apple Gets a Search Engine?
http://www.pcmag.com/article2/0,2817,2458892,00.asp
Yeah, I know.. you guys will probably hate me for bringing up the guy that said the mouse will never catch on.
:-D NoAgendaShow FTW
Written in Go? That's alone has to make this one of the most unexpected stories of the year!
World domination.
Ping is coming back in another incarnation.
In reality there are 2 Go languages.
Frustrating that I can never follow links from the ai app on my iphone
Do you think Apple is moving in that direction? If they were, I'd first expect to see some lateral movement that sets a foundation for later access. For example, making Spotlight much smarter so that we can do a search without having to go into the URL field in Safari. I don't see that happening for at least several years, and I'd first like them make the maps.apple.com URLs they create in their Maps app do more than convert to a Google Maps link when it hits their web server.
OT: On the subject of what 'I' want, I can't believe Apple doesn't use a URL shortener for Messages. URLs can very long. All it would have to do is look for links pasted that are over n<sub>1</sub> characters and have more than n<sub>2</sub> ampersands to tell it to slide up a screen with options to shorten the link or not. I think it could be that simple. They don't even have to buy a new domain. Even link.apple.com/?blah would be good enough or using Bitly (or a similar service).
I say this as they still haven't gotten the pasting of URLs in Messages down properly. With iOS 8 if there are hyperlinks with titles in a section of text they will pull out the URLs, put them in greater-than and less-than symbols following the title text for the hyperlink, but if you simply copy and paste a titled URL from Messages in Mac OS X it will be the formatted text being sent so only someone using Messages on a Mac will be able to open the link while those on iOS will only see it as plaintext. I see this as being unusually sloppy for Apple.
It's happened. Siri is now self aware.
That is exactly where I mind when when I read this.
Apple owns the entire 17 block? I didn’t know that.
Do you think they bought up a section of IPv6 addresses early?
I like it, but Spotlight would have to actually WORK first.
What bothers me ...
With an iPad (I'm using an iPad 4) go to:
daringfireball.net -- count the seconds
Then
Recode.net -- count the seconds
Then
appleinsider.com -- count the seconds
Then
Just reload this page -- count the seconds
It's possible that Apple's efforts could be related to Spotlight enhancements in both OS X Yosemite and iOS 8. Both were updated this fall with improvements that allow users to access Web search content directly, without the need to access a third-party search tool such as Google or Bing.
Though Apple doesn't compete directly with the likes of Google or Microsoft Bing, the company has been working to provide specific information directly within Siri search results. Doing so makes it unnecessary to provide potential links through the official Siri Web search partner, Bing.
This makes sense. Spotlight being the beginnings to ?search.
Tie in Internet Search into Spotlight. After Google created Android, I thought Apple would immediately start working on developing a search engine. I am still surprised we haven't seen anything more comprehensive than results from curated sites.
The video demo tour shows the speaker building a simple web crawler demo.
http://research.swtch.com/gotour
This may just be some competitive research being done at Apple with someone in the Swift team checking out Go.
Written in Go?? A Google programming language? Weird stuff. . .
Google designed it to write stuff like search engines.
Not surprising at all that Apple would use Go. It's a really good language for doing stuff in parallel.