Evidence shows Apple operating a mysterious Web crawling bot

Posted:
in General Discussion edited May 2015
A developer has discovered that Apple appears to be crawling HTML websites with its own automated bot, but the exact purpose of the mysterious homegrown software remains unknown.




Evidence of Apple's Web crawler, written in the Go programming language, was first discovered by developer Jan Moesen, who has found hits from the software dating back to Oct. 15. Others have discovered significant traffic to their websites from Apple's mysterious crawler, which appears to be shut off from time to time.

Requests from Apple's bot are only for HTML pages, and do not access the accompanying CSS, JavaScript or image files from a website. Moesen found that the Apple Web crawler identifies itself as "Mozilla/5.0 (compatible; Fetcher/0.1)".

The crawler was found originating from an IP address starting with "17.", which would indicate that the bot is emanating from Apple's servers.

It's unknown whether the Apple bot is an official project for the company, or if an employee built and ran the system in an unofficial capacity from a company machine.

Does anyone know why Apple is running a web crawler (written in @golang, no less)? http://t.co/5JDTZuplQp

-- Jan Moesen (@janmoesen)


It's possible that Apple's efforts could be related to Spotlight enhancements in both OS X Yosemite and iOS 8. Both were updated this fall with improvements that allow users to access Web search content directly, without the need to access a third-party search tool such as Google or Bing.

Though Apple doesn't compete directly with the likes of Google or Microsoft Bing, the company has been working to provide specific information directly within Siri search results. Doing so makes it unnecessary to provide potential links through the official Siri Web search partner, Bing.
«13456

Comments

  • Reply 1 of 104

    Competition for Google, obviously.

  • Reply 2 of 104
    gatorguygatorguy Posts: 24,176member
    Written in Go?? A Google programming language? Weird stuff. . .
  • Reply 3 of 104
    It's happened. Siri is now self aware.






    Also, I always think its cool that Apple owns a whole Class A IP block. Odd that MS never requested one.
  • Reply 4 of 104
    Competition for Google, obviously.

    Apple launching a search engine would be Steve's ultimate revenge. :D
  • Reply 5 of 104
    asdasdasdasd Posts: 5,686member
    Possibly spotlight.
  • Reply 6 of 104

    John C Dvorak's prediction coming true? 



    How Long Before Apple Gets a Search Engine?

    http://www.pcmag.com/article2/0,2817,2458892,00.asp

     



    • JUNE 4, 2014



      Yeah, I know.. you guys will probably hate me for bringing up the guy that said the mouse will never catch on. 



      :-D NoAgendaShow FTW


  • Reply 7 of 104
    crowleycrowley Posts: 10,453member

    Written in Go?  That's alone has to make this one of the most unexpected stories of the year!

  • Reply 8 of 104
    pfisherpfisher Posts: 758member

    World domination.

     

    Ping is coming back in another incarnation.

  • Reply 9 of 104
    dasanman69dasanman69 Posts: 13,002member
    gatorguy wrote: »
    Written in Go?? A Google programming language? Weird stuff. . .

    In reality there are 2 Go languages.
  • Reply 10 of 104
    MacProMacPro Posts: 19,718member
    ?Search
  • Reply 11 of 104
    sestewart wrote: »
    John C Dvorak's prediction coming true? 


    How Long Before Apple Gets a Search Engine?
    http://www.pcmag.com/article2/0,2817,2458892,00.asp
     

    <ul style="border:0px;color:rgb(0,0,0);list-style-type:none;margin-left:0px;vertical-align:baseline;"><li style="border:0px;padding-right:20px;padding-top:0px;vertical-align:baseline;">JUNE 4, 2014


    Yeah, I know.. you guys will probably hate me for bringing up the guy that said the mouse will never catch on. 


    :-D NoAgendaShow FTW</li>

    </ul>

    Frustrating that I can never follow links from the ai app on my iphone
  • Reply 12 of 104
    solipsismysolipsismy Posts: 5,099member
    ?Search

    Do you think Apple is moving in that direction? If they were, I'd first expect to see some lateral movement that sets a foundation for later access. For example, making Spotlight much smarter so that we can do a search without having to go into the URL field in Safari. I don't see that happening for at least several years, and I'd first like them make the maps.apple.com URLs they create in their Maps app do more than convert to a Google Maps link when it hits their web server.


    OT: On the subject of what 'I' want, I can't believe Apple doesn't use a URL shortener for Messages. URLs can very long. All it would have to do is look for links pasted that are over n<sub>1</sub> characters and have more than n<sub>2</sub> ampersands to tell it to slide up a screen with options to shorten the link or not. I think it could be that simple. They don't even have to buy a new domain. Even link.apple.com/?blah would be good enough or using Bitly (or a similar service).

    I say this as they still haven't gotten the pasting of URLs in Messages down properly. With iOS 8 if there are hyperlinks with titles in a section of text they will pull out the URLs, put them in greater-than and less-than symbols following the title text for the hyperlink, but if you simply copy and paste a titled URL from Messages in Mac OS X it will be the formatted text being sent so only someone using Messages on a Mac will be able to open the link while those on iOS will only see it as plaintext. I see this as being unusually sloppy for Apple.
  • Reply 13 of 104
    maestro64maestro64 Posts: 5,043member
    Quote:

    Originally Posted by TheWhiteFalcon View Post



    It's happened. Siri is now self aware.

     

     

    That is exactly where I mind when when I read this.

  • Reply 14 of 104
    Originally Posted by AppleInsider View Post

    A developer has discovered that Apple appears to be crawling HTML websites with its own automated bot, but the exact purpose of the mysterious homegrown software remains unknown.

     


     

    Apple owns the entire 17 block? I didn’t know that.

     

    Do you think they bought up a section of IPv6 addresses early?

     

    Originally Posted by digitalclips View Post

    ?Search



    I like it, but Spotlight would have to actually WORK first.

     

  • Reply 15 of 104
    OT:

    What bothers me ...

    With an iPad (I'm using an iPad 4) go to:

    daringfireball.net -- count the seconds

    Then

    Recode.net -- count the seconds

    Then

    appleinsider.com -- count the seconds

    Then

    Just reload this page -- count the seconds
  • Reply 16 of 104
    paxmanpaxman Posts: 4,729member
    Quote:
    Originally Posted by AppleInsider View Post

    It's possible that Apple's efforts could be related to Spotlight enhancements in both OS X Yosemite and iOS 8. Both were updated this fall with improvements that allow users to access Web search content directly, without the need to access a third-party search tool such as Google or Bing.



    Though Apple doesn't compete directly with the likes of Google or Microsoft Bing, the company has been working to provide specific information directly within Siri search results. Doing so makes it unnecessary to provide potential links through the official Siri Web search partner, Bing.

     

    This makes sense. Spotlight being the beginnings to ?search.

  • Reply 17 of 104
    droidftwdroidftw Posts: 1,009member
    dasanman69 wrote: »
    In reality there are 2 Go languages.
    But isn't there only one golang?
  • Reply 18 of 104

    Tie in Internet Search into Spotlight. After Google created Android, I thought Apple would immediately start working on developing a search engine. I am still surprised we haven't seen anything more comprehensive than results from curated sites.

  • Reply 19 of 104

    The video demo tour shows the speaker building a simple web crawler demo. ;) 

     

    http://research.swtch.com/gotour

     

    This may just be some competitive research being done at Apple with someone in the Swift team checking out Go. 

  • Reply 20 of 104
    Quote:

    Originally Posted by Gatorguy View Post



    Written in Go?? A Google programming language? Weird stuff. . .



    Google designed it to write stuff like search engines.

     

    Not surprising at all that Apple would use Go. It's a really good language for doing stuff in parallel.

Sign In or Register to comment.