Apple details new 'AppleBot' web crawler used by Siri and Spotlight

Posted:
in General Discussion edited July 2015
Apple has confirmed the existence of a long-rumored web crawling service -- first noticed last November --?and provided some details of its operations in a recently-updated support document.




According to Apple, AppleBot is a crawler that primarily serves Siri and Spotlight. Both services offer web search capabilities, though until now it was thought that Apple routed queries through an existing search engine such as Google or Bing.

AppleBot operates similarly to Google's Googlebot, and Apple advises developers that it will follow Googlebot instructions if no AppleBot-specific instructions are found. Website operators are urged to contact Apple's network operations center if they have questions or concerns about the crawler.

It's not clear how long AppleBot has been operational in production. The support document first appeared in February, but was only populated with content last month.

One web developer spotted Apple's crawler last November, after seeing numerous hits from a crawler in the 17.0.0.0/8 block. That block is assigned to Apple, one of just a handful of companies that control an entire A block.
«1345

Comments

  • Reply 1 of 82
    Interesting
  • Reply 2 of 82
    MacProMacPro Posts: 16,950member
    Maybe 'search' without selling ads is coming? If Apple ever do release this technology as a fully fledged search tool for the great unwashed I wonder what it would be called. Maybe just 'Siri'?

    I truly hope Apple do this and free us from the warped and biased searches Google now offer. (Bait ... )
  • Reply 3 of 82
    mubailimubaili Posts: 371member
    Wow. Apple has really opened up in some area, much better than some other "open" company, yea, Google, I am talking about you.
  • Reply 4 of 82
    MacProMacPro Posts: 16,950member
    mubaili wrote: »
    Wow. Apple has really opened up in some area, much better than some other "open" company, yea, Google, I am talking about you.

    Don't worry, Google is listening to you ... trying to figure out what ads to send you ... lol
  • Reply 5 of 82
    BornSlippyBornSlippy Posts: 57member
    Quote:

    Originally Posted by mubaili View Post



    Wow. Apple has really opened up in some area, much better than some other "open" company, yea, Google, I am talking about you.

    What are you talking about? So because Apple is using their own Applebot to fulfill Siri queries that means they opened up? 

     

    There's absolutely nothing wrong with Google. 

  • Reply 6 of 82
    slprescottslprescott Posts: 726member
    Sounds like the early days of Apple Maps: an in-house capability ramping up to reduce Apple's dependency on Google.

    Does Google have any legal control over the use of "Googlebot instructions"? I.e., is Apple allowed to have its own search technology use them, or does Google have any control over this? For example, do Googlebot instructions follow a particular syntax that Google may have protected as intellectual property?
  • Reply 7 of 82
    MathieuLLF wrote: »
    There's absolutely nothing wrong with Google. 
    They are perfect and anyone who questions anything they do is a stupid head.
    Their motto is "do no evil"...so there! They can't be evil!
  • Reply 8 of 82
    BornSlippyBornSlippy Posts: 57member
    Quote:

    Originally Posted by Johnny Mozzarella View Post





    They are perfect and anyone who questions anything they do is a stupid head.

    Their motto is "do no evil"...so there! They can't be evil!

    No corporation is perfect, neither is Apple.  

     

    Google has obviously made some missteps but no more or less than any other in their position. 

  • Reply 9 of 82
    MacProMacPro Posts: 16,950member
    This is going into my 'Hall of Shame' Gallery ;)
    MathieuLLF wrote: »

    There's absolutely nothing wrong with Google. 
  • Reply 10 of 82
    MacProMacPro Posts: 16,950member
    They are perfect and anyone who questions anything they do is a stupid head.
    Their motto is "do no evil"...so there! They can't be evil!

    'absolutely perfect' :D
  • Reply 11 of 82
    thrangthrang Posts: 666member
    I said years ago that Apple is developing an alternative search engine...the overbuilding of all the server farms was one indication, but hiding in plain sight was the introduction of Spotlight, which allows Internet searches from within the application (no ads). And Sherlock was of course a very early predecessor of the same concept.

    Given the importance of search on the web, relying on others is not a good model. Even if they don't launch it for a while, its presence can give them leverage...
  • Reply 12 of 82
    BornSlippyBornSlippy Posts: 57member
    Quote:

    Originally Posted by digitalclips View Post



    This is going into my 'Hall of Shame' Gallery image

    Why's that? Google makes an amazing search engine among other products. 

     

    I tried using Duck Duck Go but it sucks for anything local. Google just provides me with the best search results for my purposes.

  • Reply 13 of 82
    tmaytmay Posts: 2,336member
    Quote:

    Originally Posted by MathieuLLF View Post

     

    Why's that? Google makes an amazing search engine among other products. 

     

    I tried using Duck Duck Go but it sucks for anything local. Google just provides me with the best search results for my purposes.


    ...and also all of the baggage that goes with an advertising company. Lots of baggage.

     

    There's room for yet another search business model, and maybe Apple has/will find it.

  • Reply 14 of 82
    thrangthrang Posts: 666member
    MathieuLLF wrote: »
    Why's that? Google makes an amazing search engine among other products. 

    I tried using Duck Duck Go but it sucks for anything local. Google just provides me with the best search results for my purposes.

    It's not about their search algorithms...it's about tracking every breath you take...its insidious...

    You have the choice to use it, and until there is something better, most will. But you should be surprised that many would prefer a viable alternative.
  • Reply 15 of 82
    MacProMacPro Posts: 16,950member
    thrang wrote: »
    I said years ago that Apple is developing an alternative search engine...the overbuilding of all the server farms was one indication, but hiding in plain sight was the introduction of Spotlight, which allows Internet searches from within the application (no ads). And Sherlock was of course a very early predecessor of the same concept.

    Given the importance of search on the web, relying on others is not a good model. Even if they don't launch it for a while, its presence can give them leverage...

    I've been saying the same thing for years too. I seem to recall a few years ago that ideas was shot down by many in AI for reasons such as 'Apple couldn't compete with Google' or 'They didn't have the technology' and even 'Why would Apple want to?'

    My answer to all these is, given the trust Apple has out there by its users, if Apple offer search without strings it will be another massive blow to Google's ability to make money off iOS and OS X users just as Maps has been. That's reason enough for me. If Apple's search is the default most will use it IMHO.
  • Reply 16 of 82
    BornSlippyBornSlippy Posts: 57member
    Quote:

    Originally Posted by tmay View Post

     

    ...and also all of the baggage that goes with an advertising company. Lots of baggage.

     

    There's room for yet another search business model, and maybe Apple has/will find it.


     

    What baggage may that be? They display non intrusive ads. If they really bother you, you can use an ad blocker and the problem is solved. 

     

    Microsoft has pumped billions of $ into search and even they can't find their footing in search, I highly doubt Apple will be able to either other than with a very niche market. Google makes an incredible product and that's why people use it.

  • Reply 17 of 82
    BornSlippyBornSlippy Posts: 57member
    Quote:

    Originally Posted by thrang View Post





    It's not about their search algorithms...it's about tracking every breath you take...its insidious...



    You have the choice to use it, and until there is something better, most will. But you should be surprised that many would prefer a viable alternative.

    There are alternatives and people aren't using them. Google for now just has the best working solution. 

  • Reply 18 of 82
    BornSlippyBornSlippy Posts: 57member
    Quote:

    Originally Posted by digitalclips View Post





    I've been saying the same thing for years too. I seem to recall a few years ago that ideas was shot down by many in AI for reasons such as 'Apple couldn't compete with Google' or 'They didn't have the technology' and even 'Why would Apple want to?'



    My answer to all these is, given the trust Apple has out there by its users, if Apple offer search without strings it will be another massive blow to Google's ability to make money off iOS and OS X users just as Maps has been. That's reason enough for me. If Apple's search is the default most will use it IMHO.

    Most iOS users maybe, but that's really just a small niche compared to the full market out there. Also who's to say that most people wouldn't just switch back to Google? People care about the quality of the search results rather than who is providing it. If Apple does release a search it had better be better than the launch of Apple Maps which was one of the most botched launches ever. Apple isn't a services company but rather a hardware and software company. 

  • Reply 19 of 82
    MacProMacPro Posts: 16,950member
    MathieuLLF wrote: »
    Why's that? Google makes an amazing search engine among other products. 

    I tried using Duck Duck Go but it sucks for anything local. Google just provides me with the best search results for my purposes.

    LOL nothing personal ... But why? "There's absolutely nothing wrong with Google. " is a pretty bold statement that's why. I collect these from AI. Years later they are fun to trot out. You know, the sort of thing I mean, like this iPhone related quote ... "It’s kind of one more entrant into an already very busy space with lots of choice for consumers ... But in terms of a sort of a sea-change for BlackBerry, I would think that’s overstating it."
  • Reply 20 of 82
    wizard69wizard69 Posts: 12,282member
    :\
    Maybe 'search' without selling ads is coming?
    Or maybe search with with Apple ads????.
    If Apple ever do release this technology as a fully fledged search tool for the great unwashed I wonder what it would be called. Maybe just 'Siri'?
    Certainly Siri is a component of search or maybe search is a component of Siri. Either way Apple has been making improvements to Siri continuously. It would make sense to have control over how searches are done as the AI improves in Siri. The AI could generate very specific types of search that optimized for an engine Apple controls. An Apple web crawler would allow them to structure the data for the AI and search mechanisms in a way you can't do searching with Google.
    I truly hope Apple do this and free us from the warped and biased searches Google now offer. (Bait ... )

    Well,Google won't go away. The fact is I'm often searching for very specific things of economic interests. That is a good part of search is to enable buying something. So in that regard Google serves my needs very well. At least for work related searches.

    Of course not all searching is related to commerce. This is where Apple could leverage its own search engine to keep Siri AI like. Could you imagine Siri answering searches with spoken ads. Sickening isn't it.
Sign In or Register to comment.