Apple details new 'AppleBot' web crawler used by Siri and Spotlight

124

Comments

  • Reply 61 of 82
    patsupatsu Posts: 430member
    matrix07 wrote: »
    Am I the only one who switched to Bing? I searched for "Beyoncé the met ball" and the result I got was more accurate than Google. ;)

    The latest news appear faster on Bing usually. I used to run some searches on both Bing and Google to see what's missing from either. ;-)
  • Reply 62 of 82
    suddenly newtonsuddenly newton Posts: 13,819member
    MathieuLLF wrote: »
    Why's that? Google makes an amazing search engine among other products. 

    I tried using Duck Duck Go but it sucks for anything local. Google just provides me with the best search results for my purposes.

    I agree that they do a good job ranking relevant results for most searches, but I still don't like them. As for "amazing" other products, no, they do not.
  • Reply 63 of 82
    dick applebaumdick applebaum Posts: 12,527member
    wizard69 wrote: »
    MathieuLLF wrote: »
    What baggage may that be? They display non intrusive ads. If they really bother you, you can use an ad blocker and the problem is solved. 
    Advertising generates huge amounts of waste. In the old days that amounted to tons of paper coming through your mailbox every day. These days advertisers burn up your bandwidth with fat ads that effectively waste your bandwidth.

    I don't know what the original poster was complaining about but the world of advertising get is a bit creepy.
    Microsoft has pumped billions of $ into search and even they can't find their footing in search, I highly doubt Apple will be able to either other than with a very niche market. Google makes an incredible product and that's why people use it.

    Apple won't take Google head on with search. Rather they will redefine it. Search will become an extension of Siri type technologies.

    As for Google they have their place in the world. The problem is they make every search attempt an opportunity to advertise. Sometimes you want that sort of info but other times you don't.

    Bandwidth, Bandwidth, Bandwidth ...

    Harken back to those thrilling days of yesteryear -- the late 1990s -- where you were lucky if you had a 96K baud modem connection to the Internet. My home in in the foothills around Tucson had 48K baud service (when it worked) -- yes, 4800 characters per second!

    In those days you designed a web page so that it:
    • would load in 20 seconds on a 24Kb modem
    • provided only the most important content -- say, an overview or summary
    • had links to secondary content -- detais
    • would popup a link in a separate page to avoid a reload when returning to the original page
    • possiblbly one or two small images per page
    • no distracting adds

    If you wanted to search for something, you went to one of several popular search sites or a search aggregator.

    The search results pages would also prudently conserve bandwidth, as described above.


    Then, everything changed with the introduction of "Banner Ads" at the top of some webpages ...


    I always thought these were stupid ... Why would an enterprise spend [then] 10s of thousands of dollars designing a web site to attract and hold customers' interest -- only to tempt them, with a prominent Banner Ad, to leave their site for another.


    Anyway, there are different things you can do with a web crawler/bot than just build ads/search results.

    For example, the bot could curate the site by separating actual content from the attendant cruft -- then repackage it as a short summary paragraph and follow-on detail paragraphs -- much like the 1990s web site described above,

    Why would you want to do this"

    Bandwidth, Bandwidth, Bandwidth ...


    ... Say, your using Siri on your Apple Watch or New AppleTV remote which has the limited bandwidth of, BLE, 811.g ...

    you can directly search/play a song or a movie from a home library of thousands ...

    You could search the web, at large, and quickly get a concise, to-the-point summary with no distractions -- and follow links to more details as desired.


    That's what I suspect Apple is doing, with this bot.
  • Reply 64 of 82
    mstonemstone Posts: 11,510member
    Quote:

    Originally Posted by AppleInsider View Post



    One web developer spotted Apple's crawler last November, after seeing numerous hits from a crawler in the 17.0.0.0/8 block. That block is assigned to Apple, one of just a handful of companies that control an entire A block.

    Controlling an 'A' block is nice but with IPV6 even individuals will eventually be able to control an equivalent number of IP addresses. I wish the telcos would hurry up and support it. We have a 'C' block plus another /26 but can't use IPV6.

  • Reply 65 of 82
    dick applebaumdick applebaum Posts: 12,527member
    mjtomlin wrote: »
    slprescott wrote: »
    Sounds like the early days of Apple Maps: an in-house capability ramping up to reduce Apple's dependency on Google.


    Does Google have any legal control over the use of "Googlebot instructions"? I.e., is Apple allowed to have its own search technology use them, or does Google have any control over this? For example, do Googlebot instructions follow a particular syntax that Google may have protected as intellectual property?

    It's not about control, it's about limiting access through an honor system. A web developer can add a 'robots.txt' file to any directory that is filled with instructions that basically tells these 'bots' what they can and can't do. Web crawlers are supposed to honor any settings in this file. Some web crawlers define their own instructions. Any other web crawler can also look for and abide by those rules - which is what Apple is saying their crawler does - they recognize any Google-made instructions and honor the rules set up by the developer/administrator.

    I've developed a couple of sites where the owner didn't want the site "on the Internet" - they didn't want their site to show up in any search results so I created this robots.txt file...

    User-Agent: *
    Disallow: /

    This basically tells all ( * ) robots/crawlers they are not allowed to traverse through the root ( / ) level of the site, which bars them from the entire site.

    On top of that there are HTML meta tags for robots specific to that page.

    Not, necessarily true ...

    You can write a bot/crawler that ignores the meta tags.

    There are lots ways to access a web site without using HTML or a browser.
  • Reply 66 of 82
    MacProMacPro Posts: 19,718member
    bobjohnson wrote: »
    Google was created as a research project when Larry and Sergey were at Stanford together. They actually tried to sell it to Excite (for $750,000) and only decided to make a real go of it when Excite didn't want them.  

    My point exactly, money was always in their minds.
  • Reply 67 of 82
    mstonemstone Posts: 11,510member
    Quote:

    Originally Posted by digitalclips View Post





    My point exactly, money was always in their minds.



    Money is on everyone's mind.

  • Reply 68 of 82
    mstonemstone Posts: 11,510member
    Quote:
    Originally Posted by Maestro64 View Post



    Yes they have the suggest as you type or spell correct as you type, these helpfully things are really tools to allow them to push you to sites they want you to see first.

    But not necessarily because of some financial interest. They are pointing you to pages that you probably want to see first. Yes it has been shown that they, at times, have fiddled with the rankings on some searches, but in general, I have found the organic search to be very fair, especially with our page rankings, which have steadily improved over the years by just being good pages.  We don't use Google AdWords or any sort of advertising yet we rank very favorably compared to competitors who do use Google advertising.

  • Reply 69 of 82
    thomprthompr Posts: 1,521member
    Quote:
    Originally Posted by MathieuLLF View Post

     

    No I agree that services are important, my point was just that so far Apple hasn't really shown any strength in their services. Apple Maps was a disaster (sure it's improving but it's still not all there), iCloud was useless at first and is slowly getting better but still lags behind others, etc. 

     

    Google doesn't sell any of your personal information. Your privacy is 100% respected, all they do is display ads which will match what advertisers are looking for but your personal info is never shared or sold to anyone. I'm perfectly fine with that. 


    If memory serves, Google got caught bypassing the user's privacy settings in Safari and gathering personal data anyway.

     

    This is significant enough, in my opinion, to  completely destroy your contentions that Google is just a tech company "doing its thing" and respects your privacy 100%.

  • Reply 70 of 82
    nolamacguynolamacguy Posts: 4,758member
    MathieuLLF wrote: »
    Google has obviously made some missteps but no more or less than any other in their position. 

    no more or no less? why is that? are all companies equal? google has chose to take certain missteps, like ignoring Do Not Track instructions -- is that no more or less than another company?
  • Reply 71 of 82
    nolamacguynolamacguy Posts: 4,758member
    MathieuLLF wrote: »
    Why's that? Google makes an amazing search engine among other products. 

    but that isnt what you said. you said there is nothing wrong with them. a bold, definitive, blanket statement.
  • Reply 72 of 82
    sockrolidsockrolid Posts: 2,789member

    Originally Posted by digitalclips View Post



    Maybe 'search' without selling ads is coming? If Apple ever do release this technology as a fully fledged search tool for the great unwashed I wonder what it would be called. Maybe just 'Siri'?



    I truly hope Apple do this and free us from the warped and biased searches Google now offer. (Bait ... )

     

    Would be a logical step for Apple.  I haven't used Google search for years, but apparently they put pay-for-placement ads at the top of the first page of hits, they spam you with their own services by jacking up Google sites' "Page ranking," and generally act like the unregulated monopoly search provider that they are.  

     

    Oh yeah, and Google also tracks the f__k out of all of their users.  To get higher ad revenue from their real customers: advertisers.

     

    I've used DuckDuckGo for years.  Web site and iOS app.  Will never go back to Google.

  • Reply 73 of 82
    gatorguygatorguy Posts: 24,176member
    sockrolid wrote: »
    Would be a logical step for Apple.  I haven't used Google search for years, but apparently they put pay-for-placement ads at the top of the first page of hits, they spam you with their own services by jacking up Google sites' "Page ranking," and generally act like the unregulated monopoly search provider that they are.  

    Oh yeah, and Google also tracks the f__k out of all of their users.  To get higher ad revenue from their real customers: advertisers.

    I've used DuckDuckGo for years.  Web site and iOS app.  Will never go back to Google.
    Try a couple of searches and see how the results are displayed. That way you'll know rather than guessing. It won't cost you anything, and probably not even attract a targeted ad. They won't know who you are if you have no Google account whereas some other companies with trackers (on this very page!) well may and worse be more than willing to sell it.
  • Reply 74 of 82
    MacProMacPro Posts: 19,718member
    wizard69 wrote: »
    :\
    Or maybe search with with Apple ads????.
    Certainly Siri is a component of search or maybe search is a component of Siri. Either way Apple has been making improvements to Siri continuously. It would make sense to have control over how searches are done as the AI improves in Siri. The AI could generate very specific types of search that optimized for an engine Apple controls. An Apple web crawler would allow them to structure the data for the AI and search mechanisms in a way you can't do searching with Google.
    Well,Google won't go away. The fact is I'm often searching for very specific things of economic interests. That is a good part of search is to enable buying something. So in that regard Google serves my needs very well. At least for work related searches.

    Of course not all searching is related to commerce. This is where Apple could leverage its own search engine to keep Siri AI like. Could you imagine Siri answering searches with spoken ads. Sickening isn't it.

    I had to chuckle at the thought of Siri (in a Google-esque alternative universe) saying ..."Ok I found that ... but first a word from one of my sponsors ..." then still in her usual voice "Beggin'Strips, the treat your dog can't tell from real bacon! Available from your corner Publix just near where you gas up your Jeep and buy your Gin." pause ... 'Ok back to where to find the nearest Emergency Room ..."
  • Reply 75 of 82
    jbdragonjbdragon Posts: 2,305member
    MathieuLLF wrote: »
    Most iOS users maybe, but that's really just a small niche compared to the full market out there. Also who's to say that most people wouldn't just switch back to Google? People care about the quality of the search results rather than who is providing it. If Apple does release a search it had better be better than the launch of Apple Maps which was one of the most botched launches ever. Apple isn't a services company but rather a hardware and software company. 

    I can't take all the ad's on Google!!!. It's crazy how many paid ad's get down to you. I use DuckDuckGo and it works quite well. In fact I was just a few weeks ago trying to find a manual for a 15 year old piece of electric requirement for industrial environments. All Google could find was some book type place, I couldn't download a copy. Ur was in this crappy book open look format. Hard to really look at and read let alone do much with. That's all Google gave me. On the other hand I then went to DuckDuckGo and right obj the top of the page, a link to grab a PDF copy of the manual which I downloaded and printed out and it's helped greatly!!!. Google is far from the be all end all of searches!!! I was kind of shocked with these results. I tried adv few times well Google and I had no luck. Pretty disappointing with that outcome. Google may have some fancy things with its search engine, but for results, gets lots of ad's and failed results or almost no ad's and the reeks mms results you're looking for. 10 pages plus of Google search results and it's all crap is worthless.
  • Reply 76 of 82
    SpamSandwichSpamSandwich Posts: 33,407member
    Quote:

    Originally Posted by JBDragon View Post





    I can't take all the ad's on Google!!!. It's crazy how many paid ad's get down to you. I use DuckDuckGo and it works quite well. In fact I was just a few weeks ago trying to find a manual for a 15 year old piece of electric requirement for industrial environments. All Google could find was some book type place, I couldn't download a copy. Ur was in this crappy book open look format. Hard to really look at and read let alone do much with. That's all Google gave me. On the other hand I then went to DuckDuckGo and right obj the top of the page, a link to grab a PDF copy of the manual which I downloaded and printed out and it's helped greatly!!!. Google is far from the be all end all of searches!!! I was kind of shocked with these results. I tried adv few times well Google and I had no luck. Pretty disappointing with that outcome. Google may have some fancy things with its search engine, but for results, gets lots of ad's and failed results or almost no ad's and the reeks mms results you're looking for. 10 pages plus of Google search results and it's all crap is worthless.



    Install the browser plug-ins called "Disconnect" and "Ghostery" and prepare for the pleasure of ad-free browsing.

  • Reply 77 of 82
    gatorguygatorguy Posts: 24,176member

    Install the browser plug-ins called "Disconnect" and "Ghostery" and prepare for the pleasure of ad-free browsing.
    I use Ghostery myself. The first few visits at my favorite sites were real eye-openers. There's far more ad companies than just Google at work here, along with several data sellers.

    You should also consider a Facebook blocker. They've become insideous.
  • Reply 78 of 82
    SpamSandwichSpamSandwich Posts: 33,407member
    Quote:

    Originally Posted by Gatorguy View Post





    I use Ghostery myself. The first few visits at my favorite sites were real eye-openers. There's far more ad companies than just Google at work here, along with several data sellers.



    Yeah, Solipsism turned me on to those ad blockers.

  • Reply 79 of 82
    mstonemstone Posts: 11,510member
    Quote:
    Originally Posted by JBDragon View Post





    I can't take all the ad's on Google!!!. It's crazy how many paid ad's get down to you. I use DuckDuckGo and it works quite well. 

     

    It depends on what you search for and where you are, but I don't usually get any ads when searching on Google. I don't mind the 10 Adwords text links on the right or the first sponsored link. In some cases DuckDuck has way more ads. Try searching for the word 'coffee' on each for a nice example. DuckDuck has a whole slew of photo ads for coffee products, Google just points you to the nearest coffee shops and gives you the food specifications of coffee. Try it.

  • Reply 80 of 82
    MacProMacPro Posts: 19,718member

    I'm not sure there's been an official agreement, but I personally think Apple and Microsoft are working in concert to crush Googs.

    Which would make me smile.

    Oh the irony! lol
Sign In or Register to comment.