Apple Confirms ‘Applebot’ Web Crawler
05/06/2015 08:35
Apple has confirmed to Apple Insider that they are creating a web crawler, called “Applebot”. This web crawler will be used to get information from sites for use within Siri and Spotlight. Right now Apple uses Google and Yahoo to get info from sites. However, this will make it so they get the information quicker and more of what they want.
Apple goes on outlining that if the site does not support Applebot yet, the robot will follow the rules made for Googlebot. This will make it so the Applebot will still be able to access everything even if not all sites support it yet.