Apple is Using 'Applebot' to Crawl the Web for Siri and Spotlight

Apple has revealed that it is using 'Applebot' to crawl the web for Siri and Spotlight Suggestions in a newly modified support document.

Applebot respects customary robots.txt rules and robots meta tags and originates in the 17.0.0.0 net block.

It's user-agent strings will contain “Applebot” together with additional agent information.


For example:
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Applebot/0.1)

If robots instructions don't mention Applebot but do mention Googlebot, the Apple robot will follow Googlebot instructions.

If you have questions or concerns about the Applebot you can find contact information at the link below.

Read More [via AppleInsider]