Search Engine Robots are programs that automatically traverse the web pages by retrieving a document, and recursively retrieving all documents that are referenced. The process of traversing the web page is also called Crawling. Robots are also called Web Crawalers, Spiders, Web Wanderers. These name as slightly misleading as they give the impression the software itself moves between sites like a virus; this not the case,
a robot simply visits sites by requesting documents from them and extracts the information which are important for the search engine.

Robots are eyes of any search engine. So every thing a search engine is indexing and collecting data is becouse of Robots, so optimizing your site for the robots is very important. If search engine robot is not able to crawl your site or some web page of the site, then the chance of that page coming in search result is slim.