A web crawler is a bot used by search engines to systematically browse and index web pages. It collects information about page content, structure, and links, enabling search engines to update their databases and deliver relevant results to users. Crawlers follow hyperlinks from page to page, helping maintain the search engine's ranking algorithms by determining the relevance and quality of content across the Internet.