web crawler
Definitions
A program that automatically browses the web and collects or indexes information from websites.
A person or tool that systematically explores websites by following links.
Examples
The web crawler indexed thousands of pages overnight.
A web crawler follows links to discover new websites.