How Search engine crawl and index web

Most search engines are on the look out for new content. Search engines crawl and index the web with spiders. The job is to then categorize the content and find out if the links are valid.