To avoid undesirable content during the lookup indexes, site owners can instruct spiders never to crawl specific files or directories in the typical robots.txt file in the basis Listing of the area. Furthermore, a webpage is often explicitly excluded from a online search engine's database by utilizing a meta tag distinct to robots (commonly ). When