Definition of Crawlers

Crawlers, robots or spiders work for search engines and crawl the web gathering information on new web pages, updating old pages and deleting dead pages. Spiders are like your Nan at a jumble sale: they like to have a good old poke around. 

Search the Glossary

Glossary Index

• A
• B
• C
• D
• E
• F
• G
• H
• I
• J
• K
• L
• M
• N
• O
• P
• Q
• R
• S
• T
• U
• V
• W
• X
• Y
• Z