Our Readers Ask: What is Spidering?

Recalling that the Internet is sometimes called the world wide web makes the concept of spidering make somewhat more sense.

Search engines like Google, Feedster and Altavista — basically any web application that needs to grab a snapshot of the entire Internet so that it can be searched, indexed or otherwise sliced and diced — generally develop this snapshot using automated web browsing computer programs that start at one web page, collect all the links on that page and visit each of them in turn, collect all the links on those pages, and so on and so on and so on.

This process, because it is conceptually like “walking throught the web” is called spidering or crawling.


Rusty's picture
Rusty on September 9, 2004 - 13:20 Permalink

I was thinking the other day while visiting a blog and visiting links mentioned on the blog and cutting and pasting bits of articles and links and emailing them to people that the term “mining” was more applicable than “surfing”.