A Spider's quest to be human
Search engine spiders have two purposes: To find new
web sites, and to score them for keyword relevancy.
Finding the new web sites is easy. They find it by following
links, and they index it. No problem. The difficult
task is scoring. Every search engine's desire if to
produce the most relevant results, so that when you
enter keywords, the most accurate results for your search
show at the top. The question s: How do they accomplish
Over the years, many search engines have developed extremely
complex algorithms for their spiders to score with.
On one hand they have tried to mimic what a human would
react to, and the other hand they have played a cat
and mouse game with search engine optimizers attempting
to fool the spider into a higher rank. If nobody in
the world ever thought to optimize their site for search
engines, the spiders would have a much easier job. You
see, the moment you optimize for a keyword, your site
is no longer naturally ranked. So the spiders are designed
to attempt to detect this.
All of this being true, spiders must score a web page
carefully. For a particular keyword or key phrase, it
has to detect if it is relevant, AND if it looks TOO
Over the years optimization practices have gone from
very simple concepts to a highly complex game of balance.
When search engines discover a tactic, they write counter-tactics
into their spiders, and so on.
Remember this: Most spiders are designed to try to look
at your site from a human perspective for information.
Think about what makes a site good to you (aside from
artistic value). You will tend to prefer sites with
lots of information, lots of pages, and clearly written
content. If you can do that with your site, you are
ahead of the game.
Spiders have no appreciation of art.
Spiders are not human. You may have spent $10,000 on
that amazing homepage graphic that everyone raves about,
but all the spider sees is:
So no matter how attached you may be to that image,
remember what the spider can see.
The same goes for Flash. Again, you may have won Flash
Site of the year, but the spider merely sees the call
to the flash, and ZERO content.
Weight, Balance and Substance
When a most spiders look at your page, they will look
at the words on it, and judge how relevant they are
by how many occurrences they find (including variations
of the words), how close to the beginning of the page
they are, and in which sections they find these words.
Above all, they will look for substance. It there lots
of info to digest? The balance of all of these will
heavily impact your score.
Believe it or not, with the little information you have
right now, you are more expert than most web masters
out there. With this information and some common sense,
you will be well on your way.