Table of Contents
Introduction
Achieving Relevance
Setting the Stage
Making It Work
Achieving Value
Collective Intelligence
Linking Up
Conclusion
Relevant Links
Introduction
Most Internet users count their first Web search as a seminal experience. Older users will likely remember an experience tinged with frustration, as most searches returned tens of thousands of candidate websites, many of which—regardless of the topic searched—seemed to have something to do with sex. Following the 1998 launch of Google, first-time searchers discovered to their amazement that virtually any information, no matter how arcane, lay immediately at their fingertips.
Progress from the earliest search engines to today's highly satisfactory user experience highlighted two essential characteristics of the ideal search result:
Return to Table of Contents.
Achieving Relevance
Searching the Internet is, in many ways, analogous to a trip to the local mega-home-improvement store. If the customer is after a medium-sized flat-head screwdriver, she will first find the tools aisle, then the screwdrivers section, and finally will locate her tool after a quick visual search of the flat-head screwdriver display. The customer's search is quick and successful because the store's manager has organized his merchandise in a way that makes intuitive sense to the customer.
Poor organization can lead to problems. For example, a creative manager might find his merchandise displays more esthetically pleasing if organized by color, thus placing the red-handled, medium-sized, flat-head screwdriver outside with the bricks. Alternatively the store manager may have noticed that customers who purchase screwdrivers often also purchase screws. Hungry for the cross-sell, this manager may place so many boxes of screws in the tool aisle that the screwdrivers are obscured completely. Either way, his customer is likely to depart without her screwdriver.
Return to Table of Contents.
Setting the Stage
In the early days of Internet search, every Web developer was thrust into the position of the store manager described above. Web search agents—whether human or machine—lacked the resources to classify automatically every Web page they encountered. As a result, Web page authors were expected to classify their own pages by inserting invisible markers, called meta keywords, into their pages. Search agents then indexed these keywords and used them as the basis for search returns.
The key to this technique's failure lay in simple economics. Any topical area has a limited number of keywords that describe it. Eager to capture a larger audience than competing sites, webmasters quickly learned to use the most comprehensive possible lists of keywords to describe their pages. Inevitably, sites devoted to similar topics soon had nearly identical keyword lists, thus rendering them indistinguishable from the perspective of the search engine. In the language of our hardware store example, a search for medium-sized flat-head screwdrivers soon turned up everything under the Sun that was red, or flat, or pertinent to screwing.
Ultimately, many companies paid hefty fees to Internet search engines for guaranteed placement at or near the top of their respective categories, a controversial practice that in many cases continues into the present. Meanwhile, webmasters who had a hard time breaking out within their own categories soon attempted to co-opt other categories via the cross-sell. As a result, the keyword list for a screwdriver manufacturer also made mention of screws, nails, impact wrenches, and the greatest equalizer of all: sex.
Some search engines—notably Yahoo!—cut through the noise by creating highly specific directory trees that were entirely human-edited. While this was useful, particularly in eliminating the cross-sell, it introduced a different problem: users who were unclear on their specific search target often did not know which keywords to use, and were thus unable to navigate the increasingly rigid directory structure. If the original problem lay with too many irrelevant results, the new problem was too few (still irrelevant) results.
Moore's Law ultimately came to the rescue, providing search agents with enough computing power to supplement author-defined meta keywords with an automated analysis of the full text of a Web page. Some search engines—notably Google—consider meta keywords to be more hindrance than help, and thus ignore them completely. Instead, such search engines concentrate on the frequency and placement of search terms on the visible Web page, often assigning them extra value if they are highlighted by inclusion in a header or the title of the Web page.
Return to Table of Contents.
Making It Work
A modern search engine, such as Google, gauges the relevance of a Web page to a specific set of search terms based on the following criteria:
- The frequency with which the terms appear on the page.
- The proximity of the terms to one another on the page.
- The prominence of the terms on the page. More prominent text is located near the top of the page, is located in the title or meta-description of the page (similar to meta keywords, but written to be read by humans), or is set off from surrounding text by bold, italic, heading, or similar text enhancement tags.
Text that is topical and clearly written will tend to produce a higher density of similar terms than that which is not. A clean layout, incorporating a sparse and easily legible graphic design, will yield emphasized passages, section headings, and the like that are relevant and distinct. The overall result is surprisingly effective: all other things being equal, modern search engines will score as more relevant pages that are well-written, topically focused, and cleanly designed.
Keep the following points in mind when designing a Web page for high relevance:
- Let the content dictate the keywords, rather than the reverse. Once the rest of the page is complete, include a very short and specific meta keyword list.
- If the page has prose content, write it well. Have a thesis, state it explicitly and early, and stick to it.
- Break page content up into logical sections. Label each section with a heading tag whose contents reflect keywords used inside that section.
- Add a brief, keyword-heavy description of the page in the meta description tag.
- If the page contains images, add a very brief, keyword-heavy description of the image in the image's alt tag.
- Select a unique title for the Web page that reflects the page's theme, using keywords drawn from the page's content.
Return to Table of Contents.
Achieving Value
The World Wide Web has often been called the perfect democracy. With a few days' practice and a trivial investment, virtually anybody can publish content to the Web. By following the simple principles outlined above, a writer who is functionally ignorant of a particular topic can publish a Web page that search engines will deem just as relevant as one created by that topic's foremost authority.
Clearly, relevance alone is insufficient. Most topical searches yield tens of thousands of results, the top several hundred of which score equally high for relevance. As before, the user is left to winnow through the chaff by hand. While this is some improvement over the early state of affairs in Web search, as a final solution it leaves much to be desired.
Return to Table of Contents.
Collective Intelligence
In 1996, Larry Page and Sergey Brin conceived a critical insight. When the designer of a Web page adds a hyperlink to another page, that link constitutes an endorsement of the target page. Furthermore, a Web page that contains links to valuable content becomes useful in itself. In brief, the Internet's own hyperlink structure represents the collective judgment of millions of Web authors regarding which pages are of value, and which are not. In late 1998, Page and Brin launched Google, and the world changed overnight.
Google handles far more Internet searches than any other search engine. In fact, the system works so well that many of the next largest search engines—including AOL, Yahoo!, and Netscape—are "powered" by Google, meaning that some portion of the results they return come from Google servers. Of the remainder, the vast majority have implemented some form of hyperlink-based scoring system similar to that used by Google.
How do these search engines find websites in the first place? Many rely on web spiders, which are autonomous programs that incessantly crawl the Web, searching for new or updated websites. Most also rely on at least one of a very few human-edited Web indexes, the most prominent of which are DMOZ and LookSmart. These indexes provide a stable base of closely monitored, well-categorized websites from which the web spiders can conduct their work.
Return to Table of Contents.
Linking Up
The implications are clear: in order to approach the top of an Internet search engine's list of results, a relevant Web page must contain the following:
- Links to relevant and valuable information.
- Content sufficiently relevant and valuable to induce other Web authors to link to the page in question.
The first of these objectives is simple enough to achieve. Useful content is rarely produced in a perfect vacuum; it is more than appropriate for an author to provide links to his sources.
Achieving the second objective is more difficult, and is absolutely essential. Produce interesting and useful content! The inherently democratic nature of the Web will defeat any marketing budget, no matter how large, that does not provide for the generation of content that people actually want to see.
Keep the following points in mind to enhance the value—and thus the search engine ranking—of a website:
- Produce useful content, and often. Focus on providing value, rather than on advertising a product or service. For example, a travel agency might produce a weekly column highlighting interesting facts about destinations served by the week's travel special.
- Shop the content around. Mail the link to registered users of the website. Send a link and a short description to webmasters of other sites devoted to related topics. Post the same to relevant message boards.
- Enter a description and link to each new piece of content under the appropriate category at DMOZ and LookSmart (or LookSmart's free counterpart, Zeal). Note that the DMOZ website is constantly overloaded. It may take several attempts to submit a page successfully, but it is a worthwhile investment of time. If you find yourself submitting many pages to the same category on DMOZ or Zeal, consider volunteering to edit the category.
Return to Table of Contents.
Conclusion
Modern search engine technology has successfully returned control of the Internet to its users. Advertisers find it increasingly difficult, if not impossible, to "game the system" and bombard search engine users with useless and irrelevant information.
As a result, the best way to gain the attention of potential customers is the most old-fashioned way of all: by exercising the initiative to provide them with sincere, relevant, and highly valuable service.
Return to Table of Contents.
Relevant Links
SearchEngineWatch.com. Provides tips and information about searching the web, analysis of the search engine industry, and help to site owners trying to improve their ability to be found in search engines. Paid membership, but lots of valuable information available for non-members.
Search Engine Optimization Tips. A comprehensive list of tips from Microsoft's bCentral network.
Indexing Resources on the WWW. A comprehensive list of websites, academic papers, and other sources related to Internet indexing and search.