Simplify Edition 6: How Search Engines Work

This article is part of our 'simplify' article series. If you would like to receive our simplify editions straight into your mailbox, please submit your email below. It's that simple.

How do search engines work?

Relates to: search engines, online marketing, search engine optimisation, SEO, Google

A search engine is one of the most important tools for finding information on the web. How do search engines work? What do bots and spiders do? What is SEO? What is Google PageRank?

What is a search engine?

Search engines (such as Google or Microsoft's Bing) are the primary way most Internet users locate content on the world wide web. A search engine is designed to search for information such as web pages, images or other file types on the Internet. Unlike web directories, which are primarily maintained by human editors, search engines operate algorithmically or are a mixture of algorithmic and human input. Google is the largest search engine. Google.com sees approx 620 million visitors daily!

Search engines perform three basic tasks:

They search the Internet based on a search query.

They keep an index of the words they find, and the source where they found the information.

They allow web users to look for relevant information found in that index.

Crawlers, spiders, bots

Search engines use automated software programs known as those little crawly 'spiders' or 'bots' to crawl the web and build their databases - they essentially catalogue the web. A search engine spider reads web pages, grabs the content and follows any links to other pages within the website. This is often referred to as a site being 'spidered' or 'crawled'.

Data collected from a web page is then added to the so called search engine index (think of a huge book; Google's index would be the words on the pages in the book, on which page these can be found and how you get to each of those pages). The index contains a copy of every web page and file that the spider finds and is able to 'read'. The spider returns to the websites in its index on a regular basis, scanning for any changes and updates. How often the spider returns is up to the search engines to decide.

There are now 48,4 billion indexed pages on the Internet. When you enter a query at a search engine site, your input is checked against the search engine's index. The search engine then ranks your search results in an order that it believes is most relevant - this is determined by a set of mathematical equations and specific rules, known as an 'algorithm'. Because each search engine's algorithm is slightly different, the same search terms will produce different lists of results from different search engines.

Google's PageRank (PR) and SEO

Google co-founder Larry Page once described the “perfect search engine” as something that “understands exactly what you mean and gives you back exactly what you want.” Google's PageRank was named after Larry Page. PageRank is [only] one of the methods Google uses to determine a web page's relevance or importance. The algorithm each search engine uses is a closely guarded secret, however some general rules apply that are often used to increase a website's ranking performance. This is referred to as search engine optimisation (SEO).

Google considers a vast number of variables to reflect the importance and hence the ranking of a website. These include for example: on-site optimisation techniques such as keyword placement and keyword density, keyword metatags and description tags, the site's link structure, the age of the domain, freshness of the site, site load times, and a number of other factors. More importantly, off-page website optimisation, specifically those processes that generate high quality backlinks need to form part of any online strategy.

If you would like to learn more about SEO, online marketing, social media marketing and how to improve the visibility and ranking of your website, contact us for more details.

Some interesting facts:

The very first tool used for searching on the Internet was Archie.

Google as a verb was first added to the Merriam Webster and Oxford English dictionary in 2006.

A 'googol' is 1 followed by 100 zeros.

Google's index is roughly 100 million gigabytes.

The amount of data processed daily by Google is 20PB!

Find out about our online strategy and internet marketing services, or simply get in touch.

About the author

Elke Bretz is the founder and creative director at Simplicity. She strongly believes in simplifying complex matters, and is passionate about web design, website usability, ux, user-interface design and social media. You can follow her on Twitter to keep up to date on anything that's happening in the creative space. More info about Elke on