Being the first position in a list of search results for a specific set of search terms is the best way to drive more visitors to a web site. More visitors results in more people knowing about your business, and ultimately, more sales.

The term used to cover much of the process of increasing a website’s position in the search results is ‘Search Engine Optimisation’ or SEO. A search engine is just another name for an internet search website – be it Google, Yahoo or another. The optimization part refers to changing your website internal construction, content and external link referencing in an attempt to convince the search engine software that your particular site is more relevant than all the others, and move your site up the list.

About Search Engines

Search Engines have been around since the start of the internet – there has always been the need for people to find information on what they are searching for.The technology has changed and matured as time has gone by, and arguably has become more sophisticated, as well as being one of the most hotly contested competitions in the internet arena.

Most search engines work with a ‘spider’ or ‘bot’. These strange bits of jargon relate to a program which machine-reads the home page of a website, saves all the content (words and pictures) from it, and then ‘follows’ all the links on the page to find all the other pages on that website, and repeats the process for each of those pages.Thus the analogy of a spider – crawling all over the website with many legs.The ‘bot’ name is a shortening of ‘robot’, because the whole process is done by a machine with no human intervention.

Thus a copy of most websites is actually saved on the computers of the search engine owners – a fact that boggles my mind when it comes to calculating how much disk space they have to effectively save a copy of the entire internet onto their computers.It is reported that Google runs over 250,000 computers and spends over $US1 million on electricity per month.Still, they must do it as you can always look at the ‘cached’ version of a webpage in Google – which comes not from the site but from Google’s own server.When you search on Google, you are not searching ‘live’ content but a ‘cached’ version from the last time they indexed that particular website.

Search Engines are not all the same

An important point to remember is that Search Engines are all owned by different companies who are competing with each other to become the biggest and most important search engine on the internet. Thus they are all designed and operated in different ways, and this is why one set of search terms on Google will return a different set of results to the same set of search terms in Yahoo.

Search Engine Popularity

After coming from nowhere in 2001, Google has raced to the top of the search engine popularity rankings.These ratings are compiled by independent researchers, and show the percentage of internet searches that each of the competing engines capture.

Whilst not showing any particular study (most of which show similar but different results, the breakdown falls to about this:

Google : 45%

Yahoo :20%

MSN (includes all MSN-branded sites like ninemsn.com.au and msn.com) : 16%

Others make up the rest, with no other sites achieving more than a couple of percent.

Because Google controls nearly half of the market, much of the effort goes into optimizing a website for the Google search engine, but the Yahoo and MSN should not be ignored, as the figures are a worldwide tally, not country or region-specific.

Jargon Buster

A Short History of Search Engine Optimisation

While the term SEO is relatively new, the practice has been going on for years.In the early days, SEO related mostly to crude efforts of spamming search engines with a repeated set of keywords.It was the unique technology in Google (explained shortly) that overcame these crude methods employed and ranked pages on link popularity more than META and Title tag content, which are easily abused by website owners.

Everyone has clicked on a link in a set of search results, only to be frustrated to find that the page has nothing to do with what you are searching for.You have just become victim to search engine abuse by someone owning a website who probably gets paid for every time an ad is displayed on their site.A visit is a visit, even if it was by mistake and you quickly return.

Google came up with a new way of finding search results which overcame these strategies and gave users more relevant, better quality search results, and crucially separated out paid and un-paid links, so that you knew that the search results were accurate and not the outcome of subterfuge or money changing hands.This, along with other factors such as speed and simplicity, is why Google went from less than 5% to more than 40% of the search engine market in about 3 or 4 years.

Google’s competitors soon woke up to the fact they were losing market share and have since been modifying their software frantically to ensure that they also return relevant results.The winners are the internet users who now expect (and mostly get) highly relevant results for their searches.The losers are the search engines who did not adapt, and the website owners who owned sites with no real content and relied on tricking search engines into returning them as relevant.

This activity, however, has made it more difficult for newer sites wishing to gain in popularity, which is why SEO is becoming a large field.

Google’s Advances in Search Engines

Larry Page and Sergey Brin were two PhD students who devised a better way to find relevant information than basing it on the actual content of a page, which could be easily manipulated.

Simply speaking, they based their method for finding relevant results on the academic model of including references at the end of a paper. If you counted all the times a particular academic paper was reference by other academic papers, textbooks and articles, you could assess how important other authors perceived that paper to be. A very important scientific work such as Darwins' ‘On the Origin of Species’ has probably been referenced innumerable times, while all of the papers I wrote in my academic career would never have managed a single reference between them.So Charles Darwin’s output is far more important than mine.Crucially, with this model, the author of an academic paper has no control over the number of times his/her paper is referenced by others, so it is a true independent measure of importance.

In the same way, websites contain links to each other.If you consider a link to be a ‘vote of confidence’ by the author of the website towards the linked website, then the websites with the most links ‘pointing’ at it must be the most popular, and thus the most relevant on that particular topic.

There are many more factors at play, but this is the basis of the way that Google is reported to work.I say reported because as a commercial company, they don’t disclose how their search engine works, so the general knowledge is built up from inference and suggestions by Google, plus research and testing by specialists.

The site importance is captured in a factor called ‘PageRank’ by Google.This is a measure from 0-10 on how important Google considers the site/page to be.You can see the pagerank for a site by downloading and installing the Google toolbar, which will then display the PageRank for every webpage visited. Google awards itself a PageRank of 10, and any new site on the internet will automatically get a PageRank of 0.

Google in particular states that they focus on the user searching more than the sites displayed.They put a lot of work into discovering ways in which site owners are attempting to trick the search engines into rating their site higher, and they either modify their systems to work around tricks, or worse, occasionally ‘blacklist’ a website from the search engine results altogether if they decide that website is intentionally trying to deceive the Google system. This does happen and legend has it that BMW North America got blacklisted for a time for employing tricks to fool the Google search engine into rating it higher.

Tricks employed include ‘spamming’ a search engine, which borrows the term from unsolicited emails but is not the same thing. It involves repeating keywords many times on the page in a continuous string of text, putting in a redirect to another page altogether that Search engines can’t see, duplicating content on many different domain names and making your site produce a different set of content for a search engine program and for a genuine visitor.

A good article on this is published by Google themselves in their page:

Google Webmaster Guidelines

What this means for Search Engine Optimisation and Website Owners

Because all Search Engine companies are following Google’s lead on the way to develop the most efficient and accurate search engine, the way in which a site can improve it’s ranking is becoming more complex. However, it is still possible and probable that a particular site can improve it’s rankings, even if it is a new site.

I separate SEO into two different operations: On-Site and Off-Site optimization.

On-Site optimization relates to ensuring that the content on your site is relevant and useful to users, whilst making it highly ranked by search engines. This includes making sure the META tags on your site pages are informative and accurate relative to the content on the page.It also means ensuring that each page in your site can be found via a link from another page.This ensures that the search engine spiders can find every piece of content on the site and index it.These factors are why I generally don’t like to include too many gizmos on a website – they are not machine readable and therefore of no use to search engines, therefore will have little use in generating extra visitors to the site.However the good news is that a lot of website developers love pushing ‘Flash’ content and other ‘whizz-bang’ technology, which invariably results in the resultant websites rating very poorly in search engine rankings (not to mention poor performance over slower connections and with content security-restricted computers used in corporate workplaces)

Off-Site optimization relates to ensuring that your website is seen as ‘popular’ by other websites.This has a lesser effect in MSN but a much larger effect in Google and Yahoo.To be seen as popular, all you need is incoming links to your site from other site.The more important the site linking to yours is, the more important your site will be seen.As the Google homepage has it’s own highest ranking, a single link from the Google homepage would probably put your site at the top of the rankings for any conceivable search relevant to it.Unfortunately, this can’t be done, but it is possible (and often very easy) for links to created to your site from other sites.The more of these you have, the more popular and relevant your site will be seen as, and the higher up the search engine rankings you will go.

When I put together sites, I automatically set them up for on-page optimization, by placing relevant search terms in the META Tags and cross-linking all the pages.This in itself is usually enough to get the site moving up the rankings for a particular search term, but it isn’t usually enough to get to the top.To get to the top you need more links coming in than whoever is currently at the top. Whilst not an exact science, it is a general rule that works well enough in practice.

However, what I can’t control is the incoming links to the site from other sites, as these are owned by other people. Generating these incoming links is an ongoing process that the owner must perform as part of the marketing and promotion of the site itself. They can be bought, exchanged (I’ll link to you if you link to me) or just plain requested if the other person likes your site enough. Suppliers, customers and business partners make ideal people to exchange links with, because the content is relevant to the subject matter of your website.

Summary

Hopefully I’ve cleared up how search engines work, why Google has become the largest search engine on the internet, and how that affects the ranking of a particular website in a particular search engine.

In the next articles, I will be discussing generating incoming links into your website, and how to generate visitors with pay-per-click advertising. Until then, I hope this article helps with understanding the mysterious world of search engines.