Google’s mission with Search is to serve its users with the most relevant search results, to enable them to find desired info easily and to ensure they return to Google for future info requests. The more Google is used, the more likely it is their advertising platform generates revenue. One of the biggest threats (bar spam/malware) to the integrity of their search results is Search Engine Optimisation (SEO).
The “art” of SEO started pretty early on in the existence of the internet, when the early adopters realised search results could be manipulated. Early search algorithms were predictable and could be controlled through on-page optimisation, most notably a page’s meta tags and a few simple rules for a page’s content. Google has had to figure out how to improve search results in terms or relevancy but also to secure their algorithm against manipulation of search results. At some point during that struggle, Google realised that rather than fighting SEO, setting up best practices would be the way forward. These best practices drew the line for what is permissible and what is deemed malicious (and will result in the ultimate penalty: Being booted from Google’s index for good, or simply “sandboxed” – removed from index but not banned).
Google Webmaster Tools
Google started a dialogue with SEO practitioners, most notably through Matt Cutts, the head of Google’s Webspam team. Matt discusses several SEO related issues, FAQ’s and obviously the bad neighbourhoods and black hat SEOs he comes across daily. He also actively promotes Google’s Webmaster Tools, which is a small window into how search engine crawlers (automated bots that value your site’s pages and determine how they should be ranked in relation to other pages in the index) value your site’s pages.
Submitting a new website to Google Webmaster Tools (GWT) has become one of the first steps a webmaster can take once a site launched. The goal is to notify Google of the existence of your site, which is done by setting up a sitemap, to be submitted to GWT. A sitemap is generally a small XML file placed in the root folder of your site which lists all the site’s pages and which can be generated automatically. If you run WordPress you’re looking for Google XML Sitemaps (and plugins can be found for most of the popular content management systems), and there are simple sitemap scripts out there that can automate sitemap creation.
Once Google picks up on your website’s content and judges it worthy enough to rank in the search results, you can access GWT to find out a couple of things about how your pages are valued, such as:
- The top search queries pages from your site show up and in what average position
- Any warnings and or errors that Google’s crawler encountered relating to your website and its content
- Internal and Inbound links per page
GWT also allows you to set specific instructions for Google’s crawler, which can help you get more out of search engine traffic:
- Configuring and creating a robots.txt file. This file can be used to exclude folders on your site which contain content you don’t want Google to consider for indexing. Such folders could be administrator pages, plugins, cache folders as well as print versions of your pages (which could be judged as duplicate content)
- Target a geographic area – Handy if you run a site that’s targeted at users from a specific country.
- Set crawl rate – Every time the crawler accesses your site it will follow a bunch of pages through internal links but can easily exit your site again through outbound links. If you have a number of new pages up, you can set a custom crawl rate (valid for up to 90 days) during which the crawler will spend more time on your site and potentially indexes more pages during a visit to your site.
More info
For more information on how to use data available in Google Webmaster Tools, check out 3 Useful Analytics Lessons of Google Webmaster Tools.
Also have a look at the Google Webmaster Central channel on Youtube. Matt Cutts regularly uploads new Q&A videos with some great nuggets of SEO knowledge from the horse’s mouth!
[...] reading here: Google, SEO and Webmaster Tools Related Posts:New 24/7 Webmaster web siteA Review Of Google Sitemap WordPress Plugin – Making [...]
Excellent post by @elansink on SEO and how to get the best out of Google: http://bit.ly/199Ngq
This comment was originally posted on Twitter
Google’s Webmaster Guidelines are vague in many expects and are continuously subject to different interpretations. As you said, SEO professionals can be a threat to Google, since they can manipulate serps to a certain degree. As a matter of fact, Matt Cutts rarely helps SEO experts. He’s just giving tips to beginners aiming to level the playing field, whilst trying to make us all think that Google is almost a perfect search engine.
Let’s face it, I personally see loads of spammy websites which rank highly on Google even today! You won’t get banned for sending automated queries to Google or participate in link schemes as it’s written in their guidelines for example. If that is the case, I will just subscribe my competitors to such schemes or run thousands of automated keyword ranking reports targeting their websites to get them banned!
Google is arguably the best search engine we have at the moment, but it’s an unhealthy situation for any online business to be so dependent of this one source. Webmaster Tools can indeed be useful, but is not always very reliable either. I’ve seen warnings and errors that Google’s crawler encounters disappear, although errors are actually still present. Just to mention other examples, crawl rate is usually not adjustable for small sites, links to internal pages are rarely correct, etc, etc.
What I’m saying is that Google is far from being the perfect search engine they want us to believe. Most of their free tools including Webmaster Tools and Analytics are just means of obtaining much more data about us and the websites we optimize. I have recently came across few YouTube clips about this subject and am pasting a link to episode 1 below. Look for episode 2 and 3 too, they’re hilarious!
http://www.youtube.com/watch?v=9RDe2Ia6YlM
What I see in Matt Cutts is a representative who tries to educate on best practices for on-page optmization which makes Google’s job easier. You rarely hear him speaking about link-building or white/grey-hat techniques thereof – and for a reason.
I agree that it’s not necessary to follow the info from GWT to the word, but by simply raising awareness on possible issues and correcting them is always a good way to go, benefiting traffic from all search engines besides just Google (as well as usability from a user’s p.o.v.).
Sure, Google is nowhere near perfect (and I’m not trying to say they are) and everything they do is for the grater good ($$$). I’m not too keen on all the big brother/conspiracy theories on what Google does with data. Sure, some of that data will be used in SERP weighting but I doubt the endless amount of data will be used for “evil”… I think rather than fix problems through data they spend time on education (which is where Matt Cutts comes in again).
The impression I get is that GWT is Google’s way of serving the on-page optimizers. Sometimes great content never gets the SERP appreciation it deserves because of bad/lack of optimization. Google Analytics is a different kettle of fish – I think its main purpose is to generate interest and get AdWords users to spend more. In the end, whatever Google releases is also intended to strengthen their brand and presence of course.
Two new posts on Google Webmaster Tools – SEO & Analytics: http://su.pr/2m7BjJ & http://su.pr/1s9yEu
This comment was originally posted on Twitter