How Search Engines Read Web Pages
To properly optimize web page content for the search engines you need to understand how the search engines read web pages.
When the Search Engine Arrives to the Website
When the search engine arrives to the website in looks in the root (main) folder of the site for a file called robots.txt
In the robots.txt file it looks for what directories and files it is allowed to look at and index. Some search engine spiders (web crawlers, robots) ignore these instructions.
Web Page Head Section
Once a web page is found by the search engine spider (robot, web crawler) it takes a look at the head section (the tags between the <head> and </head> tags) of the web page for:
- The title of the page
- The keyword and description meta tags
- The robots meta tag. Some use the robots meta tag to override the instructions
in the robots.txt file or if they cannot create a robots.txt file the
instructions for the search engine spiders (robots, web crawlers) are
placed here.
If there is no robots.txt file in place and no robots meta taf in the page(s) it finds, it will follow and index all the links found.
To learn more about meta tags, see our Meta Tags and Basic Meta Tags articles.
Web Page Content
Web page content is everything between the <body> and </body> tags.
This where the search engine spider (robot, web crawler) looks for the keywords and phrases you have in the page's keywords and description meta tags.
It will also find and follow links within the web page content.
The search engine spider (robot, web crawler) reads the content in the order that it is inserted into the page. As in from the top down.
Some search engines use the first few words they find in the web page coding as the description under your listing in the search engine results. This is something you should keep in mind if you use HTML tables for your web page layout.
If your website uses frames, javascript or flash you should be aware that search engines don't read these.
A website that uses a doorway page (splash page) won't be too popular with the search engine either. If you are considering one of these or using one, then it would be advisable to reconsider it.
Proper heading structure is also important when preparing your web page for the search engines.
To learn more about search engine optimization, we recommend:
- SEO Techniques - Step by Step An ebook with 20 chapters of step by step instructions and resources needed to optimize web pages for search engine placement.
- SEO Training Class This free search engine optimization class via Yahoo groups is where you can work step by step to learn how to optimize web pages and obtain help as lessons are posted.
If you found this web page a useful resource for your own website please link as follows:
To perform search engine optimization (SEO) properly
you need to understand how the search engines read web pages.
URL:
