60 Day Sandbox For Google & Askjeeves; Msn Indexes Quickest, Yahoo Next
|
Search engine listing delays have come to be called the Google Sandboxeffect are actually true in practice at each of four top tiersearch engines in one form or another. MSN, it seems has theshortest indexing delay at 30 days. This article is thesecond in a series following the spiders through a brand newweb site beginning on May 11, 2005 when the site was firstmade live on that day under a newly purchased domain name.
Previously we looked at the first 35 days and detailed thecrawling behavior of Googlebot, Teoma, MSNbot and Slurp asthey traversed the pages of this new site. We discovered theeach robot spider displays distinctly different behavior incrawling frequency and similarly differing indexing patterns.
For reference, there are about 15 to 20 new pages added tothe site daily, which are each linked from the home page fora day. Site structure is non-traditional with no categoriesand a linking structure tied to author pages listing theirarticles as well as a "related articles" index varied bylinking to relevant pages containing similar content.
So let's review where we are with each spider crawling andlook at pages crawled and compare pages indexed by engine.
The AskJeeves spider, Teoma has crawled most of the pages onthe site, yet indexes no pages 60 days later at this writing.This is clearly a site aging delay that's modeled on Google'sSandbox behavior. Although the Teoma spider from Ask.com hascrawled more pages on this site than any other engine over a60 day period and appears to be tired of crawling as they'venot returned since July 13 - their first break in 60 days.
In the first two days, Googlebot gobbled up 250 pages and didn't return until 60 days later, but has not indexed evena single page in 60 days since they made that initial crawl.But Googlebot is showing a renewed interest in crawling the site since this crawling case study article was published on several high traffic sites. Now Googlebot is looking at afew pages each day. So far no more than about 20 pages at a decidedly lackluster pace, a true "Crawl" that will keep it occupied for years if continued that slowly.
MSNbot crawled timidly for the first 45 days, looking over30 to 50 pages daily, but not until they found a robots.txtfile, which we'd neglected to post to the site for a week andthen bobbled the ball as we changed site structure, thenfailed to implement robots.txt in new subdomains until day 25 - and THEN MSNbot didn't return until day 30. If littleelse were discovered about initial crawls and indexing, wehave seen that MSNbot relies heavily on that robots.txt fileand proper implementation of that file will speed crawling.
MSNbot is now crawling with enthusiasm at anywhere between200 to 800 pages daily. As a matter of fact, we had to usea "crawl-delay" command in the robots.txt file after MSNbotbegan hitting 6 pages per second last week. The MSN index nowshows 4905 pages 60 days into this experiment. Cached pages change weekly. MSNbot has apparently found that it likes howwe changed the page structure to include a new feature whichlinks to questions from several other article pages.
Slurp gets strangely inactive then alternately hyperactive for periods of time. The Yahoo crawler will look at 40 pagesone day and then 4000 the next, then simply look at the homepage for a few days and then jump back in for 3000 pages thenext day and back to only reviewing robots.txt for two days.Consistency is not a curse suffered by Slurp. Yahoo now shows6 pages in their index, one an errors page and another is a"index/of" page as we have not posted a home page to severalsubdomains. But Slurp has crawled easily 15,000 pages to date.
Lessons learned in the first 60 days on a new site follow:
1) Google crawls 250 pages on first discovery of links to site.Then they don't return until they find more links and crawlslowly. Google has failed to index new domain for 60 days.
2) Yahoo looks for errors pages and once they find bad linkswill crawl them ceaselessly until you tell them to stop it.Then won't crawl at all for weeks until crawling heavilyone day and lightly the next in random fashion.
3) MSNbot requires robots.txt files and once they decide theylike your site, may crawl too fast, requiring "crawl-delay"instructions in that robots.txt file. Implement immediately.
4) Bad bots can strain resources and hit too many pages tooquickly until you tell them to stay out. We banned 3 botsoutright after they slammed our servers for a day or two.Noted "aipbot" crawled first then "BecomeBot" came alongand then "Pbot" from Picsearch.com crawled heavily lookingfor image files we don't have. Bad bots, stay out. Best toimplement robots.txt exclusions for all but top engines iftheir crawlers strain your server resources. We consideredexcluding the Chinese search engine named Baidu.com whenthey began crawling heavily early on. We don't expect muchtraffic from China, but why exclude one billion people?Especially since Google is rumored to be considering apossible purchase of Baidu.com as entry to Chinese market.
The bottom line is that we've discovered all engines seem todelay indexing of new domain names for at least thirty days.Google so far has delayed indexing THIS new domain for 60days since first crawling it. AskJeeves has crawled thousandsof pages, while indexing none of them. MSN indexes faster thanall engines but requires robots.txt file. Yahoo's Slurp crawlson again off again for 60 days, but indexes only six of total15,000 or more pages crawled to date.
We seem to have settled that there is a clear indexing delay,but whether this site specifically is "Sandboxed" and whetherdelays apply universally is less clear. Many webmasters claimthat they have been indexed fully within 30 days of first posting a new domain. We'd love to see others track spidersthrough new sites following launch to document their resultspublicly so that indexing and crawling behavior are proven.
© Copyright July 18, 2005 Mike Banks Valentine
Mike Banks Valentine is a search engine optimization specialistwho operates WebSite101 eCommerce Tutorial and will continue reports ofcase study chronicling search indexing of Publish101 Article Resource
Click to Contact Mike Valentine
|
|
|
All You Need To Know About Your Website Traffic Is Contained In 5 Key Stats
You built that content rich web page optimized for your Keyword phrases but is it really working? By looking at these 5 Stats readily available from your CPanel you will quickly learn what you are doing well and what you need to improve on.Most Web Hosts will provide you with a Cpanel, where you can create E-mail Accounts, Upload Web Pages and check your stats. Most of the Cpanel's are very similar if not identical to the one my Hosts uses.When it is time to check my web page traffic statistics I click on Log and then Web/FTP Stats and finally AWStats. I am now presented with a List of all kinds of Stats to look at. Daily History, Monthly History, Country, Browser, Day of Week, Hour o...(related: Seo)
One Way Linking Campaigns Ii
There is a way to generate links with the content that you have not as yet created. For this contact the established authorities (writers, publishers ) ...(related: Seo)
Analyzing Googles Backlinks Is Close To Worthless
...(related: Seo)
Page Rank - A Quick Overview For Beginners
Page Rank (PR) is a specific value for a website page given by Google. It is Google's measure of the importance of a certain site page. The scale is between 1-10. Google gives your website high PR if it is popular. It's based on the number of votes other websites give for your website.Those websites give votes to your site by putting a link to it on their websites. When you link your site to another website, that means you vote for it.From this information, if you want your site to have a high PR, then you have to get as many votes as possible from other websites. In other words, make as many links as possible to your site and you will have higher PR!This is important because PR is one of the many ...(related: Seo)
Easy Seo In 6 Simple Steps
If you want to increase traffic to your website and better your search engine positioning, here is an easy 6 step process that will enable you to do so.1. Get a free blog.Go to blogger.com and sign up for a free account.2. Link your website to your blog.In your blog template, add a link to your website. Also, set up your blog so there is a link field in every post. Set this f...(related: Seo)
Search Engine Optimization :: The Basics And Why Websites Need It
We all know that the most targeted traffic we can get for our websites is from search engines. If you have a little patience and time to set websites the right way you can have a great source of excellent traffic and, best of all, you get it for free.I will outline first in this article the basics that you have to know to get started with the optimization the right way and, if you read it carefully, you will find out how to be one step ahead the majority.If you want to have a complete optimized site you have to take care of both on-site and off-site optimization. I will first outline the basics of on-site optimization.Title Of The PageThe title of the page is one of the most important elements of on-site optimization. Your title must contain your main keyphrase of the page,also make sure that the t...(related: Seo)
Ranked #1 At Google For Invisible Entrepreneurs But No Traffic?
I am ranked #1 for that silly phrase at Google. So What?Here's a secret. You can be ranked #1 at Google for thephrase "Waterfall Watches" if you put the phrase on yourpage 4 times and in metatags twice. How do I know that?I did it in 2001 and still rank number one in Google forth...(related: Seo)
Yahoo Listing Still Worth It?
In October 2002, the Yahoo! portal changed the way it delivers search results. In the past, the most prominent results were exclusively culled from websites listed in the Yahoo directory itself. Since October, sites listed in the Yahoo directory no longer enjoy this privileged status.The Google search engine now drives the primary search results on Yahoo. While this is certainly an improvement for users of Yahoo search, it's a disaster for many businesses that counted on their Yahoo listing to deliver substantial traffic.This change has also led many site owners to question the value of a listing ...(related: Seo)
Search Engine Optimization: Get The Low-down
Been hearing the words Search Engine Optimization lately? I know I have. But what is it, you wonder? Well, that's what I'm here to talk to you about.Search Engine Optimization is a type of pull marketi...(related: Seo)
The Google Phenomenon: Greatest Hits Collection
As my readers know, I've been one of Google's harshest critics. However, I'm also a firm believer that it's okay to criticize, as long as it's justifiable and fairly balanced with deserved praise.Hence, in this article, instead of criticizing, I will be saluting Google and several of its many accomplishments.According to the dictionary, the definition of phenomenon is:"A remarkable development or occurrence. A marvel. "I think we would all agree that definition fits Google like a fine tailored suit.In 1998, singer Janet Jackson was the biggest star on the planet. On September 7, 1998, Google Inc. opened its door in Menlo Park, California. The door came with a remote control, as it wa...(related: Seo)
A Simple 4 Step Approach To Have Google Crawl Your Site Today Guaranteed!
Extra, Extra Google Now Gives you feedback. By Creating a Google Sitemap using free online tools you can get Google to crawl your site within 24 Hours. As an Added bonus after Google Crawls your site they give you feedback about any problems they encountered. To take advantage of this amazing service from yahoo just follow these 4 simple ste...(related: Seo)
Search Engine Strategies For Affiliate Websites
The major search engines are always on the lookout to improve their search results and to weed out websites with duplicate content. They are also weeding out websites that mainly contain affiliate links. This is going to affect quite a few businesses on the Internet that depend on affiliate links to generate revenue.Especially the last Google update kicked some major affiliate players out the index - leaving those websites out in the rain. Loss of traffic was immediate and loss of revenue was dramatic. Future updates can be even more dramatic - therefore threatening the existence of online businesses depending on affiliate marketing. It can also affect the actual company running the affiliate program as their sales would go down if affiliates fail to generate leads.
Search Engine Optimization --- An Overview
Search engine optimization is one of most popular online marketing tool for any website. With most websites looking forward to increasing the online sales of their products/services, search engine optimization can be a very cost-effective tool to target and procure your customers online. In this paper, we will discuss about search engine optimization, its benefits and how you can optimize your website for top search engine rankings.What is search engine optimization?Search engine optimization (also known by various other names like search engine ranking, search engine promotion, website promotion, SEO etc.) is a method through which a website is given top rankings in various search engines. According to a recent industry study, 1 in every 20 people go to the second page of search results pages, but only 1 in every 100 peo...(related: Seo)
site-map - Copyright © 2006 | You can send your articles and get links. Contact Webmaster | All Rights Reserved.
| Seo