Comments on: AdSense and robots.txt (Part 4) http://www.memwg.com/adsense-and-robotstxt-part-4/ Eric Giguere's AdSense Tips Mon, 02 Jan 2012 15:55:31 +0000 hourly 1 http://wordpress.org/?v=3.3 By: Google AdWords Case Study: Site Design : Make Easy Money With Google And AdSense http://www.memwg.com/adsense-and-robotstxt-part-4/comment-page-1/#comment-1314 Google AdWords Case Study: Site Design : Make Easy Money With Google And AdSense Tue, 11 Sep 2007 17:12:01 +0000 http://memwg.com/general/adsense-and-robotstxt-part-4/#comment-1314 [...] they won’t be very different from each other. Don’t wet your pants! If it worries you, exclude your landing pages from the search engines using your robots.txt file: that’s what it’s for. Just be sure, however, to let the [...] [...] they won’t be very different from each other. Don’t wet your pants! If it worries you, exclude your landing pages from the search engines using your robots.txt file: that’s what it’s for. Just be sure, however, to let the [...]

]]> By: Eric Giguere http://www.memwg.com/adsense-and-robotstxt-part-4/comment-page-1/#comment-474 Eric Giguere Fri, 29 Jun 2007 14:49:18 +0000 http://memwg.com/general/adsense-and-robotstxt-part-4/#comment-474 If you had no robots.txt, nothing would have prevented Google from indexing your site UNLESS you had a "noindex" declaration in the home page's meta tag. Is your website configured to redirect requests for missing/non-existent files (i.e. a 404 error) to your home page? That would explain what you saw. If you had no robots.txt, nothing would have prevented Google from indexing your site UNLESS you had a “noindex” declaration in the home page’s meta tag.

Is your website configured to redirect requests for missing/non-existent files (i.e. a 404 error) to your home page? That would explain what you saw.

]]> By: Satish Talim http://www.memwg.com/adsense-and-robotstxt-part-4/comment-page-1/#comment-469 Satish Talim Fri, 29 Jun 2007 12:39:55 +0000 http://memwg.com/general/adsense-and-robotstxt-part-4/#comment-469 So far I had no robots.txt. Reading this post, I used GWT to find more info on robots.txt analysis. I saw that the default text shown of the robots.txt file was the contents of my index.html file and the status said that the file was in error. I have now kept an empty robots.txt. Let's see what it has to say. So far I had no robots.txt. Reading this post, I used GWT to find more info on robots.txt analysis. I saw that the default text shown of the robots.txt file was the contents of my index.html file and the status said that the file was in error. I have now kept an empty robots.txt. Let’s see what it has to say.

]]> By: Eric Giguere http://www.memwg.com/adsense-and-robotstxt-part-4/comment-page-1/#comment-465 Eric Giguere Fri, 29 Jun 2007 10:15:32 +0000 http://memwg.com/general/adsense-and-robotstxt-part-4/#comment-465 No. The robots.txt file is used primarily to keep search engines out of certain areas/pages of your site. If you don't have a need to do so, you don't need a robots.txt file. If there's no file, the search engines assume that they can crawl every page they find, unless the page itself has a meta tag indicating otherwise. No. The robots.txt file is used primarily to keep search engines out of certain areas/pages of your site. If you don’t have a need to do so, you don’t need a robots.txt file. If there’s no file, the search engines assume that they can crawl every page they find, unless the page itself has a meta tag indicating otherwise.

]]> By: Satish Talim http://www.memwg.com/adsense-and-robotstxt-part-4/comment-page-1/#comment-461 Satish Talim Fri, 29 Jun 2007 06:12:54 +0000 http://memwg.com/general/adsense-and-robotstxt-part-4/#comment-461 I don't have robots.txt on my site. Is it mandatory? I don’t have robots.txt on my site. Is it mandatory?

]]>