http://deadlytechnology.com/seo/dynamic-urls/ Sun, 15 Apr 2012 21:04:00 +0000 hourly 1 http://wordpress.org/?v=3.1.3 http://deadlytechnology.com/seo/dynamic-urls/comment-page-1/#comment-3 deadlytechnology.com » Robots.txt Fri, 11 Aug 2006 00:47:33 +0000 http://deadlytechnology.com/?p=4#comment-3 [...] Note: If you are using mod-rewrite and a robots.txt file the robots who use the robots file will check there first, so if you have an instance where the whole site is excluded in the robots file, and then in mod_rewrite you are doing a 301 redirect to another site, the robots will obey the no index directive in the robots file and in theory never see the redirect. Robots files can get a little tricky especially if you are using mod_rewrite to generate essentially ‘virtual’ directories and you want to stop the indexing of certain directories when really that directory does not exist. I’m still trying to get to the bottom of this little intricity, basically you can put an entry in for the ‘virtual’ directory but it isn’t going to stop the bots from indexing the final desination URL (mod_rewritten one), in which case if you do not have any links to the final destination URL and it is only the result of another URL being mod_rewritten then I think you should be fine just excluding the first URL. Useful links: [...] [...] Note: If you are using mod-rewrite and a robots.txt file the robots who use the robots file will check there first, so if you have an instance where the whole site is excluded in the robots file, and then in mod_rewrite you are doing a 301 redirect to another site, the robots will obey the no index directive in the robots file and in theory never see the redirect. Robots files can get a little tricky especially if you are using mod_rewrite to generate essentially ‘virtual’ directories and you want to stop the indexing of certain directories when really that directory does not exist. I’m still trying to get to the bottom of this little intricity, basically you can put an entry in for the ‘virtual’ directory but it isn’t going to stop the bots from indexing the final desination URL (mod_rewritten one), in which case if you do not have any links to the final destination URL and it is only the result of another URL being mod_rewritten then I think you should be fine just excluding the first URL. Useful links: [...]

]]>