Description: Robots.txt is a plaintext file which is placed in the root directory ( abc.com/robots.txt ) of website and serves as an instruction set to automated web spiders / robots such as search engine bots (such as Googlebot and Yahoo Slurp!). Ideally all robots are supposed to obey the Robots Exclusion rules written in this file, but interestingly a lot of webmasters have misunderstood the intention of the file and are using it to hide confidential data and directories. In recent times this has made the robots.txt one of the first places for an Attacker to take a look at before he compromises a website's security. In this video we will explore how to interpret the robots.txt file (which is actually very simple) and look at a few examples of well known websites, which seeming have an interesting set of directories listed in their robots.txt file.<br><br><div style="width:425px;text-align:left" id="__ss_467034"><object style="margin:0px" width="425" height="355"><param name="movie" value="http://static.slideshare.net/swf/ssplayer2.swf?doc=analyzingrobotstxtforfunandprofit-1213413829256173-8"/><param name="allowFullScreen" value="true"/><param name="allowScriptAccess" value="always"/><embed src="http://static.slideshare.net/swf/ssplayer2.swf?doc=analyzingrobotstxtforfunandprofit-1213413829256173-8" type="application/x-shockwave-flash" allowscriptaccess="always" allowfullscreen="true" width="425" height="355"></embed></object><div style="font-size:11px;font-family:tahoma,arial;height:26px;padding-top:2px;"><img src="http://static.slideshare.net/swf/logo_embd.png" style="border:0px none;margin-bottom:-5px" alt="SlideShare"/> | View | Upload your own</div></div> Links:<br><br>1. Robots.txt<br><br>2. Robots and Spiders - 2600 Magazine<br><br>
Tags: programming ,
Disclaimer: We are a infosec video aggregator and this video is linked from an external website. The original author may be different from the user re-posting/linking it here. Please do not assume the authors to be same without verifying.