You are here: Foswiki>Dmi Web>ToolDatabase (18 Sep 2024, StijnPeeters)Edit Attach

Digital Methods Initiative Tool Archive pertaining to The Robots.txt

This is an archive of tools developed by the DMI & affiliates in the past years. Note that not all tools are still available and while some are actively maintained, others are not.

Media Analysis: Media Monitoring | Mapping | Clouding | Comparative Media Analysis
Data Treatment: Data Collection | Data Analysis | Information Visualization
Natively Digital: The Link | The URL | The Tag | The Domain | The PageRank | The Robots.txt
Device Centric: Google | Google Images | Google News | Google Blog Search | Yahoo | Wikipedia | Alexa | IssueCrawler | Twitter | Facebook | Amazon | iTunes | Wayback | YouTube | Instagram | Github
Spherical: Web Sphere | News Sphere | Blogosphere | Tag Sphere | Video Sphere | Image Sphere | Code Sphere

Robots.txt Discovery
Launch toolInstructions & Scenarios of Use
Display a site's robot exclusion policy.
 
 
Edit  | Attach | Print version | History: r16 < r15 < r14 < r13 | Backlinks | View wiki text | Edit wiki text | More topic actions
Topic revision: r16 - 18 Sep 2024, StijnPeeters
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Foswiki? Send feedback