This program is a visual editor for Robot Exclusion Files and log analyzer software. It allows the user to quickly and easily create the robots.txt files required to instruct web search engines which parts of a web site are not to be indexed. Read More...
$99.90 - NetPromoter
Robots.txt is a visual editor for Robot Exclusion Files and a log analyzer software. It allows a user to quickly and easily create the Robots.txt files, required to instruct search engine spiders, which parts of a Web site are not to be indexed Read More...
$299.90 - NetPromoter
Page Promoter is an ultimate URL registration, search engine optimization tool, that helps to enhance sites` exposure on the Internet and draw maximum volume of targeted traffic. It allows you to save time and expenses for site promotion campaign Read More...
Free - Chilkat Software, Inc.
Crawl a website with this free ActiveX spidering component. Advanced features include caching, "avoid" patterns, and robots.txt compliance. Read More...
Spiders a site and creates output files containing (1) a list of URLs in the site, (2) a Google sitemap, (3) an XML summary file with title, description, and keywords for each page, and (4) links to URLs external to the spidered domain.
Online website optimization tools: explore and see your site like webengines. Count and display broken and internals links. Get Google pagerank for each file and generate a SiteMaps file in XML format. KWC observes the directives of the robots.txt.
Using simultaneous connections entire websites can be downloaded very quickly. Links can be checked and converted to HTTP or local disk file system. Decide which file types to download and directories to crawl. Handle cookie and session login.
Manage and track visits by robots, spiders and crawlers to your website. Identify which spiders visited, when they visited, and what pages they indexed. Direct spiders to your most important pages, and exclude them from non-public pages. Easy to use
MessLess Inventory is an inventory tracking utility that is 100% customizable. All elements of the program, including data entry fields, forms, templates, reports, queries, toolbars and even hotkeys can be modified to fit your needs. Try free!
