VisBot - The Visvo Web Crawler

VisBot is Visvo's Webpage crawler/spider. The crawler fetches web documents from the Web, categorizes and indexes them for our categorized search functionality.

Visvo's search technology provides a very unique search technology. It presents a categoried view of the Web. This involves analysing, categorizing and creating a quality index. Web crawling is an essential tool and it ensures that we have the most up-to-date search results.

This page will answer most frequently asked questions about the Visvo web crawler.

1. What is a web crawler or robot?
2. Why does Visvo use a web crawler?
3. How does VisBot work?
4. Can I prevent Visvo search engine from showing a cached copy of my page?
5. Does Visvo observe the Robot Exclusion Standard?
6. Can I prevent the VisBot from indexing all or part of my site/URL?
7. Where do I put my robots.txt file?
8. How can I tell if the VisBot has visited my site/URL?
9. Why is the VisBot trying to download incorrect links from my server? Or from a server that doesn't exist?
10. How did the Visvo Website crawler find my URL?
11. What types of links does the VisBot follow?
12. Can I control the rate at which the VisBot visits my site?
13. Why has the VisBot not visited my URL?
14. Why aren't the pages the VisBot indexed showing up in the search results?
15. How can I get additional information?
16. What are the user agent and IP address used by VisBot?

A: A web crawler, also called a robot, is a software program that automatically requests content, usually web pages, from the internet. Search engines such as Visvo use web crawlers to build their searchable database or index. The Visvo crawler is polite and will not cause harm or put undue load on any websites or servers.

A: Visvo uses web crawlers to fetch web pages from the internet. These webpages are then analyzed and automatically categorized to build Visvo's index. By continuously crawling the web, Visvo ensures that it's index is always up to date.

• VisBot( the Visvo web crawler) automatically requests the webpage from its URL. This is similar to a user viewing the webpage with their browser except VisBot does not view images.
• Then, it detects hyperlinks from that page webpages on the same site or on different sites.
• Then, it adds the new webpages to its list of pages to be crawled. This process is repeated continually.
• The VisBot excludes URLs in cases when
  1. There are more than 1000 pages on a single site.
  2. The webpage is a duplicate of another webpage already fetched.
  3. The webpage is classified as a spam or phishing page in the Visvo database.
  4. The webpage is has a crawl-delay value of more than 30 seconds.
• These webpages are then categorized and analyzed to create the index that powers the Visvo search engine. These webpages show up in the search results on the sites powered by Visvo's search technology whenever a relevant match is made.

A: Yes. VisBot obeys the "noarchive" meta tag. If you place the following command in your HTML page, we will not cache the page.

A: Yes, Visbot obeys the 1994 Robots Exclusion Standard (RES), which is part of the Robot Exclusion Protocol. The Robots Exclusion Protocol allows Webmasters to restrict the robots to visit certain pages. For more information on the RES, and the Robot Exclusion Protocol, please visit http://www.robotstxt.org/wc/exclusion.html.

A: Yes. The VisBot will respect and obey commands that prevent it from indexing all or part of a given website specified in the robots.txt.

A: Your file must be at the top level of your Web site, for example, if www.website.com is the name of your website, then the robots.txt file must be at http://www.mysite.com/robots.txt.

A: To determine whether the VisBot has visited your site, check your server logs.

A: Many links on the web are broken or not changed. Whenever any Web page contains a broken or outdated link to your site, or to a site that never existed or no longer exists, VisBot will follow the link and try to fetch it. This causes the crawler to generate activity on a machine that is not a Web server.

A: The VisBot finds pages by following links (HREF tags in HTML) from other pages.

A: The VisBot will follow HREF links, SRC links and re-directs.

A. Yes. VisBot supports the "Crawl-Delay" directive in robots.txt. Using this directive you may specify the minimum delay between two successive requests in seconds. Note that Visvo ignores all websites where the crawl delay is greater than 30 seconds.

A: If the VisBot has not visited your webpage, it has not discovered any links to your webpage from other web pages. As we continue to crawl the web more and more pages will be discovered and included in the Visvo index.

A: It takes time to analyse the web page content, categorize and add it into our index.

A: Please send you queries and feedback at info@visvo.com. You can also post requests via or web interface. If you have specific requests or concerns about Visbot, please send inquiries to bot@visvo.com.

A: The information below is the user agent header and the IP address used by VisBot.

User Agent Header
Visbot/2.0 (+http://www.visvo.com/en/webmasters.jsp;bot@visvo.com)

IP Address
72.249.62.97

Email
bot@visvo.com