FreePanda

FreePandaBot

The crawler behind the FreePanda audit tools — what it does, and how to control it.

Last updated

What it is

FreePandaBot is the web crawler used by the SEO Audit Tool and the AI Visibility Audit. Both read the same crawl, so running one after the other does not visit your site twice. It visits a website only when someone asks it to by entering that site's address into the tool. It does not roam the web, and it does not build an index or a search engine. If you saw it in your logs, somebody — perhaps you, perhaps a colleague or a customer — ran an audit of your site.

How to recognise it

Requests carry this user-agent:

FreePandaBot/1.0 (+https://freepanda.in/bot)

What it does

  • Fetches robots.txt first, and obeys it.
  • Reads your home page, follows the links it finds and reads the pages your sitemap lists, up to a limit set by whoever asked for the audit: 10 pages for a visitor without an account, and never more than 500.
  • Checks the status of the links it finds, with lightweight requests that do not download the page.
  • Makes a handful of set-up checks: whether http:// redirects to https://, whether the www and non-www hosts agree, what a page that does not exist returns, and whether /llms.txt exists.

How gently it does it

  • A few requests at a time, with a pause between them.
  • A Crawl-delay in your robots.txt is honoured, up to a few seconds.
  • It stops on its own after a time limit, and audits of the same site are answered from a recent report instead of crawling again.

What it never does

  • Log in, fill in or submit a form, or send cookies.
  • Run JavaScript. It reads the HTML your server sends, as a simple crawler would.
  • Fetch anything from a private network or a non-standard port.

Blocking or allowing it

To keep it out of part of your site, or all of it, use robots.txt:

User-agent: FreePandaBot
Disallow: /private/

To block it entirely:

User-agent: FreePandaBot
Disallow: /

If your robots.txt blocks all bots with User-agent: * and you want to audit your own site, allow this one by name:

User-agent: FreePandaBot
Allow: /

Problems

If FreePandaBot is misbehaving, or you want an audit of your site removed, write to freepanda.in@gmail.com with the address of your site and we will look into it promptly.