polite-crawler

A polite crawler for agents: obeys robots.txt, fetches each page once, finds and verifies quotes.

What does this entry provide?

polite-crawler is an MCP server entry in ChatHome Skills, sourced from Official MCP Registry and scored 78/100. As a Model Context Protocol server, it exposes tools that MCP-compatible LLM clients such as Claude, Cursor, and Copilot can call directly, extending an agent's reach beyond its built-in capabilities. ChatHome Skills discovered this entry through its daily ingest of official registries and GitHub discovery queries, and re-verifies repository facts, freshness signals, and install metadata on every run. The quality score weighs official-registry presence, a working install command, license clarity, GitHub star trajectory, and repository freshness — the full formula is on the Sources page. The install command above is ready to copy. Review the last-pushed date and repository status before adding polite-crawler to a production agent workflow; the snapshot facts below are point-in-time values from the most recent ingest.

Snapshot facts

Source: Official MCP Registry
Quality: 78/100
Install method: npm
Repo: mark1russell7/crawler
Repo updated: Oct 25, 2026

Install command

npx -y polite-crawler-mcp@0.2.4

Install method: npm

Review the upstream source before running this command.

GitHub project Homepage

Quality

Repo stars

Repo updated

Oct 25, 2026

How is this score calculated?

Weighted across six dimensions, scored 0-100.
Provenance (25): listed in an official or curated registry.
Installability (25): a runnable install command beats a declared transport, which beats a source checkout.
Maintenance (20): recency of the last push; archived entries score zero here.
Adoption (20): repository stars on a log curve.

MCP manifest

{
  "_meta": {
    "io.modelcontextprotocol.registry/official": {
      "status": "active",
      "isLatest": true,
      "updatedAt": "2026-10-25T07:26:15.991495Z",
      "publishedAt": "2026-10-25T07:26:15.991495Z",
      "statusChangedAt": "2026-10-25T07:26:15.991495Z"
    }
  },
  "server": {
    "name": "io.github.mark1russell7/polite-crawler",
    "title": "polite-crawler",
    "$schema": "https://static.modelcontextprotocol.io/schemas/2025-12-11/server.schema.json",
    "version": "0.2.4",
    "packages": [
      {
        "version": "0.2.4",
        "transport": {
          "type": "stdio"
        },
        "identifier": "polite-crawler-mcp",
        "runtimeHint": "npx",
        "registryType": "npm",
        "registryBaseUrl": "https://registry.npmjs.org",
        "environmentVariables": [
          {
            "name": "POLITE_CRAWLER_DIR",
            "format": "filepath",
            "description": "The folder of the store. The default is ~/.polite-crawler."
          },
          {
            "name": "POLITE_CRAWLER_CONTACT",
            "description": "An email address or URL for the From header of each request."
          },
          {
            "name": "POLITE_CRAWLER_ALLOW_PRIVATE",
            "format": "boolean",
            "description": "Set it to 1 to let the crawler fetch loopback, private and link-local addresses. The default refuses them."
          },
          {
            "name": "POLITE_CRAWLER_ALLOW_USER_PROVIDED",
            "format": "boolean",
            "description": "Set it to 1 to let user_provided read a page that refuses AI input when the client cannot ask the user to confirm."
          },
          {
            "name": "POLITE_CRAWLER_SIGN_KEY",
            "format": "filepath",
            "description": "The file of an Ed25519 private key (PKCS#8 PEM or JWK). The crawler signs each request with it (Web Bot Auth). It needs POLITE_CRAWLER_SIGNATURE_AGENT."
          },
          {
            "name": "POLITE_CRAWLER_SIGNATURE_AGENT",
            "description": "Where a site finds the public key of POLITE_CRAWLER_SIGN_KEY: the https origin of the key directory, or the URL of a JSON Web Key Set."
          }
        ]
      }
    ],
    "repository": {
      "url": "https://github.com/mark1russell7/crawler",
      "source": "github",
      "subfolder": "packages/polite-crawler-mcp"
    },
    "websiteUrl": "https://mark1russell7.github.io/crawler/",
    "description": "A polite crawler for agents: obeys robots.txt, fetches each page once, finds and verifies quotes."
  }
}

Report an issue with this listing