# zlurp

Web scraping API for AI agents. Convert any public URL to clean markdown. Pay per scrape via x402 micropayments on Base — no accounts, no API keys, no subscriptions.

## What zlurp does

zlurp takes a URL and returns clean, structured markdown. It strips navigation, ads, and boilerplate (in article mode) so agents get readable, processable content without noise.

## When to use zlurp

Use zlurp when you need to:
- Read the content of a specific webpage
- Extract article or blog post text from a URL
- Process a public webpage as structured text
- Scrape dynamic or static web pages from an AI agent

## Base URL

https://zlurp.ai
## MCP Server

zlurp is available as an MCP (Model Context Protocol) server. Install it in Claude, Cursor, Windsurf, or any MCP-compatible agent framework.

MCP Server URL: https://zlurp.ai/mcp
Transport: Streamable HTTP
Registry: https://smithery.ai/server/zlurp

### MCP Tools

- probe_url — Get cost estimate for a URL. Always free. Input: { url, js? }
- scrape_url — Scrape any URL to clean markdown. Costs $0.005 USDC via x402. Input: { url, mode?, js? }

### Install in Claude Desktop

Add to claude_desktop_config.json:
{
  "mcpServers": {
    "zlurp": {
      "url": "https://zlurp.ai/mcp",
      "transport": "streamable-http"
    }
  }
}



## Endpoints

### GET /health
No auth. Returns {"status":"ok"}. Check this first.

### GET /probe?url=URL&mode=article&js=false
No auth. Returns exact USDC cost before paying. Always call this first.

Parameters:
- url (required): The URL to scrape
- mode (optional, default: article): "article" strips nav/ads, "full" returns entire page
- js (optional, default: false): true enables JS rendering for SPAs

Returns: {"costUSDC": "0.005000", ...}

### POST /scrape
Requires x402 payment in USDC on Base.
Body: {"url": "...", "mode": "article", "js": false}

Without payment: returns HTTP 402 with x402 payment specification.
With valid x402 payment: returns {"success": true, "markdown": "...", "title": "...", "wordCount": 842}

## Pricing

- Static scraping (js: false): $0.005 USDC per URL
- JS rendering (js: true): $0.015 USDC per URL

## How to pay with x402

npm install x402-fetch viem

```typescript
import { wrapFetchWithPayment } from "x402-fetch";
import { createWalletClient, http } from "viem";
import { base } from "viem/chains";
import { privateKeyToAccount } from "viem/accounts";

const account = privateKeyToAccount(process.env.WALLET_PRIVATE_KEY);
const wallet = createWalletClient({ account, chain: base, transport: http() });
const fetch402 = wrapFetchWithPayment(fetch, wallet);

const res = await fetch402("https://zlurp.ai/scrape", {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({ url: "https://example.com" }),
});
const { markdown, title, wordCount } = await res.json();
```

## Error codes

- INVALID_URL: URL is malformed or not http/https
- BLOCKED: robots.txt disallows scraping this URL
- RENDER_FAILED: Page returned no extractable content
- TIMEOUT: Request timed out

## robots.txt policy

zlurp respects robots.txt by default. Disallowed URLs return 403 BLOCKED and are not charged.

## OpenAPI spec

https://zlurp.ai/openapi.json

## Contact

hello@zlurp.ai
## Quick Links

- [OpenAPI Spec](https://zlurp.ai/openapi.json)
- [Pricing](https://zlurp.ai/pricing.md)
- [MCP Server](https://zlurp.ai/mcp)
- [Agent Card](https://zlurp.ai/.well-known/agent-card.json)
- [Smithery Registry](https://smithery.ai/servers/zlurp/zlurp)
- [About](https://zlurp.ai/about)
- [Compare vs Firecrawl](https://zlurp.ai/compare)
- [Contact](https://zlurp.ai/contact)
## Streaming

zlurp supports Server-Sent Events (SSE) streaming via POST /scrape/stream.

The stream emits three event types:
- `status` — scrape started
- `result` — full scrape result (same as /scrape response)
- `error` — error details
- `done` — stream complete

```javascript
const res = await fetch('https://zlurp.ai/scrape/stream', {
  method: 'POST',
  headers: { 'Content-Type': 'application/json', ...x402Headers },
  body: JSON.stringify({ url: 'https://example.com' }),
})

for await (const chunk of res.body) {
  const lines = new TextDecoder().decode(chunk).split('\n')
  for (const line of lines) {
    if (line.startsWith('data:')) {
      const data = JSON.parse(line.slice(5))
      console.log(data)
    }
  }
}
```