# Website content crawler data API: fields, sample output and cost per 1,000 rows

Crawls any public website from a start URL, follows internal links up to a set depth, and extracts title, breadcrumbs, clean text body, and word count.

This is the reference for the Actor `v0iddo/website-content-crawler`: what is in every output row, what the input takes, and what a run costs. Values below come from a real run of the Actor.

## At a glance

| | |
| --- | --- |
| Actor | `v0iddo/website-content-crawler` |
| Store page | [Website content crawler](https://apify.com/v0iddo/website-content-crawler) |
| Output | one row per result, as JSON, CSV, Excel or HTML |
| Fields per row | 8 (4 text, 2 number, 1 URL, 1 list) |
| Cap on results | `maxItems` in the input |
| Price | $2.55 per 1,000 results |

## Output fields

Each result is one row with these fields (fields that name people are left out):

| Field | Type | Meaning | Sample value |
| --- | --- | --- | --- |
| `url` | URL | Page URL | https://flask.palletsprojects.com/en/stable/ |
| `title` | text | Title | Welcome to Flask |
| `h1` | text | H1 Heading | Welcome to Flask |
| `breadcrumbs` | list | Breadcrumbs | ["Flask Documentation (3.1.x)", "Welcome to Flask"] |
| `description` | text | Description |  |
| `textContent` | text | Text content | Welcome to Flask Welcome to Flask’s documentation. Flask is a ligh... |
| `wordCount` | number | Word count | 920 |
| `depth` | number | Crawl depth | 0 |

## One record

```json
{
  "url": "https://flask.palletsprojects.com/en/stable/",
  "title": "Welcome to Flask",
  "h1": "Welcome to Flask",
  "breadcrumbs": [
    "Flask Documentation (3.1.x)",
    "Welcome to Flask"
  ],
  "textContent": "Welcome to Flask  Welcome to Flask’s documentation. Flask is a lightweight WSGI web application framework.\nIt is designed to make getting...",
  "wordCount": 920,
  "depth": 0
}
```

## Input parameters

| Parameter | Type | Required | Default | What it does |
| --- | --- | --- | --- | --- |
| `startUrls` | array | yes | ["https://docs.python.org/3/library/"] | One or more page URLs to begin crawling from. |
| `maxDepth` | integer | no | 2 | How many link-hops from the start URL to follow. |
| `maxPagesPerUrl` | integer | no | 50 | Hard cap on pages fetched per start URL, independent of the global maxItems limit. |
| `includeUrlPattern` | string | no |  | Optional regex. Only URLs matching this pattern are crawled. |
| `excludeUrlPattern` | string | no |  | Optional regex. URLs matching this pattern are skipped. |
| `maxItems` | integer | no | 100 | The run stops after this many result rows. |

## Cost per 1,000 rows

$2.55 per 1,000 results. You pay per result row. There is no separate charge per run. Higher Apify plans pay less per result, down to $1.78 per 1,000.

| Results | Cost at the listed price | Cost at the lowest plan price |
| --- | --- | --- |
| 100 | $0.25 | $0.18 |
| 1,000 | $2.55 | $1.78 |
| 10,000 | $25.50 | $17.85 |
| 100,000 | $255.00 | $178.50 |

The figures are the listed price times the number of results. The `maxItems` limit and the maximum cost setting in the run options both stop a run before it spends more than you planned.

## Get the rows as CSV over HTTP

One request starts a run, waits for it and returns the dataset. The `format` parameter picks the file type and `fields` picks the columns. Replace the token with the one from Apify Console, Settings, Integrations.

```bash
curl -X POST "https://api.apify.com/v2/acts/v0iddo~website-content-crawler/run-sync-get-dataset-items?format=csv&fields=url,title,h1" \
  -H "Authorization: Bearer YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"startUrls": ["https://docs.python.org/3/library/"], "maxDepth": 1, "maxPagesPerUrl": 15, "maxItems": 20}'
```

## Good to know

- The crawler stays within the same origin and path prefix as each start URL. Use includeUrlPattern or excludeUrlPattern to narrow scope on large sites; combine with maxPagesPerUrl to control cost.
- Set maxDepth=0 to extract only the start page. maxDepth=1 adds all pages linked from the start page. Depths above 3 can produce thousands of pages on large sites.
- Pages behind login, paywalls, or bot walls are not accessible.
- Not affiliated with any documentation platform or website.

## Links

- Actor on the Apify Store: [Website content crawler](https://apify.com/v0iddo/website-content-crawler)
- Guide with the Python and JavaScript code: [https://voidd0.github.io/website-content-crawler/](https://voidd0.github.io/website-content-crawler/)