Website content crawler data API: fields, sample output and cost per 1,000 rows

Crawls any public website from a start URL, follows internal links up to a set depth, and extracts title, breadcrumbs, clean text body, and word count.

This is the reference for the Actor v0iddo/website-content-crawler: what is in every output row, what the input takes, and what a run costs. Values below come from a real run of the Actor.

At a glance

Actor	v0iddo/website-content-crawler
Store page	Website content crawler
Output	one row per result, as JSON, CSV, Excel or HTML
Fields per row	8 (4 text, 2 number, 1 URL, 1 list)
Cap on results	maxItems in the input
Price	$2.55 per 1,000 results

Output fields

Each result is one row with these fields (fields that name people are left out):

Field	Type	Meaning	Sample value
url	URL	Page URL	https://flask.palletsprojects.com/en/stable/
title	text	Title	Welcome to Flask
h1	text	H1 Heading	Welcome to Flask
breadcrumbs	list	Breadcrumbs	["Flask Documentation (3.1.x)", "Welcome to Flask"]
description	text	Description
textContent	text	Text content	Welcome to Flask Welcome to Flask’s documentation. Flask is a ligh...
wordCount	number	Word count	920
depth	number	Crawl depth	0

One record

{
  "url": "https://flask.palletsprojects.com/en/stable/",
  "title": "Welcome to Flask",
  "h1": "Welcome to Flask",
  "breadcrumbs": [
    "Flask Documentation (3.1.x)",
    "Welcome to Flask"
  ],
  "textContent": "Welcome to Flask  Welcome to Flask’s documentation. Flask is a lightweight WSGI web application framework.\nIt is designed to make getting...",
  "wordCount": 920,
  "depth": 0
}

Input parameters

Parameter	Type	Required	Default	What it does
startUrls	array	yes	["https://docs.python.org/3/library/"]	One or more page URLs to begin crawling from.
maxDepth	integer	no	2	How many link-hops from the start URL to follow.
maxPagesPerUrl	integer	no	50	Hard cap on pages fetched per start URL, independent of the global maxItems limit.
includeUrlPattern	string	no		Optional regex. Only URLs matching this pattern are crawled.
excludeUrlPattern	string	no		Optional regex. URLs matching this pattern are skipped.
maxItems	integer	no	100	The run stops after this many result rows.

Cost per 1,000 rows

$2.55 per 1,000 results. You pay per result row. There is no separate charge per run. Higher Apify plans pay less per result, down to $1.78 per 1,000.

Results	Cost at the listed price	Cost at the lowest plan price
100	$0.25	$0.18
1,000	$2.55	$1.78
10,000	$25.50	$17.85
100,000	$255.00	$178.50

The figures are the listed price times the number of results. The maxItems limit and the maximum cost setting in the run options both stop a run before it spends more than you planned.

Get the rows as CSV over HTTP

One request starts a run, waits for it and returns the dataset. The format parameter picks the file type and fields picks the columns. Replace the token with the one from Apify Console, Settings, Integrations.

curl -X POST "https://api.apify.com/v2/acts/v0iddo~website-content-crawler/run-sync-get-dataset-items?format=csv&fields=url,title,h1" \
  -H "Authorization: Bearer YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"startUrls": ["https://docs.python.org/3/library/"], "maxDepth": 1, "maxPagesPerUrl": 15, "maxItems": 20}'

Good to know

• The crawler stays within the same origin and path prefix as each start URL. Use includeUrlPattern or excludeUrlPattern to narrow scope on large sites; combine with maxPagesPerUrl to control cost.
• Set maxDepth=0 to extract only the start page. maxDepth=1 adds all pages linked from the start page. Depths above 3 can produce thousands of pages on large sites.
• Pages behind login, paywalls, or bot walls are not accessible.
• Not affiliated with any documentation platform or website.

Links

• Actor on the Apify Store: Website content crawler
• Guide with the Python and JavaScript code: https://voidd0.github.io/website-content-crawler/