Repository navigation

scraper

Public

About

Puppeteer-based URL collector and web scraper with robots.txt support.

automation puppeteer typescript web-scraping

Custom properties

0 stars

1 watching

Report repository

Name		Name	Last commit message	Last commit date
.agents		.agents
.changeset		.changeset
.github		.github
.vscode		.vscode
examples/crawl		examples/crawl
src		src
.gitignore		.gitignore
.prettierignore		.prettierignore
.prettierrc.js		.prettierrc.js
AGENTS.md		AGENTS.md
LICENSE		LICENSE
README.md		README.md
bun.lock		bun.lock
eslint.config.mjs		eslint.config.mjs
eslint.examples.config.mjs		eslint.examples.config.mjs
eslint.local.config.mjs		eslint.local.config.mjs
knip.config.ts		knip.config.ts
package.json		package.json
paradox.config.ts		paradox.config.ts
prettier.local.config.js		prettier.local.config.js
renovate.json5		renovate.json5
tsconfig.build.json		tsconfig.build.json
tsconfig.json		tsconfig.json

Repository files navigation

Scraper

Url collector & scraper based on puppeteer headless browser.

It uses the StealthPlugin from puppeteer-extra
It respects robots.txt

Quickstart

# Build
pnpm run build

# Run (search url's & save html, css & screenshot of every one found)
node dist/index.js

API

collectUrlsAsync({ keywords: string[] })

Collects urls from google search

scrapeUrlAsync(url: string)

Scrapes url and will save: html & css plus a screenshot

About

Puppeteer-based URL collector and web scraper with robots.txt support.

automation puppeteer typescript web-scraping

Custom properties

0 stars

1 watching

Report repository

Releases

Packages

Contributors

Languages