fix

chore: use env variable to switch browser headless mode

cleanup

improve: cache usage and detection of js-required pages

fix: return 10 search results by default

fix: some response could miss meta field (#1165)

fix: track if snapshot html modified by js

fix: tweak default timing to be more conservative

fix: custom script may listen to mutationIdle

behavior change: ditch content based return timing, adopt mutationIdle as default timing

docs: mention about discarded

feat: provide option to discard links

clean: redundant ip block

fix: a small lru for invalid tokens

fix

feat: pdf upload and ip check

fix

fix: request blocker

fix: url check

fix

fix: caching condition

fix: robots-txt not loaded error conditions

docs: minor fix

docs: be neutral on brand

fix: block suspicious requests before sideload

feat: control concurrent request per page instead of server bucket

fix: unhandledRejection log level

chore: prefer ctx.URL

chore: add comments to clarify

fix: missing url query param

fix: finalizer and unhandled promise rejection

fix: provide our onw robots.txt

fix: robots-txt location

fix: robots-txt cache location

fix: curl redirections

chore: fix search result amount (#1163)

fix: curl error category

deps: cleanup

fix: bad snapshot in sideload should not throw directly

fix: runtime NODE_COMPILE_CACHE dir

feat: leveraging NODE_COMPILE_CACHE (#1162)

fix: several crash cases

feat: md options pass though to turndown

deps: bump

fix: catch jsdom errors

deps: bump

fix: guard invalid domain names

fix: potential fix for firestore grpc connection reset

fix

fix: somehow side-loading chromewebstore would 100% crash the browser