web_reader / src /api /crawler.ts

Commit History

fix
da1098f
unverified

nomagick commited on

saas: new rate limit policy
a3412ef
unverified

nomagick commited on

saas: save cache in batch
131375b
unverified

nomagick commited on

saas: minor tier policy tweak
5a7f2a4
unverified

nomagick commited on

saas: new tier policy
5a81177
unverified

nomagick commited on

fix: timeout respect
d2afa9d
unverified

nomagick commited on

fix
c72e0d3
unverified

nomagick commited on

fix: ua switch
6b1bfda
unverified

nomagick commited on

saas: refactor proxy provider (#1186)
fe26921
unverified

nomagick commited on

saas: remove sourcecode tip
6f657ae
unverified

nomagick commited on

saas: tweaks
b7d32b9
unverified

nomagick commited on

feat: serp endpoint (#1180)
12ba1bc
unverified

nomagick commited on

fix: try to partition apiroll query
4498af7
unverified

nomagick commited on

fix: expose status code/text from curl
9fbd751
unverified

nomagick commited on

fix: charge amount
65ed115
unverified

nomagick commited on

searcher: image/news/bing search (#1176)
e9d69e6
unverified

nomagick commited on

saas: new rate limit policy
51779dc
unverified

nomagick commited on

fix: curl failure should return 4xx if specified explicitly
a30a865
unverified

nomagick commited on

fix: readerlm params
f7f6a98
unverified

nomagick commited on

fix: beware readerlm capacity drain
f3654a7
unverified

nomagick commited on

fix: do img filtering in node instead of browser
a471a61
unverified

nomagick commited on

fix: bring back content based respond timing
2a30fce
unverified

nomagick commited on

fix: cache use and edge cases
3a40db2
unverified

nomagick commited on

improve: cache usage and detection of js-required pages
ff595c2
unverified

nomagick commited on

fix: track if snapshot html modified by js
9415c6a
unverified

nomagick commited on

behavior change: ditch content based return timing, adopt mutationIdle as default timing
f7dbadf
unverified

nomagick commited on

feat: pdf upload and ip check
45d1682
unverified

nomagick commited on

fix: url check
dceb361
unverified

nomagick commited on

fix
5bbd75a
unverified

nomagick commited on

fix: caching condition
f0560c6
unverified

nomagick commited on

fix: block suspicious requests before sideload
d71c89a
unverified

nomagick commited on

chore: prefer ctx.URL
a9855dc
unverified

nomagick commited on

chore: add comments to clarify
cf01a2c
unverified

nomagick commited on

fix: missing url query param
531c660
unverified

nomagick commited on

fix: bad snapshot in sideload should not throw directly
19a0bbe
unverified

nomagick commited on

feat: leveraging NODE_COMPILE_CACHE (#1162)
6e78e38
unverified

nomagick commited on

fix: several crash cases
d0e20cc
unverified

nomagick commited on

feat: md options pass though to turndown
6b9e14d
unverified

nomagick commited on

fix: catch jsdom errors
3020d58
unverified

nomagick commited on

fix: guard invalid domain names
4ca627c
unverified

nomagick commited on

fix: potential fix for firestore grpc connection reset
4830ff5
unverified

nomagick commited on

fix
fd328cb
unverified

nomagick commited on

fix: somehow side-loading chromewebstore would 100% crash the browser
8456fce
unverified

nomagick commited on

fix: expect failure while loading pdf
440ff4d
unverified

nomagick commited on

fix: potential invalid pdf issue
4bc6394
unverified

nomagick commited on

cleanup: use local project code as much as possible
66db317
unverified

nomagick commited on

fix: implement DNT in alt-gen and pdf-extract
3b1978f
unverified

nomagick commited on

restructure: nolonger a firebase application (#1160)
23a3b80
unverified

nomagick commited on