Karim shoair commited on
Commit ·
7c719ac
1
Parent(s): ecc7bcb
docs: add new api references
Browse files- docs/api-reference/proxy-rotation.md +18 -0
- docs/api-reference/response.md +18 -0
- docs/api-reference/spiders.md +42 -0
- zensical.toml +4 -1
docs/api-reference/proxy-rotation.md
ADDED
|
@@ -0,0 +1,18 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
search:
|
| 3 |
+
exclude: true
|
| 4 |
+
---
|
| 5 |
+
|
| 6 |
+
# Proxy Rotation
|
| 7 |
+
|
| 8 |
+
The `ProxyRotator` class provides thread-safe proxy rotation for any fetcher or session.
|
| 9 |
+
|
| 10 |
+
You can import it directly like below:
|
| 11 |
+
|
| 12 |
+
```python
|
| 13 |
+
from scrapling.fetchers import ProxyRotator
|
| 14 |
+
```
|
| 15 |
+
|
| 16 |
+
## ::: scrapling.engines.toolbelt.proxy_rotation.ProxyRotator
|
| 17 |
+
handler: python
|
| 18 |
+
:docstring:
|
docs/api-reference/response.md
ADDED
|
@@ -0,0 +1,18 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
search:
|
| 3 |
+
exclude: true
|
| 4 |
+
---
|
| 5 |
+
|
| 6 |
+
# Response Class
|
| 7 |
+
|
| 8 |
+
The `Response` class wraps HTTP responses returned by all fetchers, providing access to status, headers, body, cookies, and a `Selector` for parsing.
|
| 9 |
+
|
| 10 |
+
You can import the `Response` class like below:
|
| 11 |
+
|
| 12 |
+
```python
|
| 13 |
+
from scrapling.engines.toolbelt.custom import Response
|
| 14 |
+
```
|
| 15 |
+
|
| 16 |
+
## ::: scrapling.engines.toolbelt.custom.Response
|
| 17 |
+
handler: python
|
| 18 |
+
:docstring:
|
docs/api-reference/spiders.md
ADDED
|
@@ -0,0 +1,42 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
search:
|
| 3 |
+
exclude: true
|
| 4 |
+
---
|
| 5 |
+
|
| 6 |
+
# Spider Classes
|
| 7 |
+
|
| 8 |
+
Here's the reference information for the spider framework classes' parameters, attributes, and methods.
|
| 9 |
+
|
| 10 |
+
You can import them directly like below:
|
| 11 |
+
|
| 12 |
+
```python
|
| 13 |
+
from scrapling.spiders import Spider, Request, CrawlResult, SessionManager, Response
|
| 14 |
+
```
|
| 15 |
+
|
| 16 |
+
## ::: scrapling.spiders.Spider
|
| 17 |
+
handler: python
|
| 18 |
+
:docstring:
|
| 19 |
+
|
| 20 |
+
## ::: scrapling.spiders.Request
|
| 21 |
+
handler: python
|
| 22 |
+
:docstring:
|
| 23 |
+
|
| 24 |
+
## Result Classes
|
| 25 |
+
|
| 26 |
+
## ::: scrapling.spiders.result.CrawlResult
|
| 27 |
+
handler: python
|
| 28 |
+
:docstring:
|
| 29 |
+
|
| 30 |
+
## ::: scrapling.spiders.result.CrawlStats
|
| 31 |
+
handler: python
|
| 32 |
+
:docstring:
|
| 33 |
+
|
| 34 |
+
## ::: scrapling.spiders.result.ItemList
|
| 35 |
+
handler: python
|
| 36 |
+
:docstring:
|
| 37 |
+
|
| 38 |
+
## Session Management
|
| 39 |
+
|
| 40 |
+
## ::: scrapling.spiders.session.SessionManager
|
| 41 |
+
handler: python
|
| 42 |
+
:docstring:
|
zensical.toml
CHANGED
|
@@ -54,7 +54,10 @@ nav = [
|
|
| 54 |
{Selector = "api-reference/selector.md"},
|
| 55 |
{Fetchers = "api-reference/fetchers.md"},
|
| 56 |
{"MCP Server" = "api-reference/mcp-server.md"},
|
| 57 |
-
{"Custom Types" = "api-reference/custom-types.md"}
|
|
|
|
|
|
|
|
|
|
| 58 |
]},
|
| 59 |
{"Writing your retrieval system" = "development/adaptive_storage_system.md"},
|
| 60 |
{"Using Scrapling's custom types" = "development/scrapling_custom_types.md"}
|
|
|
|
| 54 |
{Selector = "api-reference/selector.md"},
|
| 55 |
{Fetchers = "api-reference/fetchers.md"},
|
| 56 |
{"MCP Server" = "api-reference/mcp-server.md"},
|
| 57 |
+
{"Custom Types" = "api-reference/custom-types.md"},
|
| 58 |
+
{Response = "api-reference/response.md"},
|
| 59 |
+
{Spiders = "api-reference/spiders.md"},
|
| 60 |
+
{"Proxy Rotation" = "api-reference/proxy-rotation.md"}
|
| 61 |
]},
|
| 62 |
{"Writing your retrieval system" = "development/adaptive_storage_system.md"},
|
| 63 |
{"Using Scrapling's custom types" = "development/scrapling_custom_types.md"}
|