Karim shoair commited on
Commit ·
f3affb8
1
Parent(s): 5297b15
docs: make the contribution rules clearer
Browse files- CONTRIBUTING.md +54 -14
CONTRIBUTING.md
CHANGED
|
@@ -63,6 +63,37 @@ Please follow these coding conventions as we do when writing code for Scrapling:
|
|
| 63 |
> Please don’t put your name in the code you contribute; git provides enough metadata to identify the author of the code.
|
| 64 |
|
| 65 |
## Development
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 66 |
Setting the scrapling logging level to `debug` makes it easier to know what's happening in the background.
|
| 67 |
```python
|
| 68 |
import logging
|
|
@@ -73,15 +104,6 @@ Bonus: You can install the beta of the upcoming update from the dev branch as fo
|
|
| 73 |
pip3 install git+https://github.com/D4Vinci/Scrapling.git@dev
|
| 74 |
```
|
| 75 |
|
| 76 |
-
## Building Documentation
|
| 77 |
-
Documentation is built using [Zensical](https://zensical.org/). You can build it locally using the following commands:
|
| 78 |
-
```bash
|
| 79 |
-
pip install zensical
|
| 80 |
-
pip install -r docs/requirements.txt
|
| 81 |
-
zensical build --clean # Build the static site
|
| 82 |
-
zensical serve # Local preview
|
| 83 |
-
```
|
| 84 |
-
|
| 85 |
## Tests
|
| 86 |
Scrapling includes a comprehensive test suite that can be executed with pytest. However, first, you need to install all libraries and `pytest-plugins` listed in `tests/requirements.txt`. Then, running the tests will result in an output like this:
|
| 87 |
```bash
|
|
@@ -93,16 +115,34 @@ Scrapling includes a comprehensive test suite that can be executed with pytest.
|
|
| 93 |
configfile: pytest.ini
|
| 94 |
plugins: asyncio-1.2.0, anyio-4.11.0, xdist-3.8.0, httpbin-2.1.0, cov-7.0.0
|
| 95 |
asyncio: mode=Mode.STRICT, asyncio_default_fixture_loop_scope=function, asyncio_default_test_loop_scope=function
|
| 96 |
-
10 workers [515 items]
|
| 97 |
-
scheduling tests via LoadScheduling
|
| 98 |
-
|
| 99 |
...<shortened>...
|
| 100 |
-
|
| 101 |
=============================== 271 passed in 52.68s ==============================
|
| 102 |
```
|
| 103 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 104 |
|
| 105 |
Bonus: You can also see the test coverage with the `pytest` plugin below
|
| 106 |
```bash
|
| 107 |
pytest --cov=scrapling tests/
|
| 108 |
```
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 63 |
> Please don’t put your name in the code you contribute; git provides enough metadata to identify the author of the code.
|
| 64 |
|
| 65 |
## Development
|
| 66 |
+
|
| 67 |
+
### Getting started
|
| 68 |
+
|
| 69 |
+
1. Fork the repository and clone your fork:
|
| 70 |
+
```bash
|
| 71 |
+
git clone https://github.com/<your-username>/Scrapling.git
|
| 72 |
+
cd Scrapling
|
| 73 |
+
git checkout dev
|
| 74 |
+
```
|
| 75 |
+
|
| 76 |
+
2. Create a virtual environment and install dependencies:
|
| 77 |
+
```bash
|
| 78 |
+
python -m venv .venv
|
| 79 |
+
source .venv/bin/activate # On Windows: .venv\Scripts\activate
|
| 80 |
+
pip install -e ".[all]"
|
| 81 |
+
pip install -r tests/requirements.txt
|
| 82 |
+
```
|
| 83 |
+
|
| 84 |
+
3. Install browser dependencies:
|
| 85 |
+
```bash
|
| 86 |
+
scrapling install
|
| 87 |
+
```
|
| 88 |
+
|
| 89 |
+
4. Set up pre-commit hooks:
|
| 90 |
+
```bash
|
| 91 |
+
pip install pre-commit
|
| 92 |
+
pre-commit install
|
| 93 |
+
```
|
| 94 |
+
|
| 95 |
+
### Tips
|
| 96 |
+
|
| 97 |
Setting the scrapling logging level to `debug` makes it easier to know what's happening in the background.
|
| 98 |
```python
|
| 99 |
import logging
|
|
|
|
| 104 |
pip3 install git+https://github.com/D4Vinci/Scrapling.git@dev
|
| 105 |
```
|
| 106 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 107 |
## Tests
|
| 108 |
Scrapling includes a comprehensive test suite that can be executed with pytest. However, first, you need to install all libraries and `pytest-plugins` listed in `tests/requirements.txt`. Then, running the tests will result in an output like this:
|
| 109 |
```bash
|
|
|
|
| 115 |
configfile: pytest.ini
|
| 116 |
plugins: asyncio-1.2.0, anyio-4.11.0, xdist-3.8.0, httpbin-2.1.0, cov-7.0.0
|
| 117 |
asyncio: mode=Mode.STRICT, asyncio_default_fixture_loop_scope=function, asyncio_default_test_loop_scope=function
|
| 118 |
+
10 workers [515 items]
|
| 119 |
+
scheduling tests via LoadScheduling
|
| 120 |
+
|
| 121 |
...<shortened>...
|
| 122 |
+
|
| 123 |
=============================== 271 passed in 52.68s ==============================
|
| 124 |
```
|
| 125 |
+
Here, `-n auto` runs tests in parallel across multiple processes to increase speed.
|
| 126 |
+
|
| 127 |
+
**Note:** You may need to run browser tests sequentially (`DynamicFetcher`/`StealthyFetcher`) to avoid conflicts. To run non-browser tests in parallel and browser tests separately:
|
| 128 |
+
```bash
|
| 129 |
+
# Non-browser tests (parallel)
|
| 130 |
+
pytest tests/ -k "not (DynamicFetcher or StealthyFetcher)" -n auto
|
| 131 |
+
|
| 132 |
+
# Browser tests (sequential)
|
| 133 |
+
pytest tests/ -k "DynamicFetcher or StealthyFetcher"
|
| 134 |
+
```
|
| 135 |
|
| 136 |
Bonus: You can also see the test coverage with the `pytest` plugin below
|
| 137 |
```bash
|
| 138 |
pytest --cov=scrapling tests/
|
| 139 |
```
|
| 140 |
+
|
| 141 |
+
## Building Documentation
|
| 142 |
+
Documentation is built using [Zensical](https://zensical.org/). You can build it locally using the following commands:
|
| 143 |
+
```bash
|
| 144 |
+
pip install zensical
|
| 145 |
+
pip install -r docs/requirements.txt
|
| 146 |
+
zensical build --clean # Build the static site
|
| 147 |
+
zensical serve # Local preview
|
| 148 |
+
```
|