Karim shoair commited on
Commit
f3affb8
·
1 Parent(s): 5297b15

docs: make the contribution rules clearer

Browse files
Files changed (1) hide show
  1. CONTRIBUTING.md +54 -14
CONTRIBUTING.md CHANGED
@@ -63,6 +63,37 @@ Please follow these coding conventions as we do when writing code for Scrapling:
63
  > Please don’t put your name in the code you contribute; git provides enough metadata to identify the author of the code.
64
 
65
  ## Development
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
66
  Setting the scrapling logging level to `debug` makes it easier to know what's happening in the background.
67
  ```python
68
  import logging
@@ -73,15 +104,6 @@ Bonus: You can install the beta of the upcoming update from the dev branch as fo
73
  pip3 install git+https://github.com/D4Vinci/Scrapling.git@dev
74
  ```
75
 
76
- ## Building Documentation
77
- Documentation is built using [Zensical](https://zensical.org/). You can build it locally using the following commands:
78
- ```bash
79
- pip install zensical
80
- pip install -r docs/requirements.txt
81
- zensical build --clean # Build the static site
82
- zensical serve # Local preview
83
- ```
84
-
85
  ## Tests
86
  Scrapling includes a comprehensive test suite that can be executed with pytest. However, first, you need to install all libraries and `pytest-plugins` listed in `tests/requirements.txt`. Then, running the tests will result in an output like this:
87
  ```bash
@@ -93,16 +115,34 @@ Scrapling includes a comprehensive test suite that can be executed with pytest.
93
  configfile: pytest.ini
94
  plugins: asyncio-1.2.0, anyio-4.11.0, xdist-3.8.0, httpbin-2.1.0, cov-7.0.0
95
  asyncio: mode=Mode.STRICT, asyncio_default_fixture_loop_scope=function, asyncio_default_test_loop_scope=function
96
- 10 workers [515 items]
97
- scheduling tests via LoadScheduling
98
-
99
  ...<shortened>...
100
-
101
  =============================== 271 passed in 52.68s ==============================
102
  ```
103
- Hence, we used `-n auto` in the command above to run tests in threads to increase speed.
 
 
 
 
 
 
 
 
 
104
 
105
  Bonus: You can also see the test coverage with the `pytest` plugin below
106
  ```bash
107
  pytest --cov=scrapling tests/
108
  ```
 
 
 
 
 
 
 
 
 
 
63
  > Please don’t put your name in the code you contribute; git provides enough metadata to identify the author of the code.
64
 
65
  ## Development
66
+
67
+ ### Getting started
68
+
69
+ 1. Fork the repository and clone your fork:
70
+ ```bash
71
+ git clone https://github.com/<your-username>/Scrapling.git
72
+ cd Scrapling
73
+ git checkout dev
74
+ ```
75
+
76
+ 2. Create a virtual environment and install dependencies:
77
+ ```bash
78
+ python -m venv .venv
79
+ source .venv/bin/activate # On Windows: .venv\Scripts\activate
80
+ pip install -e ".[all]"
81
+ pip install -r tests/requirements.txt
82
+ ```
83
+
84
+ 3. Install browser dependencies:
85
+ ```bash
86
+ scrapling install
87
+ ```
88
+
89
+ 4. Set up pre-commit hooks:
90
+ ```bash
91
+ pip install pre-commit
92
+ pre-commit install
93
+ ```
94
+
95
+ ### Tips
96
+
97
  Setting the scrapling logging level to `debug` makes it easier to know what's happening in the background.
98
  ```python
99
  import logging
 
104
  pip3 install git+https://github.com/D4Vinci/Scrapling.git@dev
105
  ```
106
 
 
 
 
 
 
 
 
 
 
107
  ## Tests
108
  Scrapling includes a comprehensive test suite that can be executed with pytest. However, first, you need to install all libraries and `pytest-plugins` listed in `tests/requirements.txt`. Then, running the tests will result in an output like this:
109
  ```bash
 
115
  configfile: pytest.ini
116
  plugins: asyncio-1.2.0, anyio-4.11.0, xdist-3.8.0, httpbin-2.1.0, cov-7.0.0
117
  asyncio: mode=Mode.STRICT, asyncio_default_fixture_loop_scope=function, asyncio_default_test_loop_scope=function
118
+ 10 workers [515 items]
119
+ scheduling tests via LoadScheduling
120
+
121
  ...<shortened>...
122
+
123
  =============================== 271 passed in 52.68s ==============================
124
  ```
125
+ Here, `-n auto` runs tests in parallel across multiple processes to increase speed.
126
+
127
+ **Note:** You may need to run browser tests sequentially (`DynamicFetcher`/`StealthyFetcher`) to avoid conflicts. To run non-browser tests in parallel and browser tests separately:
128
+ ```bash
129
+ # Non-browser tests (parallel)
130
+ pytest tests/ -k "not (DynamicFetcher or StealthyFetcher)" -n auto
131
+
132
+ # Browser tests (sequential)
133
+ pytest tests/ -k "DynamicFetcher or StealthyFetcher"
134
+ ```
135
 
136
  Bonus: You can also see the test coverage with the `pytest` plugin below
137
  ```bash
138
  pytest --cov=scrapling tests/
139
  ```
140
+
141
+ ## Building Documentation
142
+ Documentation is built using [Zensical](https://zensical.org/). You can build it locally using the following commands:
143
+ ```bash
144
+ pip install zensical
145
+ pip install -r docs/requirements.txt
146
+ zensical build --clean # Build the static site
147
+ zensical serve # Local preview
148
+ ```