First version of Scrapling full documentation

docs/Core/using scrapling custom types.md

@@ -1,21 +0,0 @@
-> You can take advantage from the custom-made types for Scrapling and use it outside the library if you want. It's better than copying their code after all :)
-
-### All current types can be imported alone like below
-```python
->>> from scrapling.core.custom_types import TextHandler, AttributesHandler
-
->>> somestring = TextHandler('{}')
->>> somestring.json()
-'{}'
->>> somedict_1 = AttributesHandler({'a': 1})
->>> somedict_2 = AttributesHandler(a=1)
-```
-
-Note `TextHandler` is a sub-class of Python's `str` so all normal operations/methods that work with Python strings will work.
-If you want to check for the type in your code, it's better to depend on Python built-in function `issubclass`.
-
-The class `AttributesHandler` is a sub-class of `collections.abc.Mapping` so it's immutable (read-only) and all operations are inherited from it. The data passed can be accessed later though the `._data` method but careful it's of type `types.MappingProxyType` so it's immutable (read-only) as well (faster than `collections.abc.Mapping` by fractions of seconds).
-
-So basically to make it simple to you if you are new to Python, the same operations and methods from Python standard `dict` type will all work with class `AttributesHandler` except the ones that try to modify the actual data.
-
-If you want to modify the data inside `AttributesHandler`, you have to convert it to dictionary first like with using the `dict` function and modify it outside.

docs/Examples/selectorless_stackoverflow.py

@@ -1,25 +0,0 @@
-"""
-I only made this example to show how Scrapling features can be used to scrape a website without writing any selector
-    so this script doesn't depend on the website structure.
-"""
-
-import requests
-
-from scrapling import Adaptor
-
-response = requests.get('https://stackoverflow.com/questions/tagged/web-scraping?sort=MostVotes&filters=NoAcceptedAnswer&edited=true&pagesize=50&page=2')
-page = Adaptor(response.text, url=response.url)
-# First we will extract the first question title and its author based on the text content
-first_question_title = page.find_by_text('Run Selenium Python Script on Remote Server')
-first_question_author = page.find_by_text('Ryan')
-# because this page changes a lot
-if first_question_title and first_question_author:
-    # If you want you can extract other questions tags like below
-    first_question = first_question_title.find_ancestor(
-        lambda ancestor: ancestor.attrib.get('id') and 'question-summary' in ancestor.attrib.get('id')
-    )
-    rest_of_questions = first_question.find_similar()
-    # But since nothing to rely on to extract other titles/authors from these elements without CSS/XPath selectors due to the website nature
-    # We will get all the rest of the titles/authors in the page depending on the first title and the first author we got above as a starting point
-    for i, (title, author) in enumerate(zip(first_question_title.find_similar(), first_question_author.find_similar()), start=1):
-        print(i, title.text, author.text)

docs/Extending Scrapling/writing storage system.md

@@ -1,17 +0,0 @@
-Scrapling by default is using SQLite but in case you want to write your storage system to store elements properties there for the auto-matching, this tutorial got you covered.
-
-You might want to use FireBase for example and share the database between multiple spiders on different machines, it's a great idea to use an online database like that because this way the spiders will share with each others.
-
-So first to make your storage class work, it must do the big 3:
-1. Inherit from the abstract class `scrapling.storage_adaptors.StorageSystemMixin` and accept a string argument which will be the `url` argument to maintain the library logic.
-2. Use the decorator `functools.lru_cache` on top of the class itself to follow the Singleton design pattern as other classes.
-3. Implement methods `save` and `retrieve`, as you see from the type hints:
-   - The method `save` returns nothing and will get two arguments from the library
-        * The first one is of type `lxml.html.HtmlElement` which is the element itself, ofc. It must be converted to dictionary using the function `scrapling.utils._StorageTools.element_to_dict` so we keep the same format then saved to your database as you wish.
-        * The second one is string which is the identifier used for retrieval. The combination of this identifier and the `url` argument from initialization must be unique for each row or the auto-match will be messed up.
-   - The method `retrieve` takes a string which is the identifier, using it with the `url` passed on initialization the element's dictionary is retrieved from the database and returned if it exist otherwise it returns `None`
-> If the instructions weren't clear enough for you, you can check my implementation using SQLite3 in [storage_adaptors](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/storage_adaptors.py) file
-
-If your class satisfy this, the rest is easy. If you are planning to use the library in a threaded application, make sure that your class supports it. The default used class is thread-safe.
-
-There are some helper functions added to the abstract class if you want to use it. It's easier to see it for yourself in the [code](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/storage_adaptors.py), it's heavily commented :)

docs/api-reference/adaptor.md

@@ -0,0 +1,20 @@
+# Adaptor Class
+
+The `Adaptor` class is the core parsing engine in Scrapling that provides HTML parsing and element selection capabilities.
+
+Here's the reference information for the `Adaptor` class, with all its parameters, attributes, and methods.
+
+You can import the `Adaptor` class directly from `scrapling`:
+
+```python
+from scrapling.parser import Adaptor
+```
+
+## ::: scrapling.parser.Adaptor
+    handler: python
+    :docstring:
+
+## ::: scrapling.parser.Adaptors
+    handler: python
+    :docstring:
+

docs/api-reference/custom-types.md

@@ -0,0 +1,21 @@
+# Custom Types API Reference
+
+Here's the reference information for all custom types of classes Scrapling implemented, with all their parameters, attributes, and methods.
+
+You can import all of them directly like below:
+
+```python
+from scrapling.core.custom_types import TextHandler, TextHandlers, AttributesHandler
+```
+
+## ::: scrapling.core.custom_types.TextHandler
+    handler: python
+    :docstring:
+
+## ::: scrapling.core.custom_types.TextHandlers
+    handler: python
+    :docstring:
+
+## ::: scrapling.core.custom_types.AttributesHandler
+    handler: python
+    :docstring:

docs/api-reference/fetchers.md

@@ -0,0 +1,25 @@
+# Fetchers Classes
+
+Here's the reference information for all fetcher-type classes' parameters, attributes, and methods.
+
+You can import all of them directly like below:
+
+```python
+from scrapling.fetchers import Fetcher, AsyncFetcher, StealthyFetcher, PlayWrightFetcher
+```
+
+## ::: scrapling.fetchers.Fetcher
+    handler: python
+    :docstring:
+
+## ::: scrapling.fetchers.AsyncFetcher
+    handler: python
+    :docstring:
+
+## ::: scrapling.fetchers.PlayWrightFetcher
+    handler: python
+    :docstring:
+
+## ::: scrapling.fetchers.StealthyFetcher
+    handler: python
+    :docstring:

docs/benchmarks.md

@@ -0,0 +1,44 @@
+Scrapling isn't just powerful - it's also blazing fast. Scrapling implements many best practices, design patterns, and numerous optimizations to save fractions of seconds. All of that while focusing exclusively on parsing HTML documents.
+
+Here are benchmarks comparing Scrapling's parsing speed to popular Python libraries in two tests. 
+
+### Text Extraction Speed Test
+
+This test consists of extracting the text content of 5000 nested div elements.
+
+Here are the results comparing Scrapling to all well-known parsing libraries:
+
+
+| # |      Library      | Time (ms) | vs Scrapling | 
+|---|:-----------------:|:---------:|:------------:|
+| 1 |     Scrapling     |   5.44    |     1.0x     |
+| 2 |   Parsel/Scrapy   |   5.53    |    1.017x    |
+| 3 |     Raw Lxml      |   6.76    |    1.243x    |
+| 4 |      PyQuery      |   21.96   |    4.037x    |
+| 5 |    Selectolax     |   67.12   |   12.338x    |
+| 6 |   BS4 with Lxml   |  1307.03  |   240.263x   |
+| 7 |  MechanicalSoup   |  1322.64  |   243.132x   |
+| 8 | BS4 with html5lib |  3373.75  |   620.175x   |
+
+As you see, Scrapling is on par with Scrapy and slightly faster than Lxml, which both libraries are built on top of. These are the closest results to Scrapling. PyQuery is also built on top of Lxml, but Scrapling is four times faster.
+
+### Extraction By Text Speed Test
+
+Scrapling can find elements based on its text content and find elements similar to these elements. The only known library with these two features, too, is AutoScraper.
+
+So, we compared this to see how fast Scrapling can be in these two tasks compared to AutoScraper.
+
+Here are the results:
+
+|   Library   | Time (ms) | vs Scrapling |
+|-------------|:---------:|:------------:|
+|  Scrapling  |   2.51    |     1.0x     |
+| AutoScraper |   11.41   |    4.546x    |
+
+Scrapling can find elements with more methods and returns the entire element's `Adaptor` object, not only text like AutoScraper. So, to make this test fair, both libraries will extract an element with text, find similar elements, and then extract the text content for all of them. 
+
+As you see, Scrapling is still 4.5 times faster at the same task. 
+
+If we made Scrapling extract the elements only without stopping to extract each element's text, we would get speed twice as fast as this, but as I said, to make it fair comparison a bit :smile:
+
+> All benchmarks' results are an average of 100 runs. See our [benchmarks.py](https://github.com/D4Vinci/Scrapling/blob/main/benchmarks.py) for methodology and to run your comparisons.

docs/contributing.md

@@ -0,0 +1,102 @@
+Thank you for your interest in contributing to Scrapling! 
+
+Everybody is invited and welcome to contribute to Scrapling. 
+
+Smaller changes have a better chance of getting included in a timely manner. Adding unit tests for new features or test cases for bugs you've fixed helps us to ensure that the Pull Request (PR) is acceptable.
+
+There is a lot to do...
+
+- If you are not a developer, you can help us improve the documentation.
+- If you are a developer, most of the features I'm planning to add in the future are moved to [roadmap file](https://github.com/D4Vinci/Scrapling/blob/main/ROADMAP.md), so consider reading it.
+
+## Running tests
+Scrapling includes a comprehensive test suite that can be executed with pytest, but first, you need to install all libraries and `pytest-plugins` inside `tests/requirements.txt`. Then, running the tests will result in an output like this:
+   ```bash
+   $ pytest tests
+   =============================== test session starts ===============================
+   platform darwin -- Python 3.12.8, pytest-8.3.3, pluggy-1.5.0 -- /Users/<redacted>/.venv/bin/python3.12
+   cachedir: .pytest_cache
+   rootdir: /Users/<redacted>/scrapling
+   configfile: pytest.ini
+   plugins: cov-5.0.0, asyncio-0.25.0, base-url-2.1.0, httpbin-2.1.0, playwright-0.5.2, anyio-4.6.2.post1, xdist-3.6.1, typeguard-4.3.0
+   asyncio: mode=Mode.AUTO, asyncio_default_fixture_loop_scope=function
+   collected 83 items 
+   
+   ...<shortened>...
+   
+   =============================== 83 passed in 157.52s (0:02:37) =====================
+   ```
+Hence, you can add `-n auto` to the command above to run tests in threads to increase speed.
+
+Bonus: You can also see the test coverage with the pytest plugin below
+```bash
+pytest --cov=scrapling tests/
+```
+
+## Installing the latest unstable version from the dev branch
+```bash
+pip3 install git+https://github.com/D4Vinci/Scrapling.git@dev
+```
+
+## Development
+Setting the scrapling logging level to `debug` makes it easier to know what's happening in the background.
+   ```python
+   >>> import logging
+   >>> logging.getLogger("scrapling").setLevel(logging.DEBUG)
+   ```
+### Code Style
+
+We use:
+
+1. Type hints for better code clarity
+2. Flake8, bandit, isort, and other hooks through `pre-commit`. <br/>Please install the hooks before committing with:
+     ```bash
+     pip install pre-commit
+     pre-commit install
+     ```
+    It will run automatically on the code you push with each commit.
+3. Conventional commit messages format. We use the below format for commit messages
+   
+   | Prefix      | When to use it           |
+   |-------------|--------------------------|
+   | `feat:`     | New feature added        |
+   | `fix:`      | Bug fix                  |
+   | `docs:`     | Documentation change/add |
+   | `test:`     | Tests                    |
+   | `refactor:` | Code refactoring         |
+   | `chore:`    | Maintenance tasks        |
+
+   Example:
+   ```
+   feat: add auto-matching for similar elements
+   
+   - Added find_similar() method
+   - Implemented pattern matching
+   - Added tests and documentation
+   ```
+
+### Push changes to the library
+
+Then, the process is straightforward.
+
+ - Read [How to get faster PR reviews](https://github.com/kubernetes/community/blob/master/contributors/guide/pull-requests.md#best-practices-for-faster-reviews) by Kubernetes (but skip step 0 and 1)
+ - Fork Scrapling [Git repository](https://github.com/D4Vinci/Scrapling.git).
+ - Make your changes, and don't forget to create a separate virtual environment for this project.
+ - Ensure all tests are passing.
+ - Create a Pull Request against the [**dev**](https://github.com/D4Vinci/Scrapling/tree/dev) branch of Scrapling.
+
+A bonus: if you have more than one version of Python installed, you can use tox to run tests on each version with:
+```bash
+pip install tox
+tox
+```
+
+> Note: All tests are automatically run with each push on Github on all supported Python versions using tox, so ensure all tests pass, or your PR will not be accepted.
+
+
+## Building Documentation
+```bash
+pip install mkdocs-material
+mkdocs serve  # Local preview
+mkdocs build  # Build the static site
+```

docs/development/automatch_storage_system.md

@@ -0,0 +1,66 @@
+Scrapling uses SQLite by default, but this tutorial covers writing your storage system to store element properties there for auto-matching.
+
+You might want to use FireBase, for example, and share the database between multiple spiders on different machines. It's a great idea to use an online database like that because the spiders will share with each other.
+
+So first, to make your storage class work, it must do the big 3:
+
+1. Inherit from the abstract class `scrapling.core.storage_adaptors.StorageSystemMixin` and accept a string argument, which will be the `url` argument to maintain the library logic.
+2. Use the decorator `functools.lru_cache` on top of the class to follow the Singleton design pattern as other classes.
+3. Implement methods `save` and `retrieve`, as you see from the type hints:
+    - The method `save` returns nothing and will get two arguments from the library
+        * The first one is of type `lxml.html.HtmlElement`, which is the element itself. It must be converted to a dictionary using the function `element_to_dict` in submodule `scrapling.core.utils._StorageTools` to keep the same format and save it to your database as you wish.
+        * The second one is a string, the identifier used for retrieval. The combination result of this identifier and the `url` argument from initialization must be unique for each row, or the auto-match will be messed up.
+    - The method `retrieve` takes a string, which is the identifier; using it with the `url` passed on initialization, the element's dictionary is retrieved from the database and returned if it exists; otherwise, it returns `None`.
+
+> If the instructions weren't clear enough for you, you can check my implementation using SQLite3 in [storage_adaptors](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/core/storage_adaptors.py) file
+
+If your class meets these criteria, the rest is easy. If you plan to use the library in a threaded application, ensure your class supports it. The default used class is thread-safe.
+
+Some helper functions are added to the abstract class if you want to use them. It's easier to see it for yourself in the [code](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/core/storage_adaptors.py); it's heavily commented :)
+
+
+## Real-World Example: Redis Storage
+
+Here's a more practical example generated by AI using Redis:
+
+```python
+import redis
+import orjson
+from functools import lru_cache
+from scrapling.core.storage_adaptors import StorageSystemMixin
+from scrapling.core.utils import _StorageTools
+
+@lru_cache(None)
+class RedisStorage(StorageSystemMixin):
+    def __init__(self, host='localhost', port=6379, db=0, url=None):
+        super().__init__(url)
+        self.redis = redis.Redis(
+            host=host,
+            port=port,
+            db=db,
+            decode_responses=False
+        )
+        
+    def save(self, element, identifier: str) -> None:
+        # Convert element to dictionary
+        element_dict = _StorageTools.element_to_dict(element)
+        
+        # Create key
+        key = f"scrapling:{self._get_base_url()}:{identifier}"
+        
+        # Store as JSON
+        self.redis.set(
+            key,
+            orjson.dumps(element_dict)
+        )
+        
+    def retrieve(self, identifier: str) -> dict:
+        # Get data
+        key = f"scrapling:{self._get_base_url()}:{identifier}"
+        data = self.redis.get(key)
+        
+        # Parse JSON if exists
+        if data:
+            return orjson.loads(data)
+        return None
+```

docs/development/scrapling_custom_types.md

@@ -0,0 +1,21 @@
+> You can take advantage of the custom-made types for Scrapling and use them outside the library if you want. It's better than copying their code, after all :)
+
+### All current types can be imported alone like below
+```python
+>>> from scrapling.core.custom_types import TextHandler, AttributesHandler
+
+>>> somestring = TextHandler('{}')
+>>> somestring.json()
+'{}'
+>>> somedict_1 = AttributesHandler({'a': 1})
+>>> somedict_2 = AttributesHandler(a=1)
+```
+
+Note that `TextHandler` is a subclass of Python's `str`, so all normal operations/methods that work with Python strings will work.
+If you want to check for the type in your code, it's better to depend on Python's built-in function `issubclass`.
+
+The class `AttributesHandler` is a subclass of `collections.abc.Mapping`, so it's immutable (read-only), and all operations are inherited from it. The data passed can be accessed later through the `_data` property, but be careful; it's of type `types.MappingProxyType`, so it's immutable (read-only) as well (faster than `collections.abc.Mapping` by fractions of seconds).
+
+So, to make it simple for you if you are new to Python, the same operations and methods from the Python standard `dict` type will all work with class `AttributesHandler` except the ones that try to modify the actual data.
+
+If you want to modify the data inside `AttributesHandler,` you have to convert it to a dictionary first, like using the `dict` function, and then modify it outside.

docs/donate.md

@@ -0,0 +1,27 @@
+I've been working on Scrapling and other public projects in my spare time and have invested considerable resources and effort to provide these projects for free to the community. By becoming a sponsor, you'd be directly funding my coffee reserves, helping me continuously update existing projects and potentially create new ones.
+
+You can sponsor me directly through [Github sponsors program](https://github.com/sponsors/D4Vinci) or [Buy Me A Coffe](https://buymeacoffee.com/d4vinci). If you are a **company** and looking to **advertise** your business through Scrapling or another project, check out the available plans on my [Github Sponsors page](https://github.com/sponsors/D4Vinci).
+
+Below is the list of our Gold tier sponsors.
+
+Thank you, stay curious, and hack the planet! ❤️
+
+---
+
+## Top Sponsors
+### Scrapeless
+
+[Scrapeless Deep SerpApi](https://www.scrapeless.com/en/product/deep-serp-api?utm_source=website&utm_medium=ads&utm_campaign=scraping&utm_term=d4vinci) From $0.10 per 1,000 queries with a 1-2 second response time! 
+
+[![Scrapeless Banner](https://raw.githubusercontent.com/D4Vinci/Scrapling/main/images/scrapeless.jpg)](https://www.scrapeless.com/?utm_source=github&utm_medium=ads&utm_campaign=scraping&utm_term=D4Vinci)
+
+Deep SerpApi is a dedicated search engine designed for large language models (LLMs) and AI agents. It aims to provide real-time, accurate, and unbiased information to help AI applications retrieve and process data efficiently.
+
+- covering 20+ Google SERP scenarios and mainstream search engines.
+- support real-time data updates to ensure real-time and accurate information.
+- It can integrate information from all available online channels and search engines.
+- Deep SerpApi will simplify the process of integrating dynamic web information into AI solutions, and ultimately achieve an ALL-in-One API for one-click search and extraction of web data.
+- **Developer Support Program**: Integrate Scrapeless Deep SerpApi into your AI tools, applications or projects. [We already support Dify, and will soon support frameworks such as Langchain, Langflow, FlowiseAI]. Then share your results on GitHub or social media, and you will get a 1-12 month free developer support opportunity, up to 500 free usage per month.
+- 🚀 **Scraping API**: Effortless and highly customizable data extraction with a single API call, providing structured data from any website.
+- ⚡ **Scraping Browser**: AI-powered and LLM-driven, it simulates human-like behavior with genuine fingerprints and headless browser support, ensuring seamless, block-free scraping.
+- 🌐 **Proxies**: Use high-quality, rotating proxies to scrape top platforms like Amazon, Shopee, and more, with global coverage in 195+ countries.

docs/fetching/choosing.md

@@ -0,0 +1,77 @@
+## Introduction
+Fetchers are classes that can do requests or fetch pages for you easily in a single-line fashion with many features and then return a [Response](#response-object) object.
+
+This feature was introduced because the only option before v0.2 was to fetch the page as you wanted, then pass it manually to the `Adaptor` class and start playing with it.
+
+> Fetchers are not wrappers built on top of other libraries, but they use these libraries as an engine to make requests/fetch pages easily for you while fully utilizing that engine and adding features for you that aren't included in those engines
+
+## Fetchers Overview
+
+Scrapling provides three different fetcher classes, each designed for specific use cases.
+
+The following table compares them and can be quickly used for guidance.
+
+
+| Feature            | Fetcher        | PlayWrightFetcher                                                              | StealthyFetcher                                                                      |
+|--------------------|----------------|--------------------------------------------------------------------------------|--------------------------------------------------------------------------------------|
+| Relative speed     | ⭐⭐⭐⭐⭐          | ⭐⭐⭐                                                                            | ⭐⭐                                                                                   |
+| Stealth            | ⭐              | ⭐⭐                                                                             | ⭐⭐⭐⭐                                                                                 |
+| Anti-Bot options   | ⭐              | ⭐⭐                                                                             | ⭐⭐⭐⭐                                                                                 |
+| JavaScript loading | ❌              | ✅                                                                              | ✅                                                                                    |
+| Memory Usage       | ⭐              | ⭐⭐⭐                                                                            | ⭐⭐⭐                                                                                  |
+| Best used for      | Basic scraping | - Dynamically loaded websites <br/>- Small automation<br/>- Slight protections | - Dynamically loaded websites <br/>- Small automation <br/>- Complicated protections |
+| Browser(s)         | ❌              | Chromium and Google Chrome                                                     | Modified Firefox                                                                     |
+| Browser API used   | ❌              | PlayWright                                                                     | PlayWright                                                                           |
+| Setup Complexity   | Simple         | Simple                                                                         | Simple                                                                               |
+
+In the following pages, we will talk about each one in detail.
+
+## Parser configuration in all fetchers
+All fetchers classes share the same import, as you will see in the upcoming pages
+```python
+>>> from scrapling.fetchers import Fetcher, AsyncFetcher, StealthyFetcher, PlayWrightFetcher
+```
+Then you use it right away without initializing like this, and it will use the default parser settings:
+```python
+>>> page = StealthyFetcher.fetch('https://example.com') 
+```
+If you want to configure the parser ([Adaptor class](../parsing/main_classes.md#adaptor)) that will be used on the response before returning it for you, then do this first:
+```python
+>>> from scrapling.fetchers import Fetcher
+>>> Fetcher.configure(auto_match=True, encoding="utf8", keep_comments=False, keep_cdata=False)  # and the rest
+```
+or
+```python
+>>> from scrapling.fetchers import Fetcher
+>>> Fetcher.auto_match=True
+>>> Fetcher.encoding="utf8"
+>>> Fetcher.keep_comments=False
+>>> Fetcher.keep_cdata=False  # and the rest
+```
+Then, continue your code as usual.
+
+The available configuration arguments are: `auto_match`, `huge_tree`, `keep_comments`, `keep_cdata`, `storage`, and `storage_args`, which are the same ones you give to the `Adaptor` class. You can display the current configuration anytime by running `<fetcher_class>.display_config()`.
+
+> Note: The `auto_match` argument is disabled by default; you must enable it to use that feature.
+
+### Set parser config per request
+As you probably understood, the logic above for setting the parser config will work globally for all requests/fetches done through that class, and it's intended for simplicity.
+
+If your use case requires a different configuration for each request/fetch, you can pass a dictionary to the request method (`fetch`/`get`/`post`/...) to an argument named `custom_config`.
+
+## Response Object
+The `Response` object is the same as the [Adaptor](../parsing/main_classes.md#adaptor) class, but it has added details about the response like response headers, status, cookies, etc... as shown below:
+```python
+>>> from scrapling.fetchers import Fetcher
+>>> page = Fetcher.get('https://example.com')
+
+>>> page.status          # HTTP status code
+>>> page.reason          # Status message
+>>> page.cookies         # Response cookies as a dictionary
+>>> page.headers         # Response headers
+>>> page.request_headers # Request headers
+>>> page.history         # Response history of redirections, if any
+>>> page.body            # Raw response body
+>>> page.encoding        # Response encoding
+```
+All fetchers return the `Response` object.

docs/fetching/dynamic.md

@@ -0,0 +1,248 @@
+# Introduction
+
+Here, we will discuss the `PlayWrightFetcher` class. This class provides flexible browser automation with multiple configuration options and some stealth capabilities. It uses [PlayWright](https://playwright.dev/python/docs/intro) as an engine for fetching websites.
+
+As we will explain later, to automate the page, you need some knowledge of [PlayWright's Page API](https://playwright.dev/python/docs/api/class-page).
+
+## Basic Usage
+You have one primary way to import this Fetcher, which is the same for all fetchers.
+
+```python
+>>> from scrapling.fetchers import PlayWrightFetcher
+```
+Check out how to configure the parsing options [here](choosing.md#parser-configuration-in-all-fetchers)
+
+Now we will go over most of the arguments one by one with examples if you want to jump to a table of all arguments for quick reference [click here](#full-list-of-arguments)
+
+> Notes: 
+> 
+> 1. Every time you fetch a website with this fetcher, it waits by default for all JavaScript to fully load and execute, so you don't have to (waits for the `domcontentloaded` state).
+> 2. Of course, the async version of the `fetch` method is the `async_fetch` method.
+
+
+This fetcher currently provides 4 main run options, but they can be mixed as you want.
+
+Which are:
+
+### 1. Vanilla Playwright
+```python
+PlayWrightFetcher.fetch('https://example.com')
+```
+Using it like that will open a Chromium browser and fetch the page. There are no tricks or extra features; it's just a plain PlayWright API.
+
+### 2. Stealth Mode
+```python
+PlayWrightFetcher.fetch('https://example.com', stealth=True)
+```
+It's the same as the vanilla PlayWright option, but it provides a simple stealth mode suitable for websites with a small-to-medium protection layer(s).
+
+Some of the things this fetcher's stealth mode does include:
+
+  * Patching the CDP runtime fingerprint.
+  * Mimics some of the real browsers' properties by injecting several JS files and using custom options.
+  * Custom flags are used on launch to hide Playwright even more and make it faster.
+  * Generates real browser headers of the same type and user OS, then append them to the request's headers.
+
+### 3. Real Chrome
+```python
+PlayWrightFetcher.fetch('https://example.com', real_chrome=True)
+```
+If you have a Google Chrome browser installed, use this option. It's the same as the first option but will use the Google Chrome browser you installed on your device instead of Chromium.
+
+This will make your requests look more like requests coming from an actual human, so it's less detectable, and you can even use the `stealth=True` mode with it for better results like below:
+```python
+PlayWrightFetcher.fetch('https://example.com', real_chrome=True, stealth=True)
+```
+If you don't have Google Chrome installed and want to use this option, you can use the command below in the terminal to install it for the library instead of installing it manually:
+```commandline
+playwright install chrome
+```
+
+### 4. CDP Connection
+```python
+PlayWrightFetcher.fetch('https://example.com', cdp_url='ws://localhost:9222')
+```
+Instead of launching a browser locally (Chromium/Google Chrome), you can connect to a remote browser through the [Chrome DevTools Protocol](https://chromedevtools.github.io/devtools-protocol/).
+
+This fetcher takes it even a step further. You can use [NSTBrowser](https://app.nstbrowser.io/r/1vO5e5)'s [docker browserless](https://hub.docker.com/r/nstbrowser/browserless) option by passing the CDP URL and enabling `nstbrowser_mode` option like below
+```python
+PlayWrightFetcher.fetch('https://example.com', cdp_url='ws://localhost:9222', nstbrowser_mode=True)
+```
+There's also a `nstbrowser_config` argument to send the config you want to send with the requests to the NSTBrowser. If you leave it empty, Scrapling defaults to an optimized NSTBrowser's docker browserless config.
+
+## Full list of arguments
+Scrapling provides many options with this fetcher, which works in all modes except the [NSTBrowser](https://app.nstbrowser.io/r/1vO5e5) mode. To make it as simple as possible, we will list the options here and give examples of using most of them.
+
+|      Argument       | Description                                                                                                                                                                                                                                                                                                                                                                                                       | Optional |
+|:-------------------:|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:--------:|
+|         url         | Target url                                                                                                                                                                                                                                                                                                                                                                                                        |    ❌     |
+|      headless       | Pass `True` to run the browser in headless/hidden (**default**) or `False` for headful/visible mode.                                                                                                                                                                                                                                                                                                              |    ✔️    |
+|  disable_resources  | Drop requests of unnecessary resources for a speed boost. It depends, but it made requests ~25% faster in my tests for some websites.<br/>Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`. _This can help save your proxy usage, but be careful with this option as it makes some websites never finish loading._ |    ✔️    |
+|      useragent      | Pass a useragent string to be used. **Otherwise, the fetcher will generate and use a real Useragent of the same browser.**                                                                                                                                                                                                                                                                                        |    ✔️    |
+|    network_idle     | Wait for the page until there are no network connections for at least 500 ms.                                                                                                                                                                                                                                                                                                                                     |    ✔️    |
+|       timeout       | The timeout (milliseconds) used in all operations and waits through the page. The default is 30000.                                                                                                                                                                                                                                                                                                               |    ✔️    |
+|        wait         | The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the `Response` object.                                                                                                                                                                                                                                                                              |    ✔️    |
+|     page_action     | Added for automation. Pass a function that takes the `page` object and does the necessary automation, then returns `page` again.                                                                                                                                                                                                                                                                                  |    ✔️    |
+|    wait_selector    | Wait for a specific css selector to be in a specific state.                                                                                                                                                                                                                                                                                                                                                       |    ✔️    |
+| wait_selector_state | Scrapling will wait for the given state to be fulfilled for the selector given with `wait_selector`. _Default state is `attached`._                                                                                                                                                                                                                                                                               |    ✔️    |
+|    google_search    | Enabled by default, Scrapling will set the referer header as if this request came from a Google search for this website's domain name.                                                                                                                                                                                                                                                                            |    ✔️    |
+|    extra_headers    | A dictionary of extra headers to add to the request. The referer set by the `google_search` argument takes priority over the referer set here if used together.                                                                                                                                                                                                                                                   |    ✔️    |
+|        proxy        | The proxy to be used with requests. It can be a string or a dictionary with the keys 'server', 'username', and 'password' only.                                                                                                                                                                                                                                                                                   |    ✔️    |
+|     hide_canvas     | Add random noise to canvas operations to prevent fingerprinting.                                                                                                                                                                                                                                                                                                                                                  |    ✔️    |
+|    disable_webgl    | Disables WebGL and WebGL 2.0 support entirely.                                                                                                                                                                                                                                                                                                                                                                    |    ✔️    |
+|       stealth       | Enables stealth mode; you should always check the documentation to see what stealth mode does currently.                                                                                                                                                                                                                                                                                                          |    ✔️    |
+|     real_chrome     | If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch and use an instance of your browser and use it.                                                                                                                                                                                                                                                                   |    ✔️    |
+|       locale        | Set the locale for the browser if wanted. The default value is `en-US`.                                                                                                                                                                                                                                                                                                                                           |    ✔️    |
+|       cdp_url       | Instead of launching a new browser instance, connect to this CDP URL to control real browsers/NSTBrowser through CDP.                                                                                                                                                                                                                                                                                             |    ✔️    |
+|   nstbrowser_mode   | Enables NSTBrowser mode, **it have to be used with `cdp_url` argument or it will get completely ignored.**                                                                                                                                                                                                                                                                                                        |    ✔️    |
+|  nstbrowser_config  | The config you want to send with requests to the NSTBrowser. _Scrapling defaults to an optimized NSTBrowser's docker browserless config if you leave this argument empty._                                                                                                                                                                                                                                        |    ✔️    |
+
+
+## Examples
+It's easier to understand with examples, so let's look at it.
+
+### Resource Control
+
+```python
+# Disable unnecessary resources
+page = PlayWrightFetcher.fetch(
+    'https://example.com',
+    disable_resources=True  # Blocks fonts, images, media, etc...
+)
+```
+
+### Network Control
+
+```python
+# Wait for network idle (Consider fetch to be finished when there are no network connections for at least 500 ms)
+page = PlayWrightFetcher.fetch('https://example.com', network_idle=True)
+
+# Custom timeout (in milliseconds)
+page = PlayWrightFetcher.fetch('https://example.com', timeout=30000)  # 30 seconds
+
+# Proxy support
+page = PlayWrightFetcher.fetch(
+    'https://example.com',
+    proxy='http://username:password@host:port'  # Or it can be a dictionary with the keys 'server', 'username', and 'password' only
+)
+```
+
+### Browser Automation
+This is where your knowledge about [PlayWright's Page API](https://playwright.dev/python/docs/api/class-page) comes into play. The function you pass here takes the page object from Playwright's API, does what you want, and then returns it again for the current fetcher to continue working on it.
+
+This function is executed right after waiting for network_idle (if enabled) and before waiting for the `wait_selector` argument, so it can be used for many things, not just automation. You can alter the page as you want.
+
+In the example below, I used page [mouse events](https://playwright.dev/python/docs/api/class-mouse) to move the mouse wheel to scroll the page and then move the mouse.
+```python
+from playwright.sync_api import Page
+
+def scroll_page(page: Page):
+    page.mouse.wheel(10, 0)
+    page.mouse.move(100, 400)
+    page.mouse.up()
+    return page
+
+page = PlayWrightFetcher.fetch(
+    'https://example.com',
+    page_action=scroll_page
+)
+```
+Of course, if you use the async fetch version, the function must also be async.
+```python
+from playwright.async_api import Page
+
+async def scroll_page(page: Page):
+   await page.mouse.wheel(10, 0)
+   await page.mouse.move(100, 400)
+   await page.mouse.up()
+   return page
+
+page = await PlayWrightFetcher.async_fetch(
+    'https://example.com',
+    page_action=scroll_page
+)
+```
+
+### Wait Conditions
+
+```python
+# Wait for the selector
+page = PlayWrightFetcher.fetch(
+    'https://example.com',
+    wait_selector='h1',
+    wait_selector_state='visible'
+)
+```
+This is the last wait the fetcher will do before returning the response (if enabled). You pass a CSS selector to the `wait_selector` argument, and the fetcher will wait for the state you passed in the `wait_selector_state` argument to be fulfilled. If you didn't pass a state, the default would be `attached`, which means it will wait for the element to be present in the DOM.
+
+After that, the fetcher will check again to see if all JS files are loaded and executed (the `domcontentloaded` state) and wait for them to be. If you have enabled `network_idle` with this, the fetcher will wait for `network_idle` to be fulfilled again, as explained above.
+
+The states the fetcher can wait for can be either ([source](https://playwright.dev/python/docs/api/class-page#page-wait-for-selector)):
+
+- `attached`: Wait for an element to be present in DOM.
+- `detached`: Wait for an element to not be present in DOM.
+- `visible`: wait for an element to have a non-empty bounding box and no `visibility:hidden`. Note that an element without any content or with `display:none` has an empty bounding box and is not considered visible.
+- `hidden`: wait for an element to be either detached from DOM, or have an empty bounding box or `visibility:hidden`. This is opposite to the `'visible'` option.
+
+### Some Stealth Features
+
+```python
+# Full stealth mode
+page = PlayWrightFetcher.fetch(
+    'https://example.com',
+    stealth=True,
+    hide_canvas=True,
+    disable_webgl=True,
+    google_search=True
+)
+
+# Custom user agent
+page = PlayWrightFetcher.fetch(
+    'https://example.com',
+    useragent='Mozilla/5.0...'
+)
+
+# Set browser locale
+page = PlayWrightFetcher.fetch(
+    'https://example.com',
+    locale='en-US'
+)
+```
+Hence, the `hide_canvas` argument doesn't disable canvas but hides it by adding random noise to canvas operations to prevent fingerprinting. Also, if you didn't set a useragent (preferred), the fetcher will generate a real Useragent of the same browser and use it.
+
+The `google_search` argument is enabled by default, making the request look like it came from Google. So, a request for `https://example.com` will set the referer to `https://www.google.com/search?q=example`. Also, if used together, it takes priority over the referer set by the `extra_headers` argument.
+
+### General example
+```python
+from scrapling.fetchers import PlayWrightFetcher
+
+def scrape_dynamic_content():
+    # Use PlayWright for JavaScript content
+    page = PlayWrightFetcher.fetch(
+        'https://example.com/dynamic',
+        network_idle=True,
+        wait_selector='.content'
+    )
+    
+    # Extract dynamic content
+    content = page.css('.content')
+    
+    return {
+        'title': content.css_first('h1::text'),
+        'items': [
+            item.text for item in content.css('.item')
+        ]
+    }
+```
+
+## When to Use
+
+Use PlayWrightFetcher when:
+
+- Need browser automation
+- Want multiple browser options
+- Using a real Chrome browser
+- Need custom browser config
+- Want flexible stealth options 
+
+If you want more stealth and control without much config, check out the [StealthyFetcher](stealthy.md).

docs/fetching/static.md

@@ -0,0 +1,300 @@
+# Introduction
+
+The `Fetcher` class provides fast and lightweight HTTP requests with some stealth capabilities. This class uses [httpx](https://www.python-httpx.org/) as an engine for making requests. For advanced usages, you will need some knowledge about [httpx](https://www.python-httpx.org/), but it becomes simpler and simpler with user feedback and updates.
+
+## Basic Usage
+You have one primary way to import this Fetcher, which is the same for all fetchers.
+
+```python
+>>> from scrapling.fetchers import Fetcher
+```
+Check out how to configure the parsing options [here](choosing.md#parser-configuration-in-all-fetchers)
+
+### Shared arguments
+All methods for making requests here share some arguments, so let's discuss them first.
+
+- **url**: The URL you want to request, of course :)
+- **proxy**: As the name implies, the proxy for this request is used to route all traffic (HTTP and HTTPS). The format accepted here is `http://username:password@localhost:8030`.
+- **stealthy_headers**: Generate and use real browser's headers, then create a referer header as if this request came from a Google search page of this URL's domain. Enabled by default, all headers generated can be overwritten by you through the `headers` argument.
+- **follow_redirects**: As the name implies, tell the fetcher to follow redirections. Enabled by default
+- **timeout**: The timeout to wait for each request to be finished in milliseconds. The default is 30000ms (30 seconds).
+- **retries**: The number of retries that [httpx](https://www.python-httpx.org/) will do for failed requests. The default number of retries is 3.
+
+Other than this, you can pass any arguments that `httpx.<method_name>` takes, and that's why I said, in the beginning, you need a bit of knowledge about [httpx](https://www.python-httpx.org/), but in the following examples, we will try to cover most cases.
+
+### HTTP Methods
+Examples are the best way to explain this
+
+> Hence: `OPTIONS` and `HEAD` methods are not supported.
+#### GET
+```python
+>>> from scrapling.fetchers import Fetcher
+>>> # Basic GET
+>>> page = Fetcher.get('https://example.com')
+>>> page = Fetcher.get('https://httpbin.org/get', stealthy_headers=True, follow_redirects=True)
+>>> page = Fetcher.get('https://httpbin.org/get', proxy='http://username:password@localhost:8030')
+>>> # With parameters
+>>> page = Fetcher.get('https://example.com/search', params={'q': 'query'})
+>>>
+>>> # With headers
+>>> page = Fetcher.get('https://example.com', headers={'User-Agent': 'Custom/1.0'})
+>>> # Basic HTTP authentication
+>>> page = Fetcher.get("https://example.com", auth=("my_user", "password123"))
+```
+And for asynchronous requests, it's a small adjustment 
+```python
+>>> from scrapling.fetchers import AsyncFetcher
+>>> # Basic GET
+>>> page = await AsyncFetcher.get('https://example.com')
+>>> page = await AsyncFetcher.get('https://httpbin.org/get', stealthy_headers=True, follow_redirects=True)
+>>> page = await AsyncFetcher.get('https://httpbin.org/get', proxy='http://username:password@localhost:8030')
+>>> # With parameters
+>>> page = await AsyncFetcher.get('https://example.com/search', params={'q': 'query'})
+>>>
+>>> # With headers
+>>> page = await AsyncFetcher.get('https://example.com', headers={'User-Agent': 'Custom/1.0'})
+>>> # Basic HTTP authentication
+>>> page = await AsyncFetcher.get("https://example.com", auth=("my_user", "password123"))
+```
+Needless to say, the `page` object in all cases is [Response](choosing.md#response-object) object, which is an `Adaptor` as we said, so you will use it directly
+```python
+>>> page.css('.something.something')
+
+>>> page = Fetcher.get('https://api.github.com/events')
+>>> page.json()
+[{'id': '<redacted>',
+  'type': 'PushEvent',
+  'actor': {'id': '<redacted>',
+   'login': '<redacted>',
+   'display_login': '<redacted>',
+   'gravatar_id': '',
+   'url': 'https://api.github.com/users/<redacted>',
+   'avatar_url': 'https://avatars.githubusercontent.com/u/<redacted>'},
+  'repo': {'id': '<redacted>',
+...
+```
+#### POST
+```python
+>>> from scrapling.fetchers import Fetcher
+>>> # Basic POST
+>>> page = Fetcher.post('https://httpbin.org/post', data={'key': 'value'})
+>>> page = Fetcher.post('https://httpbin.org/post', data={'key': 'value'}, stealthy_headers=True, follow_redirects=True)
+>>> page = Fetcher.post('https://httpbin.org/post', data={'key': 'value'}, proxy='http://username:password@localhost:8030')
+>>> # Another example of form-encoded data
+>>> page = Fetcher.post('https://example.com/submit', data={'username': 'user', 'password': 'pass'})
+>>> # JSON data
+>>> page = Fetcher.post('https://example.com/api', json={'key': 'value'})
+>>> # Uploading file
+>>> r = Fetcher.post("https://httpbin.org/post", files={'upload-file': open('something.xlsx', 'rb')})
+```
+And for asynchronous requests, it's a small adjustment
+```python
+>>> from scrapling.fetchers import AsyncFetcher
+>>> # Basic POST
+>>> page = await AsyncFetcher.post('https://httpbin.org/post', data={'key': 'value'})
+>>> page = await AsyncFetcher.post('https://httpbin.org/post', data={'key': 'value'}, stealthy_headers=True, follow_redirects=True)
+>>> page = await AsyncFetcher.post('https://httpbin.org/post', data={'key': 'value'}, proxy='http://username:password@localhost:8030')
+>>> # Another example of form-encoded data
+>>> page = await AsyncFetcher.post('https://example.com/submit', data={'username': 'user', 'password': 'pass'})
+>>> # JSON data
+>>> page = await AsyncFetcher.post('https://example.com/api', json={'key': 'value'})
+>>> # Uploading file
+>>> r = await AsyncFetcher.post("https://httpbin.org/post", files={'upload-file': open('something.xlsx', 'rb')})
+```
+#### PUT
+```python
+>>> from scrapling.fetchers import Fetcher
+>>> # Basic PUT
+>>> page = Fetcher.put('https://example.com/update', data={'status': 'updated'})
+>>> page = Fetcher.put('https://example.com/update', data={'status': 'updated'}, stealthy_headers=True, follow_redirects=True)
+>>> page = Fetcher.put('https://example.com/update', data={'status': 'updated'}, proxy='http://username:password@localhost:8030')
+>>> # Another example of form-encoded data
+>>> page = Fetcher.put("https://httpbin.org/put", data={'key': ['value1', 'value2']})
+```
+And for asynchronous requests, it's a small adjustment
+```python
+>>> from scrapling.fetchers import AsyncFetcher
+>>> # Basic PUT
+>>> page = await AsyncFetcher.put('https://example.com/update', data={'status': 'updated'})
+>>> page = await AsyncFetcher.put('https://example.com/update', data={'status': 'updated'}, stealthy_headers=True, follow_redirects=True)
+>>> page = await AsyncFetcher.put('https://example.com/update', data={'status': 'updated'}, proxy='http://username:password@localhost:8030')
+>>> # Another example of form-encoded data
+>>> page = await AsyncFetcher.put("https://httpbin.org/put", data={'key': ['value1', 'value2']})
+```
+
+#### DELETE
+```python
+>>> from scrapling.fetchers import Fetcher
+>>> page = Fetcher.delete('https://example.com/resource/123')
+>>> page = Fetcher.delete('https://example.com/resource/123', stealthy_headers=True, follow_redirects=True)
+>>> page = Fetcher.delete('https://example.com/resource/123', proxy='http://username:password@localhost:8030')
+```
+And for asynchronous requests, it's a small adjustment
+```python
+>>> from scrapling.fetchers import AsyncFetcher
+>>> page = await AsyncFetcher.delete('https://example.com/resource/123')
+>>> page = await AsyncFetcher.delete('https://example.com/resource/123', stealthy_headers=True, follow_redirects=True)
+>>> page = await AsyncFetcher.delete('https://example.com/resource/123', proxy='http://username:password@localhost:8030')
+```
+
+## Examples
+Some well-rounded examples to aid newcomers to Web Scraping
+
+### Basic HTTP Request
+
+```python
+from scrapling.fetchers import Fetcher
+
+# Make a request
+page = Fetcher.get('https://example.com')
+
+# Check the status
+if page.status == 200:
+    # Extract title
+    title = page.css_first('title::text')
+    print(f"Page title: {title}")
+    
+    # Extract all links
+    links = page.css('a::attr(href)')
+    print(f"Found {len(links)} links")
+```
+
+### Product Scraping
+
+```python
+from scrapling.fetchers import Fetcher
+
+def scrape_products():
+    page = Fetcher.get('https://example.com/products')
+    
+    # Find all product elements
+    products = page.css('.product')
+    
+    results = []
+    for product in products:
+        results.append({
+            'title': product.css_first('.title::text'),
+            'price': product.css_first('.price::text').re_first(r'\d+\.\d{2}'),
+            'description': product.css_first('.description::text'),
+            'in_stock': product.has_class('in-stock')
+        })
+    
+    return results
+```
+
+### Pagination Handling
+
+```python
+from scrapling.fetchers import Fetcher
+
+def scrape_all_pages():
+    base_url = 'https://example.com/products?page={}'
+    page_num = 1
+    all_products = []
+    
+    while True:
+        # Get current page
+        page = Fetcher.get(base_url.format(page_num))
+        
+        # Find products
+        products = page.css('.product')
+        if not products:
+            break
+            
+        # Process products
+        for product in products:
+            all_products.append({
+                'name': product.css_first('.name::text'),
+                'price': product.css_first('.price::text')
+            })
+            
+        # Next page
+        page_num += 1
+        
+    return all_products
+```
+
+### Form Submission
+
+```python
+from scrapling.fetchers import Fetcher
+
+# Submit login form
+response = Fetcher.post(
+    'https://example.com/login',
+    data={
+        'username': 'user@example.com',
+        'password': 'password123'
+    }
+)
+
+# Check login success
+if response.status == 200:
+    # Extract user info
+    user_name = response.css_first('.user-name::text')
+    print(f"Logged in as: {user_name}")
+```
+
+### Table Extraction
+
+```python
+from scrapling.fetchers import Fetcher
+
+def extract_table():
+    page = Fetcher.get('https://example.com/data')
+    
+    # Find table
+    table = page.css_first('table')
+    
+    # Extract headers
+    headers = [
+        th.text for th in table.css('thead th')
+    ]
+    
+    # Extract rows
+    rows = []
+    for row in table.css('tbody tr'):
+        cells = [td.text for td in row.css('td')]
+        rows.append(dict(zip(headers, cells)))
+        
+    return rows
+```
+
+### Navigation Menu
+
+```python
+from scrapling.fetchers import Fetcher
+
+def extract_menu():
+    page = Fetcher.get('https://example.com')
+    
+    # Find navigation
+    nav = page.css_first('nav')
+    
+    menu = {}
+    for item in nav.css('li'):
+        link = item.css_first('a')
+        if link:
+            menu[link.text] = {
+                'url': link.attrib['href'],
+                'has_submenu': bool(item.css('.submenu'))
+            }
+            
+    return menu
+```
+
+## When to Use
+
+Use `Fetcher` when:
+
+- Need fast HTTP requests
+- Want minimal overhead
+- Don't need JavaScript
+- Want simple configuration
+- Need basic stealth features
+
+Use other fetchers when:
+
+- Need browser automation.
+- Need advanced anti-bot/stealth.
+- Need JavaScript support.

docs/fetching/stealthy.md

@@ -0,0 +1,218 @@
+# Introduction
+
+Here, we will discuss the `StealthyFetcher` class. This class is similar to [PlayWrightFetcher](dynamic.md#introduction) in many ways, like browser automation and using [PlayWright](https://playwright.dev/python/docs/intro) as an engine for fetching websites. The main difference is that this class provides advanced anti-bot protection bypass capabilities and a modified Firefox browser called [Camoufox](https://github.com/daijro/camoufox), from which most stealth comes.
+
+As with [PlayWrightFetcher](dynamic.md#introduction), you will need some knowledge about [PlayWright's Page API](https://playwright.dev/python/docs/api/class-page) to automate the page, as we will explain later.
+
+## Basic Usage
+You have one primary way to import this Fetcher, which is the same for all fetchers.
+
+```python
+>>> from scrapling.fetchers import StealthyFetcher
+```
+Check out how to configure the parsing options [here](choosing.md#parser-configuration-in-all-fetchers)
+
+> Notes: 
+> 
+> 1. Every time you fetch a website with this fetcher, it waits by default for all JavaScript to fully load and execute, so you don't have to (waits for the `domcontentloaded` state).
+> 2. Of course, the async version of the `fetch` method is the `async_fetch` method.
+
+## Full list of arguments
+Before jumping to [examples](#examples), here's the full list of arguments
+
+
+|       Argument       | Description                                                                                                                                                                                                                                                                                                                                                                                                       | Optional |
+|:--------------------:|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:--------:|
+|         url          | Target url                                                                                                                                                                                                                                                                                                                                                                                                        |    ❌     |
+|       headless       | Pass `True` to run the browser in headless/hidden (**default**), `virtual` to run it in virtual screen mode, or `False` for headful/visible mode. The `virtual` mode requires having `xvfb` installed.                                                                                                                                                                                                            |    ✔️    |
+|     block_images     | Prevent the loading of images through Firefox preferences. _This can help save your proxy usage, but be careful with this option as it makes some websites never finish loading._                                                                                                                                                                                                                                 |    ✔️    |
+|  disable_resources   | Drop requests of unnecessary resources for a speed boost. It depends, but it made requests ~25% faster in my tests for some websites.<br/>Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`. _This can help save your proxy usage, but be careful with this option as it makes some websites never finish loading._ |    ✔️    |
+|    google_search     | Enabled by default, Scrapling will set the referer header as if this request came from a Google search for this website's domain name.                                                                                                                                                                                                                                                                            |    ✔️    |
+|    extra_headers     | A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._                                                                                                                                                                                                                                                 |    ✔️    |
+|     block_webrtc     | Blocks WebRTC entirely.                                                                                                                                                                                                                                                                                                                                                                                           |    ✔️    |
+|     page_action      | Added for automation. A function that takes the `page` object and does the automation you need, then returns `page` again.                                                                                                                                                                                                                                                                                        |    ✔️    |
+|        addons        | List of Firefox addons to use. **Must be paths to extracted addons.**                                                                                                                                                                                                                                                                                                                                             |    ✔️    |
+|       humanize       | Humanize the cursor movement. The cursor movement takes either True or the MAX duration in seconds. The cursor typically takes up to 1.5 seconds to move across the window.                                                                                                                                                                                                                                       |    ✔️    |
+|     allow_webgl      | Enabled by default. Disabling WebGL is not recommended, as many WAFs now check if WebGL is enabled.                                                                                                                                                                                                                                                                                                               |    ✔️    |
+|        geoip         | Recommended to use with proxies; Automatically use IP's longitude, latitude, timezone, country, locale, & spoof the WebRTC IP address. It will also calculate and spoof the browser's language based on the distribution of language speakers in the target region.                                                                                                                                               |    ✔️    |
+|     os_randomize     | If enabled, Scrapling will randomize the OS fingerprints used. The default is matching the fingerprints with the current OS.                                                                                                                                                                                                                                                                                      |    ✔️    |
+|     disable_ads      | Disabled by default; this installs the `uBlock Origin` addon on the browser if enabled.                                                                                                                                                                                                                                                                                                                           |    ✔️    |
+|     network_idle     | Wait for the page until there are no network connections for at least 500 ms.                                                                                                                                                                                                                                                                                                                                     |    ✔️    |
+|       timeout        | The timeout used in all operations and waits through the page. It's in milliseconds, and the default is 30000.                                                                                                                                                                                                                                                                                                    |    ✔️    |
+|         wait         | The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the `Response` object.                                                                                                                                                                                                                                                                              |    ✔️    |
+|    wait_selector     | Wait for a specific css selector to be in a specific state.                                                                                                                                                                                                                                                                                                                                                       |    ✔️    |
+| wait_selector_state  | Scrapling will wait for the given state to be fulfilled for the selector given with `wait_selector`. _Default state is `attached`._                                                                                                                                                                                                                                                                               |    ✔️    |
+|        proxy         | The proxy to be used with requests. It can be a string or a dictionary with the keys 'server', 'username', and 'password' only.                                                                                                                                                                                                                                                                                   |    ✔️    |
+| additional_arguments | Arguments passed to Camoufox as additional settings that take higher priority than Scrapling's.                                                                                                                                                                                                                                                                                                                   |    ✔️    |
+
+
+## Examples
+It's easier to understand with examples, so now we will go over most of the arguments individually with examples.
+
+### Browser Modes
+
+```python
+# Headless/hidden mode (default)
+page = StealthyFetcher.fetch('https://example.com', headless=True)
+
+# Virtual display mode (requires having `xvfb` installed)
+page = StealthyFetcher.fetch('https://example.com', headless='virtual')
+
+# Visible browser mode
+page = StealthyFetcher.fetch('https://example.com', headless=False)
+```
+
+### Resource Control
+
+```python
+# Block images
+page = StealthyFetcher.fetch('https://example.com', block_images=True)
+
+# Disable unnecessary resources
+page = StealthyFetcher.fetch('https://example.com', disable_resources=True)  # Blocks fonts, images, media, etc.
+```
+
+### Additional stealth options
+
+```python
+page = StealthyFetcher.fetch(
+   'https://example.com',
+   block_webrtc=True,  # Block WebRTC
+   allow_webgl=False,  # Disable WebGL
+   humanize=True,      # Make the mouse move as how a human would move it
+   geoip=True,         # Use IP's longitude, latitude, timezone, country, and locale, then spoof the WebRTC IP address...
+   os_randomize=True,  # Randomize the OS fingerprints used. The default is matching the fingerprints with the current OS.
+   disable_ads=True,   # Block ads with uBlock Origin addon (enabled by default)
+   google_search=True
+)
+
+# Custom user agent
+page = StealthyFetcher.fetch(
+    'https://example.com',
+    useragent='Mozilla/5.0...'
+)
+
+# Custom humanization duration
+page = StealthyFetcher.fetch(
+    'https://example.com',
+    humanize=1.5  # Max 1.5 seconds for cursor movement
+)
+```
+
+The `google_search` argument is enabled by default. It makes the request as if it came from Google, so for a request for `https://example.com`, it will set the referer to `https://www.google.com/search?q=example`. Also, if used together, it takes priority over the referer set by the `extra_headers` argument.
+
+### Network Control
+
+```python
+# Wait for network idle (Consider fetch to be finished when there are no network connections for at least 500 ms)
+page = StealthyFetcher.fetch('https://example.com', network_idle=True)
+
+# Custom timeout (in milliseconds)
+page = StealthyFetcher.fetch('https://example.com', timeout=30000)  # 30 seconds
+
+# Proxy support
+page = StealthyFetcher.fetch(
+    'https://example.com',
+    proxy='http://username:password@host:port' # Or it can be a dictionary with the keys 'server', 'username', and 'password' only
+)
+```
+
+### Browser Automation
+This is where your knowledge about [PlayWright's Page API](https://playwright.dev/python/docs/api/class-page) comes into play. The function you pass here takes the page object from Playwright's API, does what you want, and then returns it again for the current fetcher to continue working on it.
+
+This function is executed right after waiting for `network_idle` (if enabled) and before waiting for the `wait_selector` argument, so it can be used for many things, not just automation. You can alter the page as you want.
+
+In the example below, I used page [mouse events](https://playwright.dev/python/docs/api/class-mouse) to move the mouse wheel to scroll the page and then move the mouse.
+```python
+from playwright.sync_api import Page
+
+def scroll_page(page: Page):
+    page.mouse.wheel(10, 0)
+    page.mouse.move(100, 400)
+    page.mouse.up()
+    return page
+
+page = StealthyFetcher.fetch(
+    'https://example.com',
+    page_action=scroll_page
+)
+```
+Of course, if you use the async fetch version, the function must also be async.
+```python
+from playwright.async_api import Page
+
+async def scroll_page(page: Page):
+   await page.mouse.wheel(10, 0)
+   await page.mouse.move(100, 400)
+   await page.mouse.up()
+   return page
+
+page = await StealthyFetcher.async_fetch(
+    'https://example.com',
+    page_action=scroll_page
+)
+```
+
+### Wait Conditions
+```python
+# Wait for the selector
+page = StealthyFetcher.fetch(
+    'https://example.com',
+    wait_selector='h1',
+    wait_selector_state='visible'
+)
+```
+This is the last wait the fetcher will do before returning the response (if enabled). You pass a CSS selector to the `wait_selector` argument, and the fetcher will wait for the state you passed in the `wait_selector_state` argument to be fulfilled. If you didn't pass a state, the default would be `attached`, which means it will wait for the element to be present in the DOM.
+
+After that, the fetcher will check again to see if all JS files are loaded and executed (the `domcontentloaded` state) and wait for them to be. If you have enabled `network_idle` with this, the fetcher will wait for `network_idle` to be fulfilled again, as explained above.
+
+The states the fetcher can wait for can be either ([source](https://playwright.dev/python/docs/api/class-page#page-wait-for-selector)):
+
+- `attached`: wait for the element to be present in DOM.
+- `detached`: wait for the element to not be present in DOM.
+- `visible`: wait for the element to have a non-empty bounding box and no `visibility:hidden`. Note that an element without any content or with `display:none` has an empty bounding box and is not considered visible.
+- `hidden`: Wait for the element to be detached from DOM, have an empty bounding box, or have `visibility:hidden`. This is opposite to the `'visible'` option.
+
+### Firefox Addons
+
+```python
+# Custom Firefox addons
+page = StealthyFetcher.fetch(
+    'https://example.com',
+    addons=['/path/to/addon1', '/path/to/addon2']
+)
+```
+The paths here must be paths of extracted addons, which will be installed automatically upon browser launch.
+
+### Real-world example (Amazon)
+This is for educational purposes only; this example was generated by AI, which shows too how easy it is to work with Scrapling through AI
+```python
+def scrape_amazon_product(url):
+    # Use StealthyFetcher to bypass protection
+    page = StealthyFetcher.fetch(url)
+
+    # Extract product details
+    return {
+        'title': page.css_first('#productTitle::text').clean(),
+        'price': page.css_first('.a-price .a-offscreen::text'),
+        'rating': page.css_first('[data-feature-name="averageCustomerReviews"] .a-popover-trigger .a-color-base::text'),
+        'reviews_count': page.css('#acrCustomerReviewText::text').re_first(r'[\d,]+'),
+        'features': [
+            li.clean() for li in page.css('#feature-bullets li span::text')
+        ],
+        'availability': page.css_first('#availability').get_all_text(strip=True),
+        'images': [
+            img.attrib['src'] for img in page.css('#altImages img')
+        ]
+    }
+```
+
+## When to Use
+
+Use StealthyFetcher when:
+
+- Bypassing anti-bot protection
+- Need a reliable browser fingerprint
+- Full JavaScript support needed
+- Want automatic stealth features
+- Need browser automation

docs/index.md

@@ -1,2 +1,107 @@
-# This section is still under work but any help is highly appreciated
-## I will try to make full detailed documentation with Sphinx ASAP.
+# Scrapling
+
+Scrapling is an Undetectable, high-performance, intelligent Web scraping library for Python 3 to make Web Scraping easy!
+
+Scrapling isn't only about making undetectable requests or fetching pages under the radar!
+
+It has its own parser that adapts to website changes and provides many element selection/querying options other than traditional selectors, powerful DOM traversal API, and many other features while significantly outperforming popular parsing alternatives.
+
+Scrapling is built from the ground up by Web scraping experts for beginners and experts. The goal is to provide powerful features while maintaining simplicity and minimal boilerplate code.
+
+```python
+>> from scrapling.fetchers import Fetcher, AsyncFetcher, StealthyFetcher, PlayWrightFetcher
+>> StealthyFetcher.auto_match = True
+# Fetch websites' source under the radar!
+>> page = StealthyFetcher.fetch('https://example.com', headless=True, network_idle=True)
+>> print(page.status)
+200
+>> products = page.css('.product', auto_save=True)  # Scrape data that survives website design changes!
+>> # Later, if the website structure changes, pass `auto_match=True`
+>> products = page.css('.product', auto_match=True)  # and Scrapling still finds them!
+```
+## Key Features
+### Fetch websites as you prefer with async support
+- **HTTP Requests**: Fast and stealthy HTTP requests with the `Fetcher` class.
+- **Dynamic Loading & Automation**: Fetch dynamic websites with the `PlayWrightFetcher` class through your real browser, Scrapling's stealth mode, Playwright's Chromium browser, or [NSTbrowser](https://app.nstbrowser.io/r/1vO5e5)'s browserless!
+- **Anti-bot Protections Bypass**: Easily bypass protections with the `StealthyFetcher` and `PlayWrightFetcher` classes.
+
+### Easy Scraping
+- **Smart Element Tracking**: Relocate elements after website changes using an intelligent similarity system and integrated storage.
+- **Flexible Selection**: CSS selectors, XPath selectors, filters-based search, text search, regex search, and more.
+- **Find Similar Elements**: Automatically locate elements similar to the element you found!
+- **Smart Content Scraping**: Extract data from multiple websites without specific selectors using Scrapling powerful features.
+
+### High Performance
+- **Lightning Fast**: Built from the ground up with performance in mind, outperforming most popular Python scraping libraries.
+- **Memory Efficient**: Optimized data structures for minimal memory footprint.
+- **Fast JSON serialization**: 10x faster than standard library.
+
+### Developer Friendly
+- **Powerful Navigation API**: Easy DOM traversal in all directions.
+- **Rich Text Processing**: All strings have built-in regex, cleaning methods, and more. All elements' attributes are optimized dictionaries that use less memory than standard dictionaries with added methods.
+- **Auto Selectors Generation**: Generate robust short and full CSS/XPath selectors for any element.
+- **Familiar API**: Similar to Scrapy/BeautifulSoup and the same CSS pseudo-elements used in Scrapy.
+- **Type hints**: Complete type/doc-strings coverage for future-proofing and best autocompletion support.
+
+## Star History
+Scrapling’s GitHub stars have grown steadily since its release (see chart below).
+
+<div id="chartContainer">
+  <a href="https://github.com/D4Vinci/Scrapling">
+    <img id="chartImage" alt="Star History Chart" src="https://api.star-history.com/svg?repos=D4Vinci/Scrapling&type=Date" height="400"/>
+  </a>
+</div>
+
+<script>
+const observer = new MutationObserver((mutations) => {
+  mutations.forEach((mutation) => {
+    if (mutation.attributeName === 'data-md-color-media') {
+      const colorMedia = document.body.getAttribute('data-md-color-media');
+      const isDarkScheme = document.body.getAttribute('data-md-color-scheme') === 'slate';
+      const chartImg = document.querySelector('#chartImage');
+      const baseUrl = 'https://api.star-history.com/svg?repos=D4Vinci/Scrapling&type=Date';
+      
+      if (colorMedia === '(prefers-color-scheme)' ? isDarkScheme : colorMedia.includes('dark')) {
+        chartImg.src = `${baseUrl}&theme=dark`;
+      } else {
+        chartImg.src = baseUrl;
+      }
+    }
+  });
+});
+
+observer.observe(document.body, {
+  attributes: true,
+  attributeFilter: ['data-md-color-media', 'data-md-color-scheme']
+});
+</script>
+
+## Installation
+Scrapling is a breeze to get started with!<br/>Starting from version 0.2.9, we require at least Python 3.9 to work.
+
+Run this command to install it with Python's pip.
+```bash
+pip3 install scrapling
+```
+You are ready if you plan to use the parser only (the `Adaptor` class).
+
+But if you are going to make requests or fetch pages with Scrapling, then run this command to install browsers' dependencies needed to use the Fetchers
+```bash
+scrapling install
+```
+If you have any installation issues, please open an [issue](https://github.com/D4Vinci/Scrapling/issues/new/choose).
+
+## How the documentation is organized
+Scrapling has a lot of documentation, so we try to follow a guideline called the [Diátaxis documentation framework](https://diataxis.fr/).
+
+## Support
+
+If you like Scrapling and want to support its development:
+
+- ⭐ Star the [GitHub repository](https://github.com/D4Vinci/Scrapling)
+- 💝 Consider [sponsoring the project or buying me a coffe](donate.md) :wink:
+- 🐛 Report bugs and suggest features through [GitHub Issues](https://github.com/D4Vinci/Scrapling/issues)
+
+## License
+
+This project is licensed under BSD-3 License. See the [LICENSE](https://github.com/D4Vinci/Scrapling/blob/main/LICENSE) file for details.

docs/overview.md

@@ -0,0 +1,328 @@
+We will start by quickly reviewing the parsing capabilities. Then, we will fetch websites with custom browsers, make requests, and parse the response.
+
+Here's an HTML document generated by ChatGPT we will be using as an example throughout this page:
+```html
+<html>
+  <head>
+    <title>Complex Web Page</title>
+    <style>
+      .hidden { display: none; }
+    </style>
+  </head>
+  <body>
+    <header>
+      <nav>
+        <ul>
+          <li> <a href="#home">Home</a> </li>
+          <li> <a href="#about">About</a> </li>
+          <li> <a href="#contact">Contact</a> </li>
+        </ul>
+      </nav>
+    </header>
+    <main>
+      <section id="products" schema='{"jsonable": "data"}'>
+        <h2>Products</h2>
+        <div class="product-list">
+          <article class="product" data-id="1">
+            <h3>Product 1</h3>
+            <p class="description">This is product 1</p>
+            <span class="price">$10.99</span>
+            <div class="hidden stock">In stock: 5</div>
+          </article>
+
+          <article class="product" data-id="2">
+            <h3>Product 2</h3>
+            <p class="description">This is product 2</p>
+            <span class="price">$20.99</span>
+            <div class="hidden stock">In stock: 3</div>
+          </article>
+
+          <article class="product" data-id="3">
+            <h3>Product 3</h3>
+            <p class="description">This is product 3</p>
+            <span class="price">$15.99</span>
+            <div class="hidden stock">Out of stock</div>
+          </article>
+        </div>
+      </section>
+      
+      <section id="reviews">
+        <h2>Customer Reviews</h2>
+        <div class="review-list">
+          <div class="review" data-rating="5">
+            <p class="review-text">Great product!</p>
+            <span class="reviewer">John Doe</span>
+          </div>
+          <div class="review" data-rating="4">
+            <p class="review-text">Good value for money.</p>
+            <span class="reviewer">Jane Smith</span>
+          </div>
+        </div>
+      </section>
+    </main>
+    <script id="page-data" type="application/json">
+      {
+        "lastUpdated": "2024-09-22T10:30:00Z",
+        "totalProducts": 3
+      }
+    </script>
+  </body>
+</html>
+```
+Starting with loading raw HTML above like this
+```python
+from scrapling.parser import Adaptor
+page = Adaptor(html_doc)
+page  # <data='<html><head><title>Complex Web Page</tit...'>
+```
+Get all text content on the page recursively
+```python
+page.get_all_text(ignore_tags=('script', 'style'))
+# 'Complex Web Page\nHome\nAbout\nContact\nProducts\nProduct 1\nThis is product 1\n$10.99\nIn stock: 5\nProduct 2\nThis is product 2\n$20.99\nIn stock: 3\nProduct 3\nThis is product 3\n$15.99\nOut of stock\nCustomer Reviews\nGreat product!\nJohn Doe\nGood value for money.\nJane Smith'
+```
+
+## Finding elements
+If there's an element you want to find on the page, you will! Your creativity level is the only limitation!
+
+Finding the first HTML `section` element
+```python
+section_element = page.find('section')
+# <data='<section id="products" schema='{"jsonabl...' parent='<main><section id="products" schema='{"j...'>
+```
+Find all `section` elements
+```python
+section_elements = page.find_all('section')
+# [<data='<section id="products" schema='{"jsonabl...' parent='<main><section id="products" schema='{"j...'>, <data='<section id="reviews"><h2>Customer Revie...' parent='<main><section id="products" schema='{"j...'>]
+```
+Find all `section` elements whose `id` attribute value is `products`
+```python
+section_elements = page.find_all('section', {'id':"products"})
+# Same as
+section_elements = page.find_all('section', id="products")
+# [<data='<section id="products" schema='{"jsonabl...' parent='<main><section id="products" schema='{"j...'>]
+```
+Find all `section` elements that its `id` attribute value contains `product`
+```python
+section_elements = page.find_all('section', {'id*':"product"})
+```
+Find all `h3` elements whose text content matches this regex `Product \d`
+```python
+page.find_all('h3', re.compile(r'Product \d'))
+# [<data='<h3>Product 1</h3>' parent='<article class="product" data-id="1"><h3...'>, <data='<h3>Product 2</h3>' parent='<article class="product" data-id="2"><h3...'>, <data='<h3>Product 3</h3>' parent='<article class="product" data-id="3"><h3...'>]
+```
+Find all `h3` and `h2` elements whose text content matches regex `Product` only
+```python
+page.find_all(['h3', 'h2'], re.compile(r'Product'))
+# [<data='<h3>Product 1</h3>' parent='<article class="product" data-id="1"><h3...'>, <data='<h3>Product 2</h3>' parent='<article class="product" data-id="2"><h3...'>, <data='<h3>Product 3</h3>' parent='<article class="product" data-id="3"><h3...'>, <data='<h2>Products</h2>' parent='<section id="products" schema='{"jsonabl...'>]
+```
+Find all elements that its text content matches exactly `Products` (Whitespaces are not taken into consideration)
+```python
+page.find_by_text('Products', first_match=False)
+# [<data='<h2>Products</h2>' parent='<section id="products" schema='{"jsonabl...'>]
+```
+Or find all elements whose text content matches regex `Product \d`
+```python
+page.find_by_regex(r'Product \d', first_match=False)
+# [<data='<h3>Product 1</h3>' parent='<article class="product" data-id="1"><h3...'>, <data='<h3>Product 2</h3>' parent='<article class="product" data-id="2"><h3...'>, <data='<h3>Product 3</h3>' parent='<article class="product" data-id="3"><h3...'>]
+```
+Find all elements that are similar to the element you want
+```python
+target_element = page.find_by_regex(r'Product \d', first_match=True)
+# <data='<h3>Product 1</h3>' parent='<article class="product" data-id="1"><h3...'>
+target_element.find_similar()
+# [<data='<h3>Product 2</h3>' parent='<article class="product" data-id="2"><h3...'>, <data='<h3>Product 3</h3>' parent='<article class="product" data-id="3"><h3...'>]
+```
+Find the first element that matches a CSS selector
+```python
+page.css_first('.product-list [data-id="1"]')
+# <data='<article class="product" data-id="1"><h3...' parent='<div class="product-list"> <article clas...'>
+```
+Find all elements that match a CSS selector
+```python
+page.css('.product-list article')
+# [<data='<article class="product" data-id="1"><h3...' parent='<div class="product-list"> <article clas...'>, <data='<article class="product" data-id="2"><h3...' parent='<div class="product-list"> <article clas...'>, <data='<article class="product" data-id="3"><h3...' parent='<div class="product-list"> <article clas...'>]
+```
+Find the first element that matches an XPath selector
+```python
+page.xpath_first("//*[@id='products']/div/article")
+# <data='<article class="product" data-id="1"><h3...' parent='<div class="product-list"> <article clas...'>
+```
+Find all elements that match an XPath selector
+```python
+page.xpath("//*[@id='products']/div/article")
+# [<data='<article class="product" data-id="1"><h3...' parent='<div class="product-list"> <article clas...'>, <data='<article class="product" data-id="2"><h3...' parent='<div class="product-list"> <article clas...'>, <data='<article class="product" data-id="3"><h3...' parent='<div class="product-list"> <article clas...'>]
+```
+
+With this, we just scratched the surface of these functions; more advanced options with these selection methods are shown later.
+## Accessing elements' data
+It's as simple as
+```python
+>>> section_element.tag
+'section'
+>>> print(section_element.attrib)
+{'id': 'products', 'schema': '{"jsonable": "data"}'}
+>>> section_element.attrib['schema'].json()  # If an attribute value can be converted to json, then use `.json()` to convert it
+{'jsonable': 'data'}
+>>> section_element.text  # Direct text content
+''
+>>> section_element.get_all_text()  # All text content recursively
+'Products\nProduct 1\nThis is product 1\n$10.99\nIn stock: 5\nProduct 2\nThis is product 2\n$20.99\nIn stock: 3\nProduct 3\nThis is product 3\n$15.99\nOut of stock'
+>>> section_element.html_content  # The HTML content of the element
+'<section id="products" schema=\'{"jsonable": "data"}\'><h2>Products</h2>\n        <div class="product-list">\n          <article class="product" data-id="1"><h3>Product 1</h3>\n            <p class="description">This is product 1</p>\n            <span class="price">$10.99</span>\n            <div class="hidden stock">In stock: 5</div>\n          </article><article class="product" data-id="2"><h3>Product 2</h3>\n            <p class="description">This is product 2</p>\n            <span class="price">$20.99</span>\n            <div class="hidden stock">In stock: 3</div>\n          </article><article class="product" data-id="3"><h3>Product 3</h3>\n            <p class="description">This is product 3</p>\n            <span class="price">$15.99</span>\n            <div class="hidden stock">Out of stock</div>\n          </article></div>\n      </section>'
+>>> print(section_element.prettify())  # The prettified version
+'''
+<section id="products" schema='{"jsonable": "data"}'><h2>Products</h2>
+    <div class="product-list">
+      <article class="product" data-id="1"><h3>Product 1</h3>
+        <p class="description">This is product 1</p>
+        <span class="price">$10.99</span>
+        <div class="hidden stock">In stock: 5</div>
+      </article><article class="product" data-id="2"><h3>Product 2</h3>
+        <p class="description">This is product 2</p>
+        <span class="price">$20.99</span>
+        <div class="hidden stock">In stock: 3</div>
+      </article><article class="product" data-id="3"><h3>Product 3</h3>
+        <p class="description">This is product 3</p>
+        <span class="price">$15.99</span>
+        <div class="hidden stock">Out of stock</div>
+      </article>
+    </div>
+</section>
+'''
+>>> section_element.path  # All the ancestors in the DOM tree of this element
+[<data='<main><section id="products" schema='{"j...' parent='<body> <header><nav><ul><li> <a href="#h...'>,
+ <data='<body> <header><nav><ul><li> <a href="#h...' parent='<html><head><title>Complex Web Page</tit...'>,
+ <data='<html><head><title>Complex Web Page</tit...'>]
+>>> section_element.generate_css_selector
+'#products'
+>>> section_element.generate_full_css_selector
+'body > main > #products > #products'
+>>> section_element.generate_xpath_selector
+"//*[@id='products']"
+>>> section_element.generate_full_xpath_selector
+"//body/main/*[@id='products']"
+```
+
+## Navigation
+Using the elements we found above 
+
+```python
+>>> section_element.parent
+<data='<main><section id="products" schema='{"j...' parent='<body> <header><nav><ul><li> <a href="#h...'>
+>>> section_element.parent.tag
+'main'
+>>> section_element.parent.parent.tag
+'body'
+>>> section_element.children
+[<data='<h2>Products</h2>' parent='<section id="products" schema='{"jsonabl...'>,
+ <data='<div class="product-list"> <article clas...' parent='<section id="products" schema='{"jsonabl...'>]
+>>> section_element.siblings
+[<data='<section id="reviews"><h2>Customer Revie...' parent='<main><section id="products" schema='{"j...'>]
+>>> section_element.next  # gets the next element, the same logic applies to `quote.previous`
+<data='<section id="reviews"><h2>Customer Revie...' parent='<main><section id="products" schema='{"j...'>
+>>> section_element.children.css('h2::text')
+['Products']
+>>> page.css_first('[data-id="1"]').has_class('product')
+True
+```
+If your case needs more than the element's parent, you can iterate over the whole ancestors' tree of any element like the one below
+```python
+for ancestor in quote.iterancestors():
+    # do something with it...
+```
+You can search for a specific ancestor of an element that satisfies a function; all you need to do is to pass a function that takes an `Adaptor` object as an argument and return `True` if the condition satisfies or `False` otherwise like below:
+```python
+>>> section_element.find_ancestor(lambda ancestor: ancestor.css('nav'))
+<data='<body> <header><nav><ul><li> <a href="#h...' parent='<html><head><title>Complex Web Page</tit...'>
+```
+
+## Fetching websites
+Instead of passing the raw HTML to Scrapling, you can get a website's response directly through HTTP requests or by fetching it from browsers.
+
+A fetcher is made for every use case.
+
+### HTTP Requests
+For simple HTTP requests, there's a `Fetcher` class that can be imported as below:
+```python
+from scrapling.fetchers import Fetcher
+```
+But that's class, so you will need to create an instance of the Fetcher first like this:
+```python
+from scrapling.fetchers import Fetcher
+fetcher = Fetcher()
+page = fetcher.get('https://httpbin.org/get')
+```
+This is intended, and you will find it with all fetchers because there are settings you can pass to `Fetcher()` initialization, but more on this later.
+
+If you are going to use the default settings anyway, you can do this instead for a cleaner approach:
+```python
+from scrapling.fetchers import Fetcher
+page = Fetcher.get('https://httpbin.org/get')
+```
+With that out of the way, here's how to do all HTTP methods:
+```python
+>>> from scrapling.fetchers import Fetcher
+>>> page = Fetcher.get('https://httpbin.org/get', stealthy_headers=True, follow_redirects=True)
+>>> page = Fetcher.post('https://httpbin.org/post', data={'key': 'value'}, proxy='http://username:password@localhost:8030')
+>>> page = Fetcher.put('https://httpbin.org/put', data={'key': 'value'})
+>>> page = Fetcher.delete('https://httpbin.org/delete')
+```
+For Async requests, you will just replace the import like below:
+```python
+>>> from scrapling.fetchers import AsyncFetcher
+>>> page = await AsyncFetcher.get('https://httpbin.org/get', stealthy_headers=True, follow_redirects=True)
+>>> page = await AsyncFetcher.post('https://httpbin.org/post', data={'key': 'value'}, proxy='http://username:password@localhost:8030')
+>>> page = await AsyncFetcher.put('https://httpbin.org/put', data={'key': 'value'})
+>>> page = await AsyncFetcher.delete('https://httpbin.org/delete')
+```
+
+> Note: You have the `stealthy_headers` argument, which, when enabled, makes requests to generate real browser headers and use them, including a referer header, as if this request came from Google's search of this URL's domain. It's enabled by default.
+
+This is just the tip of this fetcher; check the full page from [here](fetching/static.md)
+
+### Dynamic loading
+We have you covered if you deal with dynamic websites like most today!
+
+The `PlayWrightFetcher` class provides many options to fetch/load websites' pages through browsers.
+```python
+>>> from scrapling.fetchers import PlayWrightFetcher
+>>> page = PlayWrightFetcher.fetch('https://www.google.com/search?q=%22Scrapling%22', disable_resources=True)  # Vanilla Playwright option
+>>> page.css_first("#search a::attr(href)")
+'https://github.com/D4Vinci/Scrapling'
+>>> # The async version of fetch
+>>> page = await PlayWrightFetcher.async_fetch('https://www.google.com/search?q=%22Scrapling%22', disable_resources=True)
+>>> page.css_first("#search a::attr(href)")
+'https://github.com/D4Vinci/Scrapling'
+```
+It's named like that because it's built on top of [Playwright](https://playwright.dev/python/), and it currently provides 4 main run options that can be mixed as you want:
+
+- Vanilla Playwright without any modifications other than the ones you chose.
+- Stealthy Playwright with custom stealth mode explicitly written for it. It's not top-tier stealth mode but bypasses many online tests like [Sannysoft's](https://bot.sannysoft.com/). Check out the `StealthyFetcher` class below for more advanced stealth mode.
+- Real browsers by passing the `real_chrome` argument or the CDP URL of your browser to be controlled by the Fetcher, and most of the options can be enabled on it.
+- [NSTBrowser](https://app.nstbrowser.io/r/1vO5e5)'s [docker browserless](https://hub.docker.com/r/nstbrowser/browserless) option by passing the CDP URL and enabling `nstbrowser_mode` option.
+
+> Note: All requests done by this fetcher are waited by default for all javascript to be fully loaded and executed. In detail, it waits for the `load` and `domcontentloaded` load states to be reached; you can make it wait for the `networkidle` load state by passing 'network_idle=True', as you will see later.
+
+Again, this is just the tip of this fetcher. Check out the full page from [here](fetching/dynamic.md) for all details and the complete list of arguments.
+
+### Dynamic anti-protection loading
+We also have you covered if you deal with dynamic websites with annoying anti-protections!
+
+The `StealthyFetcher` class uses a modified Firefox browser called [Camoufox](https://github.com/daijro/camoufox), bypassing most anti-bot protections by default. Scrapling adds extra layers of flavors and configurations to further increase performance and undetectability.
+```python
+>>> page = StealthyFetcher().fetch('https://www.browserscan.net/bot-detection')  # Running headless by default
+>>> page.status == 200
+True
+>>> page = StealthyFetcher().fetch('https://www.browserscan.net/bot-detection', humanize=True, os_randomize=True) # and the rest of arguments...
+>>> # The async version of fetch
+>>> page = await StealthyFetcher().async_fetch('https://www.browserscan.net/bot-detection')
+>>> page.status == 200
+True
+```
+> Note: All requests done by this fetcher are waited by default for all javascript to be fully loaded and executed. In detail, it waits for the `load` and `domcontentloaded` load states to be reached; you can make it wait for the `networkidle` load state by passing 'network_idle=True', as you will see later.
+
+Again, this is just the tip of this fetcher. Check out the full page from [here](fetching/dynamic.md) for all details and the complete list of arguments.
+
+---
+
+That's Scrapling at a glance. If you want to learn more about it, continue to the next section.

docs/parsing/automatch.md

@@ -0,0 +1,220 @@
+## Introduction
+Auto-matching is one of Scrapling's most powerful features. It allows your scraper to survive website changes by intelligently tracking and relocating elements.
+
+Let's say you are scraping a page with a structure like this:
+```html
+<div class="container">
+    <section class="products">
+        <article class="product" id="p1">
+            <h3>Product 1</h3>
+            <p class="description">Description 1</p>
+        </article>
+        <article class="product" id="p2">
+            <h3>Product 2</h3>
+            <p class="description">Description 2</p>
+        </article>
+    </section>
+</div>
+```
+And you want to scrape the first product, the one with the `p1` ID. You will probably write a selector like this
+```python
+page.css('#p1')
+```
+When website owners implement structural changes like
+```html
+<div class="new-container">
+    <div class="product-wrapper">
+        <section class="products">
+            <article class="product new-class" data-id="p1">
+                <div class="product-info">
+                    <h3>Product 1</h3>
+                    <p class="new-description">Description 1</p>
+                </div>
+            </article>
+            <article class="product new-class" data-id="p2">
+                <div class="product-info">
+                    <h3>Product 2</h3>
+                    <p class="new-description">Description 2</p>
+                </div>
+            </article>
+        </section>
+    </div>
+</div>
+```
+The selector will no longer function, and your code needs maintenance. That's where Scrapling's auto-matching feature comes into play.
+
+With Scrapling, you can enable the `automatch` feature the first time you select an element, and the next time you select that element and it doesn't exist, Scrapling will remember its properties and search on the website for the element with the highest percentage of similarity to that element and without AI :)
+
+```python
+from scrapling import Adaptor, Fetcher
+# Before the change
+page = Adaptor(page_source, auto_match=True, url='example.com')
+# or
+Fetcher.auto_match = True
+page = Fetcher.get('https://example.com')
+# then
+element = page.css('#p1' auto_save=True)
+if not element:  # One day website changes?
+    element = page.css('#p1', auto_match=True)  # Scrapling still finds it!
+# the rest of your code...
+```
+Below, I will show you one usage example for this feature. Then, we will dive deep into how to use it and provide details about this feature.
+
+## Real-World Scenario
+Let's use a real website as an example and use one of the fetchers to fetch its source. To do this, we need to find a website that will soon change its design/structure, take a copy of its source, and then wait for the website to make the change. Of course, that's nearly impossible to know unless I know the website's owner, but that will make it a staged test, haha.
+
+To solve this issue, I will use [The Web Archive](https://archive.org/)'s [Wayback Machine](https://web.archive.org/). Here is a copy of [StackOverFlow's website in 2010](https://web.archive.org/web/20100102003420/http://stackoverflow.com/); pretty old, eh?</br>Let's test if the automatch feature can extract the same button in the old design from 2010 and the current design using the same selector :)
+
+If I want to extract the Questions button from the old design, I can use a selector like this: `#hmenus > div:nth-child(1) > ul > li:nth-child(1) > a` This selector is too specific because it was generated by Google Chrome.
+
+
+Now, let's test the same selector in both versions
+```python
+>> from scrapling import Fetcher
+>> selector = '#hmenus > div:nth-child(1) > ul > li:nth-child(1) > a'
+>> old_url = "https://web.archive.org/web/20100102003420/http://stackoverflow.com/"
+>> new_url = "https://stackoverflow.com/"
+>> Fetcher.configure(auto_match = True, automatch_domain='stackoverflow.com')
+>> 
+>> page = Fetcher.get(old_url, timeout=30)
+>> element1 = page.css_first(selector, auto_save=True)
+>> 
+>> # Same selector but used in the updated website
+>> page = Fetcher.get(new_url)
+>> element2 = page.css_first(selector, auto_match=True)
+>> 
+>> if element1.text == element2.text:
+...    print('Scrapling found the same element in the old and new designs!')
+'Scrapling found the same element in the old and new designs!'
+```
+Note that I used a new argument called `automatch_domain`; this is because, for Scrapling, these are two different domains(`archive.org` and `stackoverflow.com`), so scrapling will isolate their `auto_match` data. To tell Scrapling they are the same website, we need to pass the custom domain we want to use while saving auto-match data for them both so Scrapling doesn't isolate them.
+
+The code will be the same in a real-world scenario, except it will use the same URL for both requests, so you won't need to use the `automatch_domain` argument. This is the closest example I can give to real-world cases, so I hope it didn't confuse you :)
+
+Hence, in the two examples above, I used both the `Adaptor` class and the `Fetcher` class to show you that the logic for automatch is the same.
+
+## How the automatch feature works
+Auto-matching works in two phases:
+
+1. **Save Phase**: Store unique properties of elements
+2. **Match Phase**: Find elements with similar properties later
+
+Let's say you have an element you got through selection or any method and want the library to find it the next time you scrape this website, even if it had structural/design changes. 
+
+As little technical details as possible, the general logic goes as the following:
+
+  1. You tell Scrapling to save that element's unique properties in one of the ways we will show below.
+  2. Scrapling uses its configured database (SQLite by default) and saves each element's unique properties.
+  3. Now, because everything about the element can be changed or removed from the website's owner(s), nothing from the element can be used as a unique identifier for the database. To solve this issue, I made the storage system rely on two things:
+     1. The domain of the current website. If you are using the `Adaptor` class, you should pass it while initializing the class, or if you are using one of the fetchers, the domain will be taken from the URL automatically.
+     2. An `identifier` to query that element's properties from the database. You don't always have to set the identifier yourself, as you will see later when we discuss this.
+
+     Together, they will be used to retrieve the element's unique properties from the database later.
+
+  4. Later, when the website structural changes, you tell Scrapling to automatch the element. Scrapling retrieves the element's unique properties and matches all elements on the page against the unique properties we already have for this element. A score is calculated for their similarity to the element we want. In that comparison, everything is taken into consideration, as you will see later 
+  5. The element(s) with the highest similarity score to the wanted element are returned.
+
+### The unique properties
+You might wonder, if all aspects of an element can be removed or changed, what unique properties we are talking about.
+
+For Scrapling, the unique elements we are relying on are:
+
+- Element tag name, text, attributes (names and values), siblings (tag names only), and path (tag names only).
+- Element's parent tag name, attributes (names and values), and text.
+
+But you need to understand that the comparison between elements is not exact; it's more about finding how similar these values are. So everything is considered, even the values' order, like the order in which the element class names were written before and the order in which the same element class names are written now.
+
+## How to use automatch feature
+The automatch feature can be used on any element you have, and it's added as arguments to CSS/XPath Selection methods, as you saw above, but we will get back to that later.
+
+First, you must enable the automatch feature by passing `auto_match=True` to the [Adaptor](main_classes.md#adaptor) class when you initialize it or enable it in the fetcher you are using of the available fetchers, as we will show.
+
+Examples:
+```python
+>>> from scrapling import Adaptor, Fetcher
+>>> page = Adaptor(html_doc, auto_match=True)
+# OR
+>>> Fetcher.auto_match = True
+>>> page = Fetcher.fetch('https://example.com')
+```
+If you are using the [Adaptor](main_classes.md#adaptor) class, you need to pass the url of the website you are using with the argument `url` so Scrapling can separate the properties saved for each element by domain.
+
+If you didn't pass a URL, the word `default` will be used in place of the URL field while saving the element's unique properties. So, this will only be an issue if you used the same identifier later for a different website and didn't pass the URL parameter while initializing it. The save process will overwrite the previous data, and auto-matching only uses the latest saved properties.
+
+Besides those arguments, we have `storage` and `storage_args`. Both are for the class to be used to connect to the database; by default, it's set to the SQLite class that the library is using. Those arguments shouldn't matter unless you want to write your own storage system, which we will cover on a [separate page in the development section](../development/automatch_storage_system.md).
+
+Now, after enabling the automatch feature globally, you have two main ways to use it.
+
+### The CSS/XPath Selection way
+As you have seen in the example above, first, you have to use the `auto_save` argument while selecting an element that exists on the page like below
+```python
+element = page.css('#p1' auto_save=True)
+```
+and when the element doesn't exist, you can use the same selector and the `auto_match` argument, and the library will find it for you
+```python
+element = page.css('#p1', auto_match=True)
+```
+Pretty simple, eh?
+
+Well, a lot happened under the hood here. Remember the identifier part we mentioned before that you need to set so you can retrieve the element you want? Here, with the `css`/`css_first`/`xpath`/`xpath_first` methods, the identifier is set automatically as the selector you passed here to make things easier :)
+
+Also, that's why here, for all these methods, you can pass the `identifier` argument to set it yourself, and there are cases for this, or you can use it to save the properties with the `auto_save` argument.
+
+### The manual way
+You manually save and retrieve an element, then relocate it, which all happens within the automatch feature, as shown below. This allows you to automatch any element you have by any way or any selection method!
+
+First, let's say you got an element like this by text:
+```python
+>>> element = page.find_by_text('Tipping the Velvet', first_match=True)
+```
+You can save its unique properties with the `save` method like below, but you must set the identifier yourself. For this example, I chose `my_special_element` as an identifier, but it's best to use a meaningful identifier in your code for the same reason you use meaningful variable names :)
+```python
+>>> page.save(element, 'my_special_element')
+```
+Now, later, when you want to retrieve it and relocate it inside the page with auto-matching, it would be like this
+```python
+>>> element_dict = page.retrieve('my_special_element')
+>>> page.relocate(element_dict, adaptor_type=True)
+[<data='<a href="catalogue/tipping-the-velvet_99...' parent='<h3><a href="catalogue/tipping-the-velve...'>]
+>>> page.relocate(element_dict, adaptor_type=True).css('::text')
+['Tipping the Velvet']
+```
+Hence, the `retrieve` and relocate` methods are used.
+
+if you want to keep it as `lxml.etree` object, leave the `adaptor_type` argument
+```python
+>>> page.relocate(element_dict)
+[<Element a at 0x105a2a7b0>]
+```
+
+## Troubleshooting
+
+### No Matches Found
+```python
+# 1. Check if data was saved
+element_data = page.retrieve('identifier')
+if not element_data:
+    print("No data saved for this identifier")
+
+# 2. Try with different identifier
+products = page.css('.product', auto_match=True, identifier='old_selector')
+
+# 3. Save again with new identifier
+products = page.css('.new-product', auto_save=True, identifier='new_identifier')
+```
+
+### Wrong Elements Matched
+```python
+# Use more specific selectors
+products = page.css('.product-list .product', auto_save=True)
+
+# Or save with more context
+product = page.find_by_text('Product Name').parent
+page.save(product, 'specific_product')
+```
+
+## Known Issues
+In the auto-matching save process, the unique properties of the first element from the selection results are the only ones that get saved. So if the selector you are using selects different elements on the page in different locations, auto-matching will return the first element to you only when you relocate it later. This doesn't include combined CSS selectors (Using commas to combine more than one selector, for example), as these selectors get separated, and each selector gets executed alone.
+
+## Final thoughts
+Explaining this feature in detail without complications turned out to be challenging, but still, if there's something left unclear, you can head out to the [discussions section](https://github.com/D4Vinci/Scrapling/discussions), and I will reply to you ASAP or reach out to me privately and have a chat :)

docs/parsing/main_classes.md

@@ -0,0 +1,539 @@
+## Introduction
+After exploring the various ways to select elements with Scrapling and related features, Let's take a step back and examine the [Adaptor](#adaptor) class generally and other objects to better understand the parsing engine.
+
+The [Adaptor](#adaptor) class is the core parsing engine in Scrapling that provides HTML parsing and element selection capabilities. You can always import it with any of the following imports
+```python
+from scrapling import Adaptor
+from scrapling.parser import Adaptor
+```
+then use it directly as you already learned in the [overview](../overview.md) page
+```python
+adaptor = Adaptor(
+    text='<html>...</html>',
+    url='https://example.com'
+)
+
+# Then select elements as you like
+elements = adaptor.css('.product')
+```
+In Scrapling, the main object you deal with after passing an HTML source or fetching a website is, of course, an [Adaptor](#adaptor) object. Any operation you do, like selection, navigation, etc., will return either an [Adaptor](#adaptor) object or an [Adaptors](#adaptors) object, given that the result is element/elements from the page, not text or similar.
+
+In other words, the main page is a [Adaptor](#adaptor) object, and the elements within are [Adaptor](#adaptor) objects, and so on. Any text, such as the text content inside elements or the text inside element attributes, is a [TextHandler](#texthandler) object, and the attributes of each element are stored as [AttributesHandler](#attributeshandler). We will return to both objects later, so let's focus on the [Adaptor](#adaptor) object.
+
+## Adaptor
+### Arguments explained
+The most important ones are `text` and `body`. Both are used to pass the HTML code you want to parse, but the first one accepts `str`, and the latter accepts `bytes` like how you used to do with `parsel` :)
+
+Otherwise, you have the arguments `url`, `auto_match`, `storage`, and `storage_args`. All these arguments are settings used with the `auto_match` feature, and they don't make a difference if you are not going to use that feature, so just ignore them for now, and we will explain them in the [automatch](automatch.md) feature page.
+
+Then you have the arguments for adjustments for parsing or adjusting/manipulating the HTML while the library parsing it:
+
+- **encoding**: This is the encoding that will be used while parsing the HTML. The default is `UTF-8`.
+- **keep_comments**: This tells the library whether to keep HTML comments while parsing the page. It's disabled by default, as it can mess up your scraping in many ways.
+- **keep_cdata**: Same logic as the HTML comments. [cdata](https://stackoverflow.com/questions/7092236/what-is-cdata-in-html) is removed by default for cleaner HTML. This also means when you check for the raw html content, you will find it doesn't have the cdata.
+
+I have intended to ignore the arguments `huge_tree` and `root` to avoid making this page more complicated than needed.
+You may notice that I'm doing that a lot, and that's because it's something you don't need to know to use the library. The development section will cover these missing parts if you are that interested.
+
+After that, for the main page and elements within, most properties don't get initialized until you use it like the text content of a page/element, and this is one of the reasons for Scrapling speed :)
+
+### Properties
+You have already seen much of this on the [overview](../overview.md) page, but don't worry if you didn't. We will review it more thoroughly using more advanced methods/usages. For clarity, the properties for traversal are separated below in the [traversal](#traversal) section.
+
+Let's say we are parsing this HTML page for simplicity:
+```html
+<html>
+  <head>
+    <title>Some page</title>
+  </head>
+  <body>
+    <div class="product-list">
+      <article class="product" data-id="1">
+        <h3>Product 1</h3>
+        <p class="description">This is product 1</p>
+        <span class="price">$10.99</span>
+        <div class="hidden stock">In stock: 5</div>
+      </article>
+    
+      <article class="product" data-id="2">
+        <h3>Product 2</h3>
+        <p class="description">This is product 2</p>
+        <span class="price">$20.99</span>
+        <div class="hidden stock">In stock: 3</div>
+      </article>
+    
+      <article class="product" data-id="3">
+        <h3>Product 3</h3>
+        <p class="description">This is product 3</p>
+        <span class="price">$15.99</span>
+        <div class="hidden stock">Out of stock</div>
+      </article>
+    </div>
+
+    <script id="page-data" type="application/json">
+      {
+        "lastUpdated": "2024-09-22T10:30:00Z",
+        "totalProducts": 3
+      }
+    </script>
+  </body>
+</html>
+```
+Load the page directly as shown before:
+```python
+from scrapling import Adaptor
+page = Adaptor(html_doc)
+```
+Get all text content on the page recursively
+```python
+>>> page.get_all_text()
+'Some page\n\n    \n\n      \nProduct 1\nThis is product 1\n$10.99\nIn stock: 5\nProduct 2\nThis is product 2\n$20.99\nIn stock: 3\nProduct 3\nThis is product 3\n$15.99\nOut of stock'
+```
+Get the first article as explained before; we will use it as an example
+```python
+article = page.find('article')
+```
+With the same logic, get all text content on the element recursively
+```python
+>>> article.get_all_text()
+'Product 1\nThis is product 1\n$10.99\nIn stock: 5'
+```
+But if you try to get the direct text content, it will be empty; notice the logic difference
+```python
+>>> article.text
+''
+```
+The `get_all_text` method has the following optional arguments:
+
+1. **separator**: All strings collected will be concatenated using this separator. The default is '\n'
+2. **strip**: If enabled, strings will be stripped before concatenation. Disabled by default.
+3. **ignore_tags**: A tuple of all tag names you want to ignore in the final results. The default is `('script', 'style',)`.
+4. **valid_values**: If enabled, the method will only collect elements with real values, so all elements with empty text content or only whitespaces will be ignored. It's enabled by default
+
+By the way, the text returned here is not a standard string but a [TextHandler](#texthandler); we will get to this in detail later, so if the text content can be serialized to JSON, then use `.json()` on it
+```python
+>>> script = page.find('script')
+>>> script.json()
+{'lastUpdated': '2024-09-22T10:30:00Z', 'totalProducts': 3}
+```
+Let's continue to get the element tag
+```python
+>>> article.tag
+'article'
+```
+If you used it on the page directly, you will find you are operating on the root `html` element
+```python
+>>> page.tag
+'html'
+```
+Now, I think I hammered the (`page`/`element`) idea, so I won't return to it again.
+
+Getting the attributes of the element
+```python
+>>> print(article.attrib)
+{'class': 'product', 'data-id': '1'}
+```
+Get the HTML content of the element
+```python
+>>> article.html_content
+'<article class="product" data-id="1"><h3>Product 1</h3>\n        <p class="description">This is product 1</p>\n        <span class="price">$10.99</span>\n        <div class="hidden stock">In stock: 5</div>\n      </article>'
+```
+It's the same if you used the `.body` property
+```python
+>>> article.body
+'<article class="product" data-id="1"><h3>Product 1</h3>\n        <p class="description">This is product 1</p>\n        <span class="price">$10.99</span>\n        <div class="hidden stock">In stock: 5</div>\n      </article>'
+```
+Get the prettified version of the HTML content of the element
+```python
+>>> print(article.prettify())
+<article class="product" data-id="1"><h3>Product 1</h3>
+    <p class="description">This is product 1</p>
+    <span class="price">$10.99</span>
+    <div class="hidden stock">In stock: 5</div>
+</article>
+```
+To get all the ancestors in the DOM tree of this element
+```python
+>>> article.path
+[<data='<div class="product-list"> <article clas...' parent='<body> <div class="product-list"> <artic...'>,
+ <data='<body> <div class="product-list"> <artic...' parent='<html><head><title>Some page</title></he...'>,
+ <data='<html><head><title>Some page</title></he...'>]
+```
+Generate a CSS shortened selector if possible, or generate the full selector
+```python
+>>> article.generate_css_selector
+'body > div > article'
+>>> article.generate_full_css_selector
+'body > div > article'
+```
+Same case with XPath
+```python
+>>> article.generate_xpath_selector
+"//body/div/article"
+>>> article.generate_full_xpath_selector
+"//body/div/article"
+```
+
+### Traversal
+Using the elements we found above, we will go over the properties/methods for moving in the page in detail.
+
+If you are unfamiliar with the DOM tree or the tree data structure in general, the following traversal part can be confusing. I recommend you look up these concepts online for a better understanding.
+
+If you are too lazy to search about it, here's a quick explanation to give you a good idea.<br/>
+Simply put, the `html` element is the root of the website's tree, as every page starts with an `html` element.<br/>
+This element will be directly above elements like `head` and `body`. These are considered "children" of the `html` element, and the `html` element is considered their "parent." The element `body` is a "sibling" of the element `head` and vice versa.
+
+Accessing the parent of an element
+```python
+>>> article.parent
+<data='<div class="product-list"> <article clas...' parent='<body> <div class="product-list"> <artic...'>
+>>> article.parent.tag
+'div'
+```
+You can chain it as you want, which applies to all similar properties/methods we will review.
+```python
+>>> article.parent.parent.tag
+'body'
+```
+Get the children of an element
+```python
+>>> article.children
+[<data='<h3>Product 1</h3>' parent='<article class="product" data-id="1"><h3...'>,
+ <data='<p class="description">This is product 1...' parent='<article class="product" data-id="1"><h3...'>,
+ <data='<span class="price">$10.99</span>' parent='<article class="product" data-id="1"><h3...'>,
+ <data='<div class="hidden stock">In stock: 5</d...' parent='<article class="product" data-id="1"><h3...'>]
+```
+Get all elements underneath an element. It acts as a nested version of the `children` property
+```python
+>>> article.below_elements
+[<data='<h3>Product 1</h3>' parent='<article class="product" data-id="1"><h3...'>,
+ <data='<p class="description">This is product 1...' parent='<article class="product" data-id="1"><h3...'>,
+ <data='<span class="price">$10.99</span>' parent='<article class="product" data-id="1"><h3...'>,
+ <data='<div class="hidden stock">In stock: 5</d...' parent='<article class="product" data-id="1"><h3...'>]
+```
+This element returns the same result as the `children` property because its children don't have children.
+
+Another example of using the element with the `product-list` class will clear the difference between the `children` property and the `below_elements` property
+```python
+>>> products_list = page.css_first('.product-list')
+>>> products_list.children
+[<data='<article class="product" data-id="1"><h3...' parent='<div class="product-list"> <article clas...'>,
+ <data='<article class="product" data-id="2"><h3...' parent='<div class="product-list"> <article clas...'>,
+ <data='<article class="product" data-id="3"><h3...' parent='<div class="product-list"> <article clas...'>]
+
+>>> products_list.below_elements
+[<data='<article class="product" data-id="1"><h3...' parent='<div class="product-list"> <article clas...'>,
+ <data='<h3>Product 1</h3>' parent='<article class="product" data-id="1"><h3...'>,
+ <data='<p class="description">This is product 1...' parent='<article class="product" data-id="1"><h3...'>,
+ <data='<span class="price">$10.99</span>' parent='<article class="product" data-id="1"><h3...'>,
+ <data='<div class="hidden stock">In stock: 5</d...' parent='<article class="product" data-id="1"><h3...'>,
+ <data='<article class="product" data-id="2"><h3...' parent='<div class="product-list"> <article clas...'>,
+...]
+```
+Get the siblings of an element
+```python
+>>> article.siblings
+[<data='<article class="product" data-id="2"><h3...' parent='<div class="product-list"> <article clas...'>,
+ <data='<article class="product" data-id="3"><h3...' parent='<div class="product-list"> <article clas...'>]
+```
+Get the next element of the current element
+```python
+>>> article.next  # gets the next element, the same logic applies to `quote.previous`
+<data='<article class="product" data-id="2"><h3...' parent='<div class="product-list"> <article clas...'>
+```
+The same logic applies to the `previous` property
+```python
+>>> article.previous  # It's the first child, so it doesn't have a previous element
+>>> second_article = page.css_first('.product[data-id="2"]')
+>>> second_article.previous
+<data='<article class="product" data-id="1"><h3...' parent='<div class="product-list"> <article clas...'>
+```
+You can check easily and pretty fast if an element has a specific class name or not
+```python
+>>> article.has_class('product')
+True
+```
+If your case needs more than the element's parent, you can iterate over the whole ancestors' tree of any element, like the example below
+```python
+for ancestor in article.iterancestors():
+    # do something with it...
+```
+You can search for a specific ancestor of an element that satisfies a function; all you need to do is to pass a function that takes an [Adaptor](#adaptor) object as an argument and return `True` if the condition satisfies or `False` otherwise like below:
+```python
+>>> article.find_ancestor(lambda ancestor: ancestor.has_class('product-list'))
+<data='<div class="product-list"> <article clas...' parent='<body> <div class="product-list"> <artic...'>
+
+>>> article.find_ancestor(lambda ancestor: ancestor.css('.product-list'))  # Same result, different approach
+<data='<div class="product-list"> <article clas...' parent='<body> <div class="product-list"> <artic...'>
+```
+## Adaptors
+The class `Adaptors` is the "List" version of the [Adaptor](#adaptor) class. It inherits from the Python standard `List` type, so it shares all `List` properties and methods while adding more methods to make the operations you want to execute on the [Adaptor](#adaptor) instances within more straightforward.
+
+In the [Adaptor](#adaptor) class, all methods/properties that should return a group of elements return them as an [Adaptors](#adaptors) class instance. The only exceptions are when you use the CSS/XPath methods as follows:
+
+- If you selected a text node with the selector, then the return type will be [TextHandler](#texthandler)/[TextHandlers](#texthandlers). <br/>Examples:
+    ```python
+    >>> page.css('a::text')              # -> TextHandlers
+    >>> page.xpath('//a/text()')         # -> TextHandlers
+    >>> page.css_first('a::text')        # -> TextHandler
+    >>> page.xpath_first('//a/text()')   # -> TextHandler
+    >>> page.css('a::attr(href)')        # -> TextHandlers
+    >>> page.xpath('//a/@href')          # -> TextHandlers
+    >>> page.css_first('a::attr(href)')  # -> TextHandler
+    >>> page.xpath_first('//a/@href')    # -> TextHandler
+    ```
+- If you used a combined selector that returns mixed types, the result will be a Python standard `List`. <br/>Examples:
+  ```python
+  >>> page.css('.price_color')                               # -> Adaptors
+  >>> page.css('.product_pod a::attr(href)')                # -> TextHandlers
+  >>> page.css('.price_color, .product_pod a::attr(href)')  # -> List
+  ```
+
+Let's see what [Adaptors](#adaptors) class adds to the table with that out of the way.
+### Properties
+Apart from the normal operations on Python lists like iteration, slicing, etc...
+
+You can do the following:
+
+Execute CSS and XPath selectors directly on the [Adaptor](#adaptor) instances it has while the arguments and the return types are the same as [Adaptor](#adaptor)'s `css` and `xpath` methods. This, of course, makes chaining methods very straightforward.
+```python
+>>> page.css('.product_pod a')
+[<data='<a href="catalogue/a-light-in-the-attic_...' parent='<div class="image_container"> <a href="c...'>,
+ <data='<a href="catalogue/a-light-in-the-attic_...' parent='<h3><a href="catalogue/a-light-in-the-at...'>,
+ <data='<a href="catalogue/tipping-the-velvet_99...' parent='<div class="image_container"> <a href="c...'>,
+ <data='<a href="catalogue/tipping-the-velvet_99...' parent='<h3><a href="catalogue/tipping-the-velve...'>,
+ <data='<a href="catalogue/soumission_998/index....' parent='<div class="image_container"> <a href="c...'>,
+ <data='<a href="catalogue/soumission_998/index....' parent='<h3><a href="catalogue/soumission_998/in...'>,
+...]
+
+>>> page.css('.product_pod').css('a')  # Returns the same result
+[<data='<a href="catalogue/a-light-in-the-attic_...' parent='<div class="image_container"> <a href="c...'>,
+ <data='<a href="catalogue/a-light-in-the-attic_...' parent='<h3><a href="catalogue/a-light-in-the-at...'>,
+ <data='<a href="catalogue/tipping-the-velvet_99...' parent='<div class="image_container"> <a href="c...'>,
+ <data='<a href="catalogue/tipping-the-velvet_99...' parent='<h3><a href="catalogue/tipping-the-velve...'>,
+ <data='<a href="catalogue/soumission_998/index....' parent='<div class="image_container"> <a href="c...'>,
+ <data='<a href="catalogue/soumission_998/index....' parent='<h3><a href="catalogue/soumission_998/in...'>,
+...]
+```
+Run the `re` and `re_first` methods directly. They take the same arguments passed as the [Adaptor](#adaptor) class. I'm still leaving these methods to be explained in the [TextHandler](#texthandler) section below.
+
+However, in this class, the `re_first` behaves differently as it runs `re` on each [Adaptor](#adaptor) within and returns the first one with a result. The `re` method will return a [TextHandlers](#texthandlers) object as normal that has all the results combined in one [TextHandlers](#texthandlers) instance.
+```python
+>>> page.css('.price_color').re(r'[\d\.]+')
+['51.77',
+ '53.74',
+ '50.10',
+ '47.82',
+ '54.23',
+...]
+
+>>> page.css('.product_pod h3 a::attr(href)').re(r'catalogue/(.*)/index.html')
+['a-light-in-the-attic_1000',
+ 'tipping-the-velvet_999',
+ 'soumission_998',
+ 'sharp-objects_997',
+...]
+```
+With the `search` method, you can search quickly in the available [Adaptor](#adaptor) classes. The function you pass must accept an [Adaptor](#adaptor) instance as the first argument and return True/False. The method will return the first [Adaptor](#adaptor) instance that satisfies the function; otherwise, it will return `None`.
+```python
+# Find all the products with price '53.23'
+>>> search_function = lambda p: float(p.css('.price_color').re_first(r'[\d\.]+')) == 54.23
+>>> page.css('.product_pod').search(search_function)
+<data='<article class="product_pod"><div class=...' parent='<li class="col-xs-6 col-sm-4 col-md-3 co...'>
+```
+You can use the `filter` method, too, which takes a function like the `search` method but returns an `Adaptors` instance of all the [Adaptor](#adaptor) classes that satisfy the function
+```python
+# Find all products with prices over $50
+>>> filtering_function = lambda p: float(p.css('.price_color').re_first(r'[\d\.]+')) > 50
+>>> page.css('.product_pod').filter(filtering_function)
+[<data='<article class="product_pod"><div class=...' parent='<li class="col-xs-6 col-sm-4 col-md-3 co...'>,
+ <data='<article class="product_pod"><div class=...' parent='<li class="col-xs-6 col-sm-4 col-md-3 co...'>,
+ <data='<article class="product_pod"><div class=...' parent='<li class="col-xs-6 col-sm-4 col-md-3 co...'>,
+...]
+```
+
+## TextHandler
+This class is mandatory to understand, as all methods/properties that should return a string for you will return `TextHandler`, and the ones that should return a list of strings will return [TextHandlers](#texthandlers) instead.
+
+TextHandler is a subclass of the standard Python string, so you can do anything with it. So, what is the difference that requires a different naming?
+
+Of course, TextHandler provides extra methods and properties that the standard Python strings can't do. We will review them now, but remember that all methods and properties in all classes that return string(s) are returning TextHandler, which opens the door for creativity and makes the code shorter and cleaner, as you will see. Also, you can import it directly and use it on any string, which we will explain later.
+### Usage
+First, before discussing the added methods, you need to know that all operations on it, like slicing, accessing by index, etc., and methods like `split`, `replace`, `strip`, etc., all return a TextHandler again, so you can chain them as you want. If you find a method or property that returns a standard string instead of TextHandler, please open an issue, and we will override it as well.
+
+First, we start with the `re` and `re_first` methods. These are the same methods that exist in the rest of the classes ([Adaptor](#adaptor), [Adaptors](#adaptors), and [TextHandlers](#texthandlers)), so they will take the same arguments as well.
+
+    The `re` method takes a string/compiled regex pattern as the first argument. It searches the data for all strings matching the regex and returns them as a [TextHandlers](#texthandlers) instance. The `re_first` method takes the same arguments and behaves similarly, but as you probably figured out from the naming, it returns the first result only as a `TextHandler` instance.
+    
+    Also, it takes other helpful arguments, which are:
+    
+    - **replace_entities**: This is enabled by default. It replaces character entity references with their corresponding characters.
+      - **clean_match**: It's disabled by default. This makes the method ignore all whitespaces and consecutive spaces while matching.
+      - **case_sensitive**: It's enabled by default. As the name implies, disabling it will make the regex ignore letters case while compiling it.
+    
+    You have seen these examples before; the return result is [TextHandlers](#texthandlers) because we used the `re` method.
+    ```python
+    >>> page.css('.price_color').re(r'[\d\.]+')
+    ['51.77',
+     '53.74',
+     '50.10',
+     '47.82',
+     '54.23',
+    ...]
+    
+    >>> page.css('.product_pod h3 a::attr(href)').re(r'catalogue/(.*)/index.html')
+    ['a-light-in-the-attic_1000',
+     'tipping-the-velvet_999',
+     'soumission_998',
+     'sharp-objects_997',
+    ...]
+    ```
+    To explain the other arguments better, we will use a custom string for each example below
+    ```python
+    >>> from scrapling import TextHandler
+    >>> test_string = TextHandler('hi  there')  # Hence the two spaces
+    >>> test_string.re('hi there')
+    >>> test_string.re('hi there', clean_match=True)  # Using `clean_match` will clean the string before matching the regex
+    ['hi there']
+    
+    >>> test_string2 = TextHandler('Oh, Hi Mark')
+    >>> test_string2.re_first('oh, hi Mark')
+    >>> test_string2.re_first('oh, hi Mark', case_sensitive=False)  # Hence disabling `case_sensitive`
+    'Oh, Hi Mark'
+    
+    # Mixing arguments
+    >>> test_string.re('hi there', clean_match=True, case_sensitive=False)
+    ['hi There']
+    ```
+    Another use of the idea of replacing strings with `TextHandler` everywhere is a property like `html_content` returns `TextHandler` so you can do regex on the HTML content if you want:
+    ```python
+    >>> page.html_content.re('div class=".*">(.*)</div')
+    ['In stock: 5', 'In stock: 3', 'Out of stock']
+    ```
+
+- You also have the `.json()` method, which tries to convert the content to a json object quickly if possible; otherwise, it throws an error
+  ```python
+  >>> page.css_first('#page-data::text')
+    '\n      {\n        "lastUpdated": "2024-09-22T10:30:00Z",\n        "totalProducts": 3\n      }\n    '
+  >>> page.css_first('#page-data::text').json()
+    {'lastUpdated': '2024-09-22T10:30:00Z', 'totalProducts': 3}
+  ```
+  Hence, if you didn't specify a text node while selecting an element (like the text content or an attribute text content), the text content will be selected automatically like this
+  ```python
+  >>> page.css_first('#page-data').json()
+  {'lastUpdated': '2024-09-22T10:30:00Z', 'totalProducts': 3}
+  ```
+  The [Adaptor](#adaptor) class adds one thing here, too; let's say this is the page we are working with:
+  ```html
+  <html>
+      <body>
+          <div>
+            <script id="page-data" type="application/json">
+              {
+                "lastUpdated": "2024-09-22T10:30:00Z",
+                "totalProducts": 3
+              }
+            </script>
+          </div>
+      </body>
+  </html>
+  ```
+  The [Adaptor](#adaptor) class has the `get_all_text` method, which you should be aware of by now. This method returns a `TextHandler`, of course.<br/><br/>
+  So, as you know here, if you did something like this
+  ```python
+  >>> page.css_first('div::text').json()
+  ```
+  You will get an error because the `div` tag doesn't have direct text content that can be serialized to JSON; it actually doesn't have text content at all.<br/><br/>
+  In this case, the `get_all_text` method comes to the rescue, so you can do something like that
+  ```python
+  >>> page.css_first('div').get_all_text(ignore_tags=[]).json()
+    {'lastUpdated': '2024-09-22T10:30:00Z', 'totalProducts': 3}
+  ```
+  I used the `ignore_tags` argument here because the default value of it is `('script', 'style',)`, as you are aware.<br/><br/>
+  Another related behavior you should be aware of is the case while using any of the fetchers, which we will explain later. If you have a JSON response like this example:
+  ```python
+  >>> page = Adaptor("""{"some_key": "some_value"}""")
+  ```
+  Because the [Adaptor](#adaptor) class is optimized to deal with HTML pages, it will deal with it as a broken HTML response and fix it, so if you used the `html_content` property, you get this
+  ```python
+  >>> page.html_content
+  '<html><body><p>{"some_key": "some_value"}</p></body></html>'
+  ```
+  Here, you can use `json` method directly, and it will work
+  ```python
+  >>> page.json()
+  {'some_key': 'some_value'}
+  ```
+  You might wonder how this happened while the `html` tag lacks direct text?<br/>
+  Well, for these cases like JSON responses, I made the `.json()` method inside the [Adaptor](#adaptor) class to check if the current element doesn't have text content; it will use the `get_all_text` method directly.<br/><br/>It might sound hacky a bit but remember, Scrapling is currently optimized to work with HTML pages only so that's the best way till now to handle JSON responses currently without sacrificing speed. This will be changed in the upcoming versions.
+
+- Another handy method is `.clean()`, this will remove all white spaces and consecutive spaces for you and return a new `TextHandler`, wonderful
+```python
+>>> TextHandler('\n wonderful  idea, \reh?').clean()
+'wonderful idea, eh?'
+```
+
+- Another method that might be helpful in some cases is the `.sort()` method to sort the string for you as you do with lists
+```python
+>>> TextHandler('acb').sort()
+'abc'
+```
+Or do it in reverse:
+```python
+>>> TextHandler('acb').sort(reverse=True)
+'cba'
+```
+
+Other methods and properties will be added over time, but remember that this class is returned in place of strings nearly everywhere in the library.
+
+## TextHandlers
+You probably guessed it: This class is similar to [Adaptors](#adaptors) and [Adaptor](#adaptor), but here it inherits the same logic and method as standard lists, with only `re` and `re_first` as new methods.
+
+The only difference is that the `re_first` method logic here does `re` on each [TextHandler](#texthandler) within and returns the first result it has or `None`. Nothing is new to explain here, but new methods will be added here with time.
+
+## AttributesHandler
+This is a read-only version of Python's standard dictionary or `dict` that's only used to store the attributes of each element or each [Adaptor](#adaptor) instance, in other words.
+```python
+>>> print(page.find('script').attrib)
+{'id': 'page-data', 'type': 'application/json'}
+>>> type(page.find('script').attrib).__name__
+'AttributesHandler'
+```
+Because it's read-only, it will use fewer resources than the standard dictionary. Still, it has the same dictionary method/properties other than those allowing you to modify/override the data.
+
+It currently adds two extra simple methods:
+
+- The `search_values` method
+
+    In standard dictionaries, you can do `dict.get("key_name")` to check if a key exists. However, if you want to search by values instead of keys, it will take you some code lines. This method does that for you. It allows you to search the current attributes by values and returns a dictionary of each matching item.
+    
+    A simple example would be
+    ```python
+    >>> for i in page.find('script').attrib.search_values('page-data'):
+            print(i)
+    {'id': 'page-data'}
+    ```
+    But this method provides the `partial` argument as well, which allows you to search by part of the value:
+    ```python
+    >>> for i in page.find('script').attrib.search_values('page', partial=True):
+            print(i)
+    {'id': 'page-data'}
+    ```
+    These examples won't happen in the real world; most likely, a more real-world example would be using it with the `find_all` method to find all elements that have a specific value in their arguments:
+    ```python
+    >>> page.find_all(lambda element: list(element.attrib.search_values('product')))
+    [<data='<article class="product" data-id="1"><h3...' parent='<div class="product-list"> <article clas...'>,
+     <data='<article class="product" data-id="2"><h3...' parent='<div class="product-list"> <article clas...'>,
+     <data='<article class="product" data-id="3"><h3...' parent='<div class="product-list"> <article clas...'>]
+    ```
+    All these elements have 'product' as a value for the attribute `class`.
+    
+    Hence, I used the `list` function here because `search_values` returns a generator, so it would be `True` for all elements.
+
+  - The `json_string` property
+
+    This property converts current attributes to JSON string if the attributes are JSON serializable; otherwise, it throws an error
+    ```python
+    >>>  page.find('script').attrib.json_string
+    b'{"id":"page-data","type":"application/json"}'
+    ```

docs/parsing/selection.md

@@ -0,0 +1,512 @@
+## Introduction
+Scrapling currently supports parsing HTML pages exclusively, so it doesn't support XML feeds. This decision was made because the automatch feature won't work with XML, but that might change soon, so stay tuned :)
+
+In Scrapling, there are 5 main ways to find elements:
+
+1. CSS3 Selectors
+2. XPath Selectors
+3. Finding elements based on filters/conditions.
+4. Finding elements whose content contains specific text
+5. Finding elements whose content matches specific regex
+
+Of course, there are other indirect ways to find elements with Scrapling, but here we will discuss the main ways in detail. We will also bring up one of the most remarkable features of Scrapling: the ability to find elements that are similar to the element you have; you can jump to that section directly from [here](#finding-similar-elements).
+
+If you are new to Web Scraping, have little to no experience writing selectors, and want to start quickly, I recommend you jump directly to learning the `find`/`find_all` methods from [here](#filters-based-searching).
+
+## CSS/XPath selectors
+
+### What are CSS selectors?
+[CSS](https://en.wikipedia.org/wiki/CSS) is a language for applying styles to HTML documents. It defines selectors to associate those styles with specific HTML elements.
+
+Scrapling implements CSS3 selectors as described in the [W3C specification](http://www.w3.org/TR/2011/REC-css3-selectors-20110929/). CSS selectors support comes from cssselect, so it's better to read about which [selectors are supported from cssselect](https://cssselect.readthedocs.io/en/latest/#supported-selectors) and pseudo-functions/elements.
+
+Also, Scrapling implements some non-standard pseudo-elements like:
+
+* To select text nodes, use ``::text``
+* To select attribute values, use ``::attr(name)`` where name is the name of the attribute that you want the value of
+
+In short, if you come from Scrapy/Parsel, you will find the same logic for selectors here to make it easier. No need to implement a stranger logic to the one that most of us are used to :)
+
+To select elements with CSS selectors, you have the `css` and `css_first` methods. The latter is useful when you are interested in the first element it finds only, or if it's one element, etc., and the first when it's more than one, as it returns `Adaptors`.
+
+### What are XPath selectors?
+[XPath](https://en.wikipedia.org/wiki/XPath) is a language for selecting nodes in XML documents, which can also be used with HTML. This [cheatsheet] (https://devhints.io/xpath) is a good resource for learning about [XPath](https://en.wikipedia.org/wiki/XPath). Scrapling adds XPath selectors directly through LXML.
+
+In short, it is the same situation as CSS Selectors; if you come from Scrapy/Parsel, you will find the same logic for selectors here. BUT Scrapling doesn't implement the XPath extension function `has-class` as Scrapy/Parsel—instead, there's the `has_class` method that you can use on elements returned for the same purpose.
+
+To select elements with XPath selectors, you have the `xpath` and `xpath_first` methods. Again, these methods follow the same logic as the CSS selectors methods above.
+
+> Note that each method of `css`, `css_first`, `xpath`, and `xpath_first` have additional arguments, but we didn't explain them here as they are all about the automatch feature. The automatch feature will have its page later to be described in detail.
+
+### Selectors examples
+Let's see some shared examples of using CSS and XPath Selectors.
+
+Select all elements with the class `product`
+```python
+products = page.css('.product')
+products = page.xpath('//*[@class="product"]')
+```
+Note: The XPath one won't be accurate if there's another class; better rely on CSS for selecting by class
+
+Select the first element with the class `product`
+```python
+product = page.css_first('.product')
+product = page.xpath_first('//*[@class="product"]')
+```
+Which would be the same as doing
+```python
+product = page.css('.product')[0]
+product = page.xpath('//*[@class="product"]')[0]
+```
+Get the text of the first element with the `h1` tag name
+```python
+title = page.css_first('h1::text')
+title = page.xpath_first('//h1//text()')
+```
+Which is again the same as doing
+```python
+title = page.css_first('h1').text
+title = page.xpath_first('//h1').text
+```
+Get the `href` attribute of the first element with `a` tag name 
+```python
+link = page.css_first('a::attr(href)')
+link = page.xpath_first('//a/@href')
+```
+Select the text of the first element with the `h1` tag name, which contains 'Phone' and under an element with class 'product'
+```python
+title = page.css_first('.product h1:contains("Phone")::text')
+title = page.page.xpath_first('//*[@class="product"]//h1[contains(text(),"Phone")]/text()')
+```
+You can nest and chain selectors as you want, given that it returns results
+```python
+page.css_first('.product').css_first('h1:contains("Phone")::text')
+page.xpath_first('//*[@class="product"]').xpath_first('//h1[contains(text(),"Phone")]/text()')
+page.xpath_first('//*[@class="product"]').css_first('h1:contains("Phone")::text')
+```
+Another example
+
+All links that have 'image' in their 'href' attribute
+```python
+links = page.css('a[href*="image"]')
+links = page.xpath('//a[contains(@href, "image")]')
+for index, link in enumerate(links):
+    link_value = link.attrib['href']  # Cleaner than link.css('::attr(href)')
+    link_text = link.text
+    print(f'Link number {index} points to this url {link_value} with text content as "{link_text}"')
+```
+
+## Text-content selection
+Scrapling provides the ability to select elements based on their direct text content, and you have two ways to do this:
+
+1. Elements whose direct text content contains given text with many options through the `find_by_text` method.
+2. Elements whose direct text content matches the given regex pattern with many options through the `find_by_regex` method.
+
+What you can do with `find_by_text` can be done with `find_by_regex` if you are good enough with regular expressions (regex), but we are providing more options to make them easier for all users to access.
+
+With `find_by_text`, you will pass the text as the first argument; with the `find_by_regex` method, the regex pattern is the first. Both methods share the following arguments:
+
+* **first_match**: If `True` (the default), the method used will return the first result it finds.
+* **case_sensitive**: If `True`, the case of the letters will be considered.
+* **clean_match**: If `True`, all whitespaces and consecutive spaces will be ignored while matching.
+
+By default, Scrapling search for exact matching for the text you pass to `find_by_text`, so the text content of the wanted element have to be ONLY the text you inputted, but that's why it also has one extra argument, which is:
+
+* **partial**: If enabled, `find_by_text` will return elements that contain the input text. So it's not an exact match anymore
+
+Note: The method `find_by_regex` can accept both regular strings and a compiled regex pattern as its first argument, as you will see in the upcoming examples.
+
+### Finding Similar Elements
+One of the most remarkable new features that Scrapling puts on the table is the feature that allows the user to tell Scrapling to find elements similar to the element at hand. This feature inspiration came from the AutoScraper library, but here, it can be used on elements found by any method. Most likely, most of its usage would be after finding elements through text content like how AutoScraper works, so it would also be convenient to explain it here.
+
+So, how does it work?
+
+Imagine a scenario where you found a product by its title, for example, and you want to extract other products listed in the same table/container. With the element you have, you can simply call the method `.find_similar()` on it, and Scrapling will:
+
+1. Find all page elements with the same tree depth as this element. 
+2. All found elements will be checked, and those without the same tag name, parent tag name, and grandparent tag name will be dropped.
+3. Now we are sure (like 99% sure) that these elements are the ones we want, but as a last check, Scrapling will use fuzzy matching to drop the elements whose attributes don't look like the attributes of our element. There's a percentage to control this step, and I recommend you not play with it unless the default settings don't get the elements you want.
+
+That's a lot of talking, I know, but I had to go deep, I will give examples of using this method in the next section, but first, these are the arguments that can be passed to this method:
+
+* **similarity_threshold**: This is the percentage we discussed in step 3 for comparing elements' attributes. The default value is 0.2. In Simpler words, the attributes' values of both elements should be at least 20% similar. If you want to turn off this check (Step 3, basically), you can set this attribute to 0, but I recommend you read what other arguments do first.
+* **ignore_attributes**: The attribute names passed will be ignored while matching the attributes in the last step. The default value is `('href', 'src',)` because URLs can change a lot between elements, making them unreliable.
+* **match_text**: If `True`, the element's text content will be considered when matching. Using this in normal cases is not recommended, but it depends.
+
+Now, let's check out the examples below.
+
+### Examples
+Let's see some shared examples of finding elements with raw text and regex.
+
+I will use the `Fetcher` to clarify these examples, but it will be explained in detail later.
+```python
+from scrapling.fetchers import Fetcher
+page = Fetcher.get('https://books.toscrape.com/index.html')
+```
+Find the first element whose text fully matches this text
+```python
+>>> page.find_by_text('Tipping the Velvet')
+<data='<a href="catalogue/tipping-the-velvet_99...' parent='<h3><a href="catalogue/tipping-the-velve...'>
+```
+Combining it with `page.urljoin` to return the full URL from the relative `href`
+```python
+>>> page.find_by_text('Tipping the Velvet').attrib['href']
+'catalogue/tipping-the-velvet_999/index.html'
+>>> page.urljoin(page.find_by_text('Tipping the Velvet').attrib['href'])
+'https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html'
+```
+Get all matches if there are more (hence, it returned a list)
+```python
+>>> page.find_by_text('Tipping the Velvet', first_match=False)
+[<data='<a href="catalogue/tipping-the-velvet_99...' parent='<h3><a href="catalogue/tipping-the-velve...'>]
+```
+Get all elements that contain the word `the` (Partial matching)
+```python
+>>> results = page.find_by_text('the', partial=True, first_match=False)
+>>> [i.text for i in results]
+['A Light in the ...',
+ 'Tipping the Velvet',
+ 'The Requiem Red',
+ 'The Dirty Little Secrets ...',
+ 'The Coming Woman: A ...',
+ 'The Boys in the ...',
+ 'The Black Maria',
+ 'Mesaerion: The Best Science ...',
+ "It's Only the Himalayas"]
+```
+The search is case insensitive, so those results have `The`, not only the lowercase one `the`; let's limit the search to the elements with `the` only.
+```python
+>>> results = page.find_by_text('the', partial=True, first_match=False, case_sensitive=True)
+>>> [i.text for i in results]
+['A Light in the ...',
+ 'Tipping the Velvet',
+ 'The Boys in the ...',
+ "It's Only the Himalayas"]
+```
+Get the first element that its text content matches my price regex
+```python
+>>> page.find_by_regex(r'£[\d\.]+')
+<data='<p class="price_color">£51.77</p>' parent='<div class="product_price"> <p class="pr...'>
+>>> page.find_by_regex(r'£[\d\.]+').text
+'£51.77'
+```
+It's the same if you pass the compiled regex as well; Scrapling will detect the input type and act upon that:
+```python
+>>> import re
+>>> regex = re.compile(r'£[\d\.]+')
+>>> page.find_by_regex(regex)
+<data='<p class="price_color">£51.77</p>' parent='<div class="product_price"> <p class="pr...'>
+>>> page.find_by_regex(regex).text
+'£51.77'
+```
+Get all elements that match the regex
+```python
+>>> page.find_by_regex(r'£[\d\.]+', first_match=False)
+[<data='<p class="price_color">£51.77</p>' parent='<div class="product_price"> <p class="pr...'>,
+ <data='<p class="price_color">£53.74</p>' parent='<div class="product_price"> <p class="pr...'>,
+ <data='<p class="price_color">£50.10</p>' parent='<div class="product_price"> <p class="pr...'>,
+ <data='<p class="price_color">£47.82</p>' parent='<div class="product_price"> <p class="pr...'>,
+ ...]
+```
+And so on...
+
+Find all elements similar to the current element in location and attributes. For our case, ignore the 'title' attribute while matching
+```python
+>>> element = page.find_by_text('Tipping the Velvet')
+>>> element.find_similar(ignore_attributes=['title'])
+[<data='<a href="catalogue/a-light-in-the-attic_...' parent='<h3><a href="catalogue/a-light-in-the-at...'>,
+ <data='<a href="catalogue/soumission_998/index....' parent='<h3><a href="catalogue/soumission_998/in...'>,
+ <data='<a href="catalogue/sharp-objects_997/ind...' parent='<h3><a href="catalogue/sharp-objects_997...'>,
+...]
+```
+Notice that the number of elements is 19, not 20, because the current element is not included in the results.
+```python
+>>> len(element.find_similar(ignore_attributes=['title']))
+19
+```
+Get the `href` attribute from all similar elements
+```python
+>>> [
+    element.attrib['href']
+    for element in element.find_similar(ignore_attributes=['title'])
+]
+['catalogue/a-light-in-the-attic_1000/index.html',
+ 'catalogue/soumission_998/index.html',
+ 'catalogue/sharp-objects_997/index.html',
+ ...]
+```
+To increase the complexity a little bit, let's say we want to get all books' data using that element as a starting point for some reason
+```python
+>>> for product in element.parent.parent.find_similar():
+        print({
+            "name": product.css_first('h3 a::text'),
+            "price": product.css_first('.price_color').re_first(r'[\d\.]+'),
+            "stock": product.css('.availability::text')[-1].clean()
+        })
+{'name': 'A Light in the ...', 'price': '51.77', 'stock': 'In stock'}
+{'name': 'Soumission', 'price': '50.10', 'stock': 'In stock'}
+{'name': 'Sharp Objects', 'price': '47.82', 'stock': 'In stock'}
+...
+```
+### Advanced examples 
+See more advanced or real-world examples using the `find_similar` method.
+
+E-commerce Product Extraction
+```python
+def extract_product_grid(page):
+    # Find the first product card
+    first_product = page.find_by_text('Add to Cart').find_ancestor(
+        lambda e: e.has_class('product-card')
+    )
+
+    # Find similar product cards
+    products = first_product.find_similar()
+
+    return [
+        {
+            'name': p.css_first('h3::text'),
+            'price': p.css_first('.price::text').re_first(r'\d+\.\d{2}'),
+            'stock': 'In stock' in p.text,
+            'rating': p.css_first('.rating').attrib.get('data-rating')
+        }
+        for p in products
+    ]
+```
+Table Row Extraction
+```python
+def extract_table_data(page):
+    # Find the first data row
+    first_row = page.css_first('table tbody tr')
+
+    # Find similar rows
+    rows = first_row.find_similar()
+
+    return [
+        {
+            'column1': row.css_first('td:nth-child(1)::text'),
+            'column2': row.css_first('td:nth-child(2)::text'),
+            'column3': row.css_first('td:nth-child(3)::text')
+        }
+        for row in rows
+    ]
+```
+Form Field Extraction
+```python
+def extract_form_fields(page):
+    # Find first form field container
+    first_field = page.css_first('input').find_ancestor(
+        lambda e: e.has_class('form-field')
+    )
+
+    # Find similar field containers
+    fields = first_field.find_similar()
+
+    return [
+        {
+            'label': f.css_first('label::text'),
+            'type': f.css_first('input').attrib.get('type'),
+            'required': 'required' in f.css_first('input').attrib
+        }
+        for f in fields
+    ]
+```
+Extracting reviews from a website
+```python
+def extract_reviews(page):
+    # Find first review
+    first_review = page.find_by_text('Great product!')
+    review_container = first_review.find_ancestor(
+        lambda e: e.has_class('review')
+    )
+    
+    # Find similar reviews
+    all_reviews = review_container.find_similar()
+    
+    return [
+        {
+            'text': r.css_first('.review-text::text'),
+            'rating': r.attrib.get('data-rating'),
+            'author': r.css_first('.reviewer::text')
+        }
+        for r in all_reviews
+    ]
+```
+## Filters-based searching
+This search method might be arguably the best way to find elements in Scrapling because it is powerful and easier to learn for newcomers to Web Scraping than learning to write selectors. 
+
+Inspired by BeautifulSoup's `find_all` function, you can find elements using the `find_all` and `find` methods. Both methods can take multiple types of filters and return all elements in the pages that all these filters apply to.
+
+To be more specific:
+
+* Any string passed is considered a tag name.
+* Any iterable passed like List/Tuple/Set is considered an iterable of tag names.
+* Any dictionary is considered a mapping of HTML element(s) attribute names and attribute values.
+* Any regex patterns passed are used to filter elements by content like the `find_by_regex` method
+* Any functions passed are used to filter elements
+* Any keyword argument passed is considered as an HTML element attribute with its value.
+
+It collects all passed arguments and keywords, and each filter passes its results to the following filter in a waterfall-like filtering system.
+
+It filters all elements in the current page/element in the following order:
+
+1. All elements with the passed tag name(s) get collected.
+2. All elements that match all passed attribute(s) are collected; if a previous filter is used, then previously collected elements are filtered.
+3. All elements that match all passed regex patterns are collected, or if previous filter(s) are used, then previously collected elements are filtered.
+4. All elements that fulfill all passed function(s) are collected; if a previous filter(s) is used, then previously collected elements are filtered.
+
+Notes:
+
+1. As you probably understood, the filtering process always starts from the first filter it finds in the filtering order above. So, if no tag name(s) are passed but attributes are passed, the process starts from that layer, and so on.
+2. The order in which you pass the arguments doesn't matter. The only order that's taken into consideration is the order explained above.
+
+Check examples to clear any confusion :)
+
+### Examples
+```python
+>>> from scrapling.fetchers import Fetcher
+>>> page = Fetcher.get('https://quotes.toscrape.com/')
+```
+Find all elements with the tag name `div`.
+```python
+>>> page.find_all('div')
+[<data='<div class="container"> <div class="row...' parent='<body> <div class="container"> <div clas...'>,
+ <data='<div class="row header-box"> <div class=...' parent='<div class="container"> <div class="row...'>,
+...]
+```
+Find all div elements with a class that equals `quote`.
+```python
+>>> page.find_all('div', class_='quote')
+[<data='<div class="quote" itemscope itemtype="h...' parent='<div class="col-md-8"> <div class="quote...'>,
+ <data='<div class="quote" itemscope itemtype="h...' parent='<div class="col-md-8"> <div class="quote...'>,
+...]
+```
+Same as above.
+```python
+>>> page.find_all('div', {'class': 'quote'})
+[<data='<div class="quote" itemscope itemtype="h...' parent='<div class="col-md-8"> <div class="quote...'>,
+ <data='<div class="quote" itemscope itemtype="h...' parent='<div class="col-md-8"> <div class="quote...'>,
+...]
+```
+Find all elements with a class that equals `quote`.
+```python
+>>> page.find_all({'class': 'quote'})
+[<data='<div class="quote" itemscope itemtype="h...' parent='<div class="col-md-8"> <div class="quote...'>,
+ <data='<div class="quote" itemscope itemtype="h...' parent='<div class="col-md-8"> <div class="quote...'>,
+...]
+```
+Find all div elements with a class that equals `quote` and contains the element `.text`, which contains the word 'world' in its content.
+```python
+>>> page.find_all('div', {'class': 'quote'}, lambda e: "world" in e.css_first('.text::text'))
+[<data='<div class="quote" itemscope itemtype="h...' parent='<div class="col-md-8"> <div class="quote...'>]
+```
+Find all elements that don't have children.
+```python
+>>> page.find_all(lambda element: len(element.children) > 0)
+[<data='<html lang="en"><head><meta charset="UTF...'>,
+ <data='<head><meta charset="UTF-8"><title>Quote...' parent='<html lang="en"><head><meta charset="UTF...'>,
+ <data='<body> <div class="container"> <div clas...' parent='<html lang="en"><head><meta charset="UTF...'>,
+...]
+```
+Find all elements that contain the word 'world' in its content.
+```python
+>>> page.find_all(lambda element: "world" in element.text)
+[<data='<span class="text" itemprop="text">“The...' parent='<div class="quote" itemscope itemtype="h...'>,
+ <data='<a class="tag" href="/tag/world/page/1/"...' parent='<div class="tags"> Tags: <meta class="ke...'>]
+```
+Find all span elements that match the given regex
+```python
+>>> page.find_all('span', re.compile(r'world'))
+[<data='<span class="text" itemprop="text">“The...' parent='<div class="quote" itemscope itemtype="h...'>]
+```
+Find all div and span elements with class 'quote' (No span elements like that, so only div returned)
+```python
+>>> page.find_all(['div', 'span'], {'class': 'quote'})
+[<data='<div class="quote" itemscope itemtype="h...' parent='<div class="col-md-8"> <div class="quote...'>,
+ <data='<div class="quote" itemscope itemtype="h...' parent='<div class="col-md-8"> <div class="quote...'>,
+...]
+```
+Mix things up
+```python
+>>> page.find_all({'itemtype':"http://schema.org/CreativeWork"}, 'div').css('.author::text')
+['Albert Einstein',
+ 'J.K. Rowling',
+...]
+```
+A bonus pro tip: Find all elements whose `href` attribute's value ends with the word 'Einstein'.
+```python
+>>> page.find_all({'href$': 'Einstein'})
+[<data='<a href="/author/Albert-Einstein">(about...' parent='<span>by <small class="author" itemprop=...'>,
+ <data='<a href="/author/Albert-Einstein">(about...' parent='<span>by <small class="author" itemprop=...'>,
+ <data='<a href="/author/Albert-Einstein">(about...' parent='<span>by <small class="author" itemprop=...'>]
+```
+Another pro tip: Find all elements that its `href` attribute's value has '/author/' in it
+```python
+>>> page.find_all({'href*': '/author/'})
+[<data='<a href="/author/Albert-Einstein">(about...' parent='<span>by <small class="author" itemprop=...'>,
+ <data='<a href="/author/J-K-Rowling">(about)</a...' parent='<span>by <small class="author" itemprop=...'>,
+ <data='<a href="/author/Albert-Einstein">(about...' parent='<span>by <small class="author" itemprop=...'>,
+...]
+```
+And so on...
+
+## Generating selectors
+You can always generate CSS/XPath selectors for any element that can be reused here or anywhere else, and the most remarkable thing is that it doesn't matter what method you used to find that element!
+
+Generate a short CSS selector for the `url_element` element (if possible, create a short one; otherwise, it's a full selector)
+```python
+>>> url_element = page.find({'href*': '/author/'})
+>>> url_element.generate_css_selector
+'body > div > div:nth-of-type(2) > div > div > span:nth-of-type(2) > a'
+```
+Generate a full CSS selector for the `url_element` element from the start of the page
+```python
+>>> url_element.generate_full_css_selector
+'body > div > div:nth-of-type(2) > div > div > span:nth-of-type(2) > a'
+```
+Generate a short XPath selector for the `url_element` element (if possible, create a short one; otherwise, it's a full selector)
+```python
+>>> url_element.generate_xpath_selector
+'//body/div/div[2]/div/div/span[2]/a'
+```
+Generate a full XPath selector for the `url_element` element from the start of the page
+```python
+>>> url_element.generate_full_xpath_selector
+'//body/div/div[2]/div/div/span[2]/a'
+```
+> Note: <br>
+> When you tell Scrapling to create a short selector, it tries to find a unique element to use in generation as a stop point, like an element with an `id` attribute, but in our case, there wasn't any so that's why the short and the full selector will be the same.
+
+## Using selectors with regular expressions
+Like in `parsel`/`scrapy`, you have the methods `re` and `re_first` for extracting data using regular expressions. However, unlike the former, these methods are in nearly all classes like `Adaptor`/`Adaptors`/`TextHandler` and `TextHandlers`, which means you can use them directly on the element even if you didn't select a text node. 
+
+We will have a deep look at it while explaining the [TextHandler](main_classes.md#texthandler) class, but in general, it works like the below examples:
+```python
+>>> page.css_first('.price_color').re_first(r'[\d\.]+')
+'51.77'
+
+>>> page.css('.price_color').re_first(r'[\d\.]+')
+'51.77'
+
+>>> page.css('.price_color').re(r'[\d\.]+')
+['51.77',
+ '53.74',
+ '50.10',
+ '47.82',
+ '54.23',
+...]
+
+>>> page.css('.product_pod h3 a::attr(href)').re(r'catalogue/(.*)/index.html')
+['a-light-in-the-attic_1000',
+ 'tipping-the-velvet_999',
+ 'soumission_998',
+ 'sharp-objects_997',
+...]
+
+>>> filtering_function = lambda e: e.parent.tag == 'h3' and e.parent.parent.has_class('product_pod')  # As above selector
+>>> page.find('a', filtering_function).attrib['href'].re(r'catalogue/(.*)/index.html')
+['a-light-in-the-attic_1000']
+
+>>> page.find_by_text('Tipping the Velvet').attrib['href'].re(r'catalogue/(.*)/index.html')
+['tipping-the-velvet_999']
+```
+And so on. You get the idea. We will explain this in more detail on the next page while explaining the [TextHandler](main_classes.md#texthandler) class.

docs/stylesheets/extra.css

@@ -0,0 +1,3 @@
+.md-grid {
+  max-width: 65%;
+}

docs/tutorials/migrating_from_beautifulsoup.md

@@ -0,0 +1,98 @@
+# Migrating from BeautifulSoup to Scrapling
+
+<style>
+.md-grid {
+  max-width: 85%;
+}
+</style>
+
+If you're already familiar with BeautifulSoup, you're in for a treat. Scrapling is faster, provides similar parsing capabilities, and adds powerful new features for fetching and handling modern web pages. This guide will help you quickly adapt your existing BeautifulSoup code to take advantage of Scrapling's capabilities.
+
+Below is a table that covers the most common operations you'll perform when scraping web pages. Each row shows how to accomplish a specific task in BeautifulSoup and the corresponding way to do it in Scrapling.
+
+You will notice some shortcuts in BeautifulSoup are missing in Scrapling, but that's one of the reasons that makes BeautifulSoup slower than Scrapling. The point is: If the same feature can be used in a short oneliner, there is no need to sacrifice performance to make that short line shorter :)
+
+
+| Task                                                            | BeautifulSoup Code                                                                                            | Scrapling Code                                                                    |
+|-----------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------|
+| Parser import                                                   | `from bs4 import BeautifulSoup`                                                                               | `from scrapling.parser import Adaptor`                                            |
+| Parsing HTML from string                                        | `soup = BeautifulSoup(html, 'html.parser')`                                                                   | `page = Adaptor(html)`                                                            |
+| Finding a single element                                        | `element = soup.find('div', class_='example')`                                                                | `element = page.find('div', class_='example')`                                    |
+| Finding multiple elements                                       | `elements = soup.find_all('div', class_='example')`                                                           | `elements = page.find_all('div', class_='example')`                               |
+| Finding a single element (Example 2)                            | `element = soup.find('div', attrs={"class": "example"})`                                                      | `element = page.find('div', {"class": "example"})`                                |
+| Finding a single element (Example 3)                            | `element = soup.find(re.compile("^b"))`                                                                       | `element = page.find(re.compile("^b"))`<br/>`element = page.find_by_regex(r"^b")` |
+| Finding a single element (Example 4)                            | `element = soup.find(lambda e: len(list(e.children)) > 0)`                                                    | `element = page.find(lambda e: len(e.children) > 0)`                              |
+| Finding a single element (Example 5)                            | `element = soup.find(["a", "b"])`                                                                             | `element = page.find(["a", "b"])`                                                 |
+| Find element by its text content                                | `element = soup.find(text="some text")`                                                                       | `element = page.find_by_text("some text", partial=False)`                         |
+| Using CSS selectors to find the first matching element          | `elements = soup.select_one('div.example')`                                                                   | `elements = page.css_first('div.example')`                                        |
+| Using CSS selectors to find all matching element                | `elements = soup.select('div.example')`                                                                       | `elements = page.css('div.example')`                                              |
+| Get a prettified version of the page/element source             | `prettified = soup.prettify()`                                                                                | `prettified = page.prettify()`                                                    |
+| Get a Non-pretty version of the page/element source             | `source = str(soup)`                                                                                          | `source = page.body`                                                              |
+| Get tag name of an element                                      | `name = element.name`                                                                                         | `name = element.tag`                                                              |
+| Extracting text content of an element                           | `string = element.string`                                                                                     | `string = element.text`                                                           |
+| Extracting all the text in a document or beneath a tag          | `text = soup.get_text(strip=True)`                                                                            | `text = page.get_all_text(strip=True)`                                            |
+| Access the dictionary of attributes                             | `attrs = element.attrs`                                                                                       | `attrs = element.attrib`                                                          |
+| Extracting attributes                                           | `attr = element['href']`                                                                                      | `attr = element.attrib['href']`                                                   |
+| Navigating to parent                                            | `parent = element.parent`                                                                                     | `parent = element.parent`                                                         |
+| Get all parents of an element                                   | `parents = list(element.parents)`                                                                             | `parents = list(element.iterancestors())`                                         |
+| Searching for an element in the parents of an element           | `target_parent = element.find_parent("a")`                                                                    | `target_parent = element.find_ancestor(lambda p: p.tag == 'a')`                   |
+| Get all siblings of an element                                  | N/A                                                                                                           | `siblings = element.siblings`                                                     |
+| Get next sibling of an element                                  | `next_element = element.next_sibling`                                                                         | `next_element = element.next`                                                     |
+| Searching for an element in the siblings of an element          | `target_sibling = element.find_next_sibling("a")`<br/>`target_sibling = element.find_previous_sibling("a")`   | `target_sibling = element.siblings.search(lambda s: s.tag == 'a')`                |
+| Searching for elements in the siblings of an element            | `target_sibling = element.find_next_siblings("a")`<br/>`target_sibling = element.find_previous_siblings("a")` | `target_sibling = element.siblings.filter(lambda s: s.tag == 'a')`                |
+| Searching for an element in the next elements of an element     | `target_parent = element.find_next("a")`                                                                      | `target_parent = element.below_elements.search(lambda p: p.tag == 'a')`           |
+| Searching for elements in the next elements of an element       | `target_parent = element.find_all_next("a")`                                                                  | `target_parent = element.below_elements.filter(lambda p: p.tag == 'a')`           |
+| Searching for an element in the previous elements of an element | `target_parent = element.find_previous("a")`                                                                  | `target_parent = element.path.search(lambda p: p.tag == 'a')`                     |
+| Searching for elements in the previous elements of an element   | `target_parent = element.find_all_previous("a")`                                                              | `target_parent = element.path.filter(lambda p: p.tag == 'a')`                     |
+| Get previous sibling of an element                              | `prev_element = element.previous_sibling`                                                                     | `prev_element = element.previous`                                                 |
+| Navigating to children                                          | `children = list(element.children)`                                                                           | `children = element.children`                                                     |
+| Get all descendants of an element                               | `children = list(element.descendants)`                                                                        | `children = element.below_elements`                                               |
+| Filtering a group of elements that satisfies a condition        | `group = soup.find('p', 'story').css.filter('a')`                                                             | `group = page.find_all('p', 'story').filter(lambda p: p.tag == 'a')`              |
+
+
+One point to remember: BeautifulSoup provides features for modifying and manipulating the page after parsing it. Scrapling focuses more on Scraping the page faster for you, and then you can do what you want with the extracted information. So, two different tools can be used in Web SScraping, but one of them specializes in Web Scraping :)
+
+### Putting It All Together
+
+Here's a simple example of scraping a web page to extract all the links using BeautifulSoup and Scrapling.
+
+**With BeautifulSoup:**
+
+```python
+import requests
+from bs4 import BeautifulSoup
+
+url = 'http://example.com'
+response = requests.get(url)
+soup = BeautifulSoup(response.text, 'html.parser')
+
+links = soup.find_all('a')
+for link in links:
+    print(link['href'])
+```
+
+**With Scrapling:**
+
+```python
+from scrapling import Fetcher
+
+url = 'http://example.com'
+page = Fetcher.get(url=url)
+
+links = page.css('a::attr(href)')
+for link in links:
+    print(link)
+```
+
+As you can see, Scrapling simplifies the process by handling the fetching and parsing in a single step, making your code cleaner and more efficient.
+
+**Additional Notes:**
+
+- **Different parsers**: BeautifulSoup allows you to set the parser engine to use, and one of them is `lxml`. Scrapling doesn't do that and uses the `lxml` library by default for performance reasons.
+- **Element Types**: In BeautifulSoup, elements are `Tag` objects, while in Scrapling, they are `Adaptor` objects. However, they provide similar methods and properties for navigation and data extraction.
+- **Error Handling**: Both libraries return `None` when an element is not found (e.g., `soup.find()` or `page.css_first()`). To avoid errors, Check for `None` before accessing properties.
+- **Text Extraction**: Scrapling provides additional methods for handling text through `TextHandler`, such as `clean()`, which can be helpful for removing extra whitespace or unwanted characters. Please check out the documentation for the complete list.
+
+The documentation provides more details on Scrapling's features and the full list of arguments that can be passed to all methods.
+
+This guide should make your transition from BeautifulSoup to Scrapling smooth and straightforward. Happy scraping!

docs/tutorials/replacing_ai.md

@@ -0,0 +1 @@
+WIP

mkdocs.yml

@@ -0,0 +1,142 @@
+site_name: Scrapling
+site_description: Scrapling - a Python library to make Web Scraping easy again!
+site_author: Karim Shoair
+repo_url: https://github.com/D4Vinci/Scrapling
+repo_name: D4Vinci/Scrapling
+copyright: Copyright © 2025 Karim Shoair
+
+theme:
+  name: material
+  language: en
+  palette:
+    - media: "(prefers-color-scheme)"
+      toggle:
+        icon: material/link
+        name: Switch to light mode
+    - media: "(prefers-color-scheme: light)"
+      scheme: default
+      primary: indigo
+      accent: indigo
+      toggle:
+        icon: material/toggle-switch
+        name: Switch to dark mode
+    - media: "(prefers-color-scheme: dark)"
+      scheme: slate
+      primary: black
+      accent: indigo
+      toggle:
+        icon: material/toggle-switch-off
+        name: Switch to system preference
+  font:
+    text: Roboto
+    code: Roboto Mono
+  icon:
+    repo: fontawesome/brands/github-alt
+  features:
+    - announce.dismiss
+    - navigation.top
+    - navigation.footer
+    - navigation.instant
+    - navigation.indexes
+    - navigation.sections
+    - navigation.tracking
+    - navigation.instant
+    - navigation.instant.progress
+#    - navigation.tabs
+#    - navigation.expand
+#    - toc.integrate
+    - search.share
+    - search.suggest
+    - search.highlight
+    - content.tabs.link
+    - content.width.full
+    - content.action.view
+    - content.action.edit
+    - content.code.copy
+    - content.code.annotate
+    - content.code.annotation
+#  logo: assets/logo.png
+#  favicon: assets/favicon.png
+
+nav:
+  - Introduction: index.md
+  - Overview: overview.md
+  - Parsing Performance: benchmarks.md
+  - User Guide:
+    - Parsing:
+      - Querying elements: parsing/selection.md
+      - Main classes: parsing/main_classes.md
+      - Using automatch feature: parsing/automatch.md
+    - Fetching:
+      - Choosing a fetcher: fetching/choosing.md
+      - Static requests: fetching/static.md
+      - Dynamically loaded websites: fetching/dynamic.md
+      - Fully bypass protections while fetching: fetching/stealthy.md
+  - Tutorials:
+    - Using Scrapling instead of AI: tutorials/replacing_ai.md
+    - Migrating from BeautifulSoup: tutorials/migrating_from_beautifulsoup.md
+#    - Migrating from AutoScraper: tutorials/migrating_from_autoscraper.md
+  - Development:
+    - API Reference:
+      - Adaptor: api-reference/adaptor.md
+      - Fetchers: api-reference/fetchers.md
+      - Custom Types: api-reference/custom-types.md
+    - Writing your retrieval system: development/automatch_storage_system.md
+    - Using Scrapling's custom types: development/scrapling_custom_types.md
+  - Support and Sponsors: donate.md
+  - Contributing: contributing.md
+  - Changelog: 'https://github.com/D4Vinci/Scrapling/releases'
+
+markdown_extensions:
+  - admonition
+  - abbr
+#  - mkautodoc
+  - pymdownx.emoji
+  - pymdownx.details
+  - pymdownx.superfences
+  - pymdownx.highlight:
+      anchor_linenums: true
+  - pymdownx.inlinehilite
+  - pymdownx.snippets
+  - pymdownx.tabbed:
+      alternate_style: true
+  - tables
+  - codehilite:
+      css_class: highlight
+  - toc:
+      permalink: true
+
+plugins:
+  - search
+  - mkdocstrings:
+      handlers:
+        python:
+          paths: [scrapling]
+          options:
+            docstring_style: sphinx
+            show_source: true
+            show_root_heading: true
+            show_if_no_docstring: true
+            inherited_members: true
+            members_order: source
+            separate_signature: true
+            unwrap_annotated: true
+            filters:
+              - '!^_'
+            merge_init_into_class: true
+            docstring_section_style: spacy
+            signature_crossrefs: true
+            show_symbol_type_heading: true
+            show_symbol_type_toc: true
+
+extra:
+  social:
+    - icon: fontawesome/brands/github
+      link: https://github.com/D4Vinci/Scrapling
+    - icon: fontawesome/brands/python
+      link: https://pypi.org/project/scrapling/
+    - icon: fontawesome/brands/x-twitter
+      link: https://x.com/D4Vinci1
+
+extra_css:
+  - stylesheets/extra.css