Update README.md

README.md

@@ -3,219 +3,74 @@ tags:
 - setfit
 - sentence-transformers
 - text-classification
-- generated_from_setfit_trainer
+- geospatial
+- spatial-queries
 widget:
-- text: what is the closest bus station to pittsburgh international airport
-- text: how to register customary land in png
-- text: dissolved definition
-- text: color Nucleic acid
-- text: how long should i water lawn for
+- text: hotel in geneva airport
+- text: what payroll deduction is mpp
+- text: weather in erlanger ky
+- text: what is the coordinates of point p
+- text: what's the weather in roseburg
 metrics:
 - accuracy
+- f1
 pipeline_tag: text-classification
 library_name: setfit
 inference: true
 base_model: BAAI/bge-small-en-v1.5
+license: mit
 ---
 
-# SetFit with BAAI/bge-small-en-v1.5
+# Geospatial (Web Search) Query Detector
 
-This is a [SetFit](https://github.com/huggingface/setfit) model that can be used for Text Classification. This SetFit model uses [BAAI/bge-small-en-v1.5](https://huggingface.co/BAAI/bge-small-en-v1.5) as the Sentence Transformer embedding model. A [LogisticRegression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html) instance is used for classification.
+A binary [SetFit](https://github.com/huggingface/setfit) classifier that distinguishes geospatial
+from non-geospatial web search queries. Trained on 1,200 gold-labelled
+[MS MARCO](https://microsoft.github.io/msmarco/) web search queries with weak supervision from Llama 3.1, then manually verified.
 
-The model has been trained using an efficient few-shot learning technique that involves:
+Achieves **F1 = 0.931** on a held-out test set of 800 samples (421 non-spatial, 379 spatial),
+with the evaluation model trained on 200 samples (105 non-spatial, 95 spatial). The deployed model was trained on the full 1,200.
 
-1. Fine-tuning a [Sentence Transformer](https://www.sbert.net) with contrastive learning.
-2. Training a classification head with features from the fine-tuned Sentence Transformer.
 
-## Model Details
+## What counts as a geospatial query?
 
-### Model Description
-- **Model Type:** SetFit
-- **Sentence Transformer body:** [BAAI/bge-small-en-v1.5](https://huggingface.co/BAAI/bge-small-en-v1.5)
-- **Classification head:** a [LogisticRegression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html) instance
-- **Maximum Sequence Length:** 512 tokens
-- **Number of Classes:** 2 classes
-<!-- - **Training Dataset:** [Unknown](https://huggingface.co/datasets/unknown) -->
-<!-- - **Language:** Unknown -->
-<!-- - **License:** Unknown -->
-
-### Model Sources
-
-- **Repository:** [SetFit on GitHub](https://github.com/huggingface/setfit)
-- **Paper:** [Efficient Few-Shot Learning Without Prompts](https://arxiv.org/abs/2209.11055)
-- **Blogpost:** [SetFit: Efficient Few-Shot Learning Without Prompts](https://huggingface.co/blog/setfit)
-
-### Model Labels
-| Label | Examples                                                                                                                                                   |
-|:------|:-----------------------------------------------------------------------------------------------------------------------------------------------------------|
-| 1     | <ul><li>'what city is hurricane harbor located in texas'</li><li>'where is the hills located in lynchburg va'</li><li>'weather hurricane mexico'</li></ul> |
-| 0     | <ul><li>'how early can you register to vote'</li><li>'us postal services track'</li><li>'how much does it cost to run a iron'</li></ul>                    |
+As per [Mai et al. (2021)](https://agile-giss.copernicus.org/articles/2/8/2021/) and
+[Kefalidis et al. (2024)](https://www.sciencedirect.com/science/article/pii/S1569843224005594),
+a query is geospatial if it requires qualitative or quantitative geographic
+knowledge of Earth-bound features to be answered.
 
-## Uses
+This is usually the case if the query involves:
+- A geographic entity (named place on Earth: city, country, river, POI, address)
+- A geographic concept (place type: city, lake, mountain, park, building)
+- A spatial relation (near, within, north of, between, borders, crosses, distance)
 
-### Direct Use for Inference
+Non-geospatial: anatomical, microscopic, astronomical, fictional, or abstract
+'where' questions; queries needing no geographic knowledge.
 
-First install the SetFit library:
+## Model details
 
-```bash
-pip install setfit
-```
+- **Sentence Transformer body:** [BAAI/bge-small-en-v1.5](https://huggingface.co/BAAI/bge-small-en-v1.5)
+- **Classification head:** LogisticRegression
+- **Training data:** 1,200 gold-labelled MS MARCO queries (632 non-spatial, 568 spatial), sampled via K-means centroids across the full embedding space of all 1M+ queries for representativeness
+- **Labels:** `1` = geospatial, `0` = non-geospatial
 
-Then you can load this model and run inference.
+## Usage
 
 ```python
 from setfit import SetFitModel
 
-# Download from the 🤗 Hub
-model = SetFitModel.from_pretrained("setfit_model_id")
-# Run inference
-preds = model("color Nucleic acid")
+model = SetFitModel.from_pretrained("ilyankou/is-geospatial-query")
+preds = model([
+  "nearest hospital",
+  "far from the truth",
+  "close to my heart",
+  "flood risk in this area"
+])
+# => [1, 0, 0, 1]
 ```
 
-<!--
-### Downstream Use
-
-*List how someone could finetune this model on their own dataset.*
--->
-
-<!--
-### Out-of-Scope Use
-
-*List how the model may foreseeably be misused and address what users ought not to do with the model.*
--->
-
-<!--
-## Bias, Risks and Limitations
-
-*What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.*
--->
-
-<!--
-### Recommendations
-
-*What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.*
--->
-
-## Training Details
-
-### Training Set Metrics
-| Training set | Min | Median | Max |
-|:-------------|:----|:-------|:----|
-| Word count   | 2   | 6.3658 | 17  |
-
-| Label | Training Sample Count |
-|:------|:----------------------|
-| 0     | 632                   |
-| 1     | 568                   |
-
-### Training Hyperparameters
-- batch_size: (64, 64)
-- num_epochs: (3, 3)
-- max_steps: -1
-- sampling_strategy: oversampling
-- num_iterations: 20
-- body_learning_rate: (2e-05, 2e-05)
-- head_learning_rate: 0.01
-- loss: CosineSimilarityLoss
-- distance_metric: cosine_distance
-- margin: 0.25
-- end_to_end: False
-- use_amp: False
-- warmup_proportion: 0.1
-- l2_weight: 0.01
-- seed: 42
-- eval_max_steps: -1
-- load_best_model_at_end: False
-
-### Training Results
-| Epoch  | Step | Training Loss | Validation Loss |
-|:------:|:----:|:-------------:|:---------------:|
-| 0.0013 | 1    | 0.2374        | -               |
-| 0.0667 | 50   | 0.2383        | -               |
-| 0.1333 | 100  | 0.2151        | -               |
-| 0.2    | 150  | 0.0982        | -               |
-| 0.2667 | 200  | 0.034         | -               |
-| 0.3333 | 250  | 0.0104        | -               |
-| 0.4    | 300  | 0.0056        | -               |
-| 0.4667 | 350  | 0.0035        | -               |
-| 0.5333 | 400  | 0.002         | -               |
-| 0.6    | 450  | 0.0013        | -               |
-| 0.6667 | 500  | 0.0014        | -               |
-| 0.7333 | 550  | 0.0013        | -               |
-| 0.8    | 600  | 0.0009        | -               |
-| 0.8667 | 650  | 0.0011        | -               |
-| 0.9333 | 700  | 0.0008        | -               |
-| 1.0    | 750  | 0.0009        | -               |
-| 1.0667 | 800  | 0.0008        | -               |
-| 1.1333 | 850  | 0.0008        | -               |
-| 1.2    | 900  | 0.0007        | -               |
-| 1.2667 | 950  | 0.0006        | -               |
-| 1.3333 | 1000 | 0.0009        | -               |
-| 1.4    | 1050 | 0.0006        | -               |
-| 1.4667 | 1100 | 0.0005        | -               |
-| 1.5333 | 1150 | 0.0005        | -               |
-| 1.6    | 1200 | 0.0005        | -               |
-| 1.6667 | 1250 | 0.0005        | -               |
-| 1.7333 | 1300 | 0.0005        | -               |
-| 1.8    | 1350 | 0.0005        | -               |
-| 1.8667 | 1400 | 0.0005        | -               |
-| 1.9333 | 1450 | 0.0005        | -               |
-| 2.0    | 1500 | 0.0005        | -               |
-| 2.0667 | 1550 | 0.0005        | -               |
-| 2.1333 | 1600 | 0.0005        | -               |
-| 2.2    | 1650 | 0.0004        | -               |
-| 2.2667 | 1700 | 0.0004        | -               |
-| 2.3333 | 1750 | 0.0004        | -               |
-| 2.4    | 1800 | 0.0004        | -               |
-| 2.4667 | 1850 | 0.0004        | -               |
-| 2.5333 | 1900 | 0.0004        | -               |
-| 2.6    | 1950 | 0.0004        | -               |
-| 2.6667 | 2000 | 0.0004        | -               |
-| 2.7333 | 2050 | 0.0004        | -               |
-| 2.8    | 2100 | 0.0004        | -               |
-| 2.8667 | 2150 | 0.0004        | -               |
-| 2.9333 | 2200 | 0.0004        | -               |
-| 3.0    | 2250 | 0.0004        | -               |
-
-### Framework Versions
-- Python: 3.11.14
-- SetFit: 1.1.3
-- Sentence Transformers: 4.0.2
-- Transformers: 4.55.2
-- PyTorch: 2.8.0
-- Datasets: 2.15.0
-- Tokenizers: 0.21.1
-
-## Citation
-
-### BibTeX
-```bibtex
-@article{https://doi.org/10.48550/arxiv.2209.11055,
-    doi = {10.48550/ARXIV.2209.11055},
-    url = {https://arxiv.org/abs/2209.11055},
-    author = {Tunstall, Lewis and Reimers, Nils and Jo, Unso Eun Seo and Bates, Luke and Korat, Daniel and Wasserblat, Moshe and Pereg, Oren},
-    keywords = {Computation and Language (cs.CL), FOS: Computer and information sciences, FOS: Computer and information sciences},
-    title = {Efficient Few-Shot Learning Without Prompts},
-    publisher = {arXiv},
-    year = {2022},
-    copyright = {Creative Commons Attribution 4.0 International}
-}
-```
-
-<!--
-## Glossary
-
-*Clearly define terms in order to be accessible across audiences.*
--->
-
-<!--
-## Model Card Authors
-
-*Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.*
--->
-
-<!--
-## Model Card Contact
+## Training
 
-*Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.*
--->
+Weak labels were generated by running Llama 3.1 five times per query at temperature 0.3,
+then manually verified. The SetFit model was trained for 3 epochs with batch size 64
+and learning rate 2e-5 on 200 samples (95 positive and 105 negative) for validation,
+then retrained on the full gold dataset (1,200 samples) for production inference.