| --- |
| tags: |
| - setfit |
| - sentence-transformers |
| - text-classification |
| - geospatial |
| - spatial-queries |
| widget: |
| - text: hotel in geneva airport |
| - text: what payroll deduction is mpp |
| - text: weather in erlanger ky |
| - text: what is the coordinates of point p |
| - text: what's the weather in roseburg |
| metrics: |
| - accuracy |
| - f1 |
| pipeline_tag: text-classification |
| library_name: setfit |
| inference: true |
| base_model: BAAI/bge-small-en-v1.5 |
| license: mit |
| --- |
| |
| # Geospatial (Web Search) Query Detector |
|
|
| A binary [SetFit](https://github.com/huggingface/setfit) classifier that distinguishes geospatial |
| from non-geospatial web search queries. Trained on 1,200 gold-labelled |
| [MS MARCO](https://microsoft.github.io/msmarco/) web search queries with weak supervision from Llama 3.1, then manually verified. See COSIT 2026 paper preprint here - https://arxiv.org/abs/2605.11336 |
|
|
| Achieves **F1 = 0.931** on a held-out test set of 800 samples (421 non-spatial, 379 spatial), |
| with the evaluation model trained on 200 samples (105 non-spatial, 95 spatial). The deployed model was trained on the full 1,200. |
|
|
|
|
| ## What counts as a geospatial query? |
|
|
| As per [Mai et al. (2021)](https://agile-giss.copernicus.org/articles/2/8/2021/) and |
| [Kefalidis et al. (2024)](https://www.sciencedirect.com/science/article/pii/S1569843224005594), |
| a query is geospatial if it requires qualitative or quantitative geographic |
| knowledge of Earth-bound features to be answered. |
|
|
| This is usually the case if the query involves: |
| - A geographic entity (named place on Earth: city, country, river, POI, address) |
| - A geographic concept (place type: city, lake, mountain, park, building) |
| - A spatial relation (near, within, north of, between, borders, crosses, distance) |
|
|
| Non-geospatial: anatomical, microscopic, astronomical, fictional, or abstract |
| 'where' questions; queries needing no geographic knowledge. |
|
|
| ## Model details |
|
|
| - **Sentence Transformer body:** [BAAI/bge-small-en-v1.5](https://huggingface.co/BAAI/bge-small-en-v1.5) |
| - **Classification head:** LogisticRegression |
| - **Training data:** 1,200 gold-labelled MS MARCO queries (632 non-spatial, 568 spatial), sampled via K-means centroids across the full embedding space of all 1M+ queries for representativeness |
| - **Labels:** `1` = geospatial, `0` = non-geospatial |
|
|
| ## Usage |
|
|
| ```python |
| from setfit import SetFitModel |
| |
| model = SetFitModel.from_pretrained("ilyankou/is-geospatial-query") |
| preds = model([ |
| "nearest hospital", |
| "far from the truth", |
| "close to my heart", |
| "flood risk in this area" |
| ]) |
| # => [1, 0, 0, 1] |
| ``` |
|
|
| ## Training |
|
|
| Weak labels were generated by running Llama 3.1 five times per query at temperature 0.3, |
| then manually verified. The SetFit model was trained for 3 epochs with batch size 64 |
| and learning rate 2e-5 on 200 samples (95 positive and 105 negative) for validation, |
| then retrained on the full gold dataset (1,200 samples) for production inference. |
