Update README.md
Browse files
README.md
CHANGED
|
@@ -7,4 +7,100 @@ tags:
|
|
| 7 |
- earth-observation
|
| 8 |
- remote-sensing
|
| 9 |
- foundation-model
|
| 10 |
-
---
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 7 |
- earth-observation
|
| 8 |
- remote-sensing
|
| 9 |
- foundation-model
|
| 10 |
+
---
|
| 11 |
+
|
| 12 |
+
# GeoSANE: Learning Geospatial Representations From Models, Not Data
|
| 13 |
+
|
| 14 |
+
GeoSANE is a geospatial model foundry that operates directly in model-weight space. Instead of training a downstream model from scratch, GeoSANE learns a shared latent representation over a population of pretrained remote sensing models and uses that representation to generate new model candidates for a target architecture. These generated models can then be evaluated and fine-tuned on downstream tasks.
|
| 15 |
+
|
| 16 |
+
This repository provides a demo project for running GeoSANE on a downstream remote sensing benchmark. The included notebook walks through the full evaluation pipeline for TIMM backbones on the Sen1Floods11 segmentation task: preparing the downstream dataset, loading a trained GeoSANE checkpoint, generating model candidates, fine-tuning and saving the resulting checkpoint.
|
| 17 |
+
|
| 18 |
+
## Project Contents
|
| 19 |
+
|
| 20 |
+
- `geosane-demo.ipynb`: end-to-end demo notebook
|
| 21 |
+
- `shrp/`: the core SHRP library used by GeoSANE for weight tokenization, latent sampling, model reconstruction, evaluation, and fine-tuning
|
| 22 |
+
- `downstream_datasets/`: downstream benchmark loaders, including Sen1Floods11, SpaceNet, EuroSAT, DIOR, fMoW, and others
|
| 23 |
+
- `requirements.txt`: broad dependency list for setting up an environment
|
| 24 |
+
- `requirements-lock.txt`: pinned versions from a working environment
|
| 25 |
+
- `anchor_tokenized/`: cached tokenized anchor-model datasets generated during evaluation
|
| 26 |
+
- `checkpoints/`: fine-tuned model checkpoints written during notebook runs
|
| 27 |
+
|
| 28 |
+
## Quick Start
|
| 29 |
+
|
| 30 |
+
Create an environment and install dependencies:
|
| 31 |
+
|
| 32 |
+
```bash
|
| 33 |
+
python -m venv .venv
|
| 34 |
+
source .venv/bin/activate
|
| 35 |
+
pip install --upgrade pip
|
| 36 |
+
pip install -r requirements.txt
|
| 37 |
+
```
|
| 38 |
+
|
| 39 |
+
If you want to reproduce the exact environment used for this demo as closely as possible, use:
|
| 40 |
+
|
| 41 |
+
```bash
|
| 42 |
+
pip install -r requirements-lock.txt
|
| 43 |
+
```
|
| 44 |
+
|
| 45 |
+
Note: the lock file includes environment-specific PyTorch builds. You may need to install a compatible `torch` / `torchvision` pair first and then install the remaining dependencies.
|
| 46 |
+
|
| 47 |
+
## Downstream Task
|
| 48 |
+
|
| 49 |
+
[A Jupyter notebook](https://github.com/HSG-AIML/GeoSANE/blob/main/geosane-demo.ipynb) is provided in [GeoSANE GitHub page](https://github.com/HSG-AIML/GeoSANE) to show in detail how to use pretrained model weights. The details of GeoSANE are described in our paper, available on [arXiv](https://arxiv.org/abs/2603.23408).
|
| 50 |
+
The demo notebook is currently configured for:
|
| 51 |
+
|
| 52 |
+
- task: segmentation
|
| 53 |
+
- downstream dataset: Sen1Floods11
|
| 54 |
+
- generated backbone prompt: TIMM backbones such as `swin_s3_base_224.ms_in1k`
|
| 55 |
+
|
| 56 |
+
The downstream dataset file created by the notebook is:
|
| 57 |
+
|
| 58 |
+
- `data/sen1flood11_preprocessed.pt`
|
| 59 |
+
|
| 60 |
+
This file is constructed from:
|
| 61 |
+
|
| 62 |
+
```python
|
| 63 |
+
trainset = Sen1Floods11HandLabeledDataset(split="train", resize_to=(224, 224))
|
| 64 |
+
testset = Sen1Floods11HandLabeledDataset(split="val", resize_to=(224, 224))
|
| 65 |
+
```
|
| 66 |
+
|
| 67 |
+
|
| 68 |
+
## GeoSANE Checkpoints
|
| 69 |
+
|
| 70 |
+
The notebook expects a trained GeoSANE run directory. The configured `model_path` must point to a directory that contains at least:
|
| 71 |
+
|
| 72 |
+
- `params.json`
|
| 73 |
+
- `checkpoint_000300/state.pt`
|
| 74 |
+
|
| 75 |
+
In the notebook, this is configured through:
|
| 76 |
+
|
| 77 |
+
```python
|
| 78 |
+
EXPERIMENT = TimmExperimentConfig(
|
| 79 |
+
model_path=Path("/path/to/your/geosane_run"),
|
| 80 |
+
checkpoint_rel_path=Path("checkpoint_000300/state.pt"),
|
| 81 |
+
...
|
| 82 |
+
)
|
| 83 |
+
```
|
| 84 |
+
|
| 85 |
+
After downloading or copying the checkpoint directory, update `model_path` in `geosane-demo.ipynb` to your local path.
|
| 86 |
+
|
| 87 |
+
## Outputs
|
| 88 |
+
|
| 89 |
+
Running the notebook produces several artifacts:
|
| 90 |
+
|
| 91 |
+
- `data/sen1flood11_preprocessed.pt`: preprocessed downstream dataset
|
| 92 |
+
- `anchor_tokenized/`: tokenized anchor-model datasets used during sampling and evaluation
|
| 93 |
+
- `checkpoints/`: fine-tuned model checkpoints saved during evaluation
|
| 94 |
+
- `model_path/notebook_eval_results/`: JSON evaluation outputs
|
| 95 |
+
|
| 96 |
+
|
| 97 |
+
## Acknowledging this work
|
| 98 |
+
|
| 99 |
+
If you would like to cite our work, please use the following reference:
|
| 100 |
+
|
| 101 |
+
* Hanna, Joelle, Damian Falk, Stella X. Yu and Damian Borth. *GeoSANE: Learning Geospatial Representations from Models, Not Data.*, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 2026.
|
| 102 |
+
|
| 103 |
+
|
| 104 |
+
## Code
|
| 105 |
+
This repository incorporates code from the following source:
|
| 106 |
+
* [SANE](https://github.com/HSG-AIML/SANE)
|