Upload 5 files

README.md

@@ -1,3 +1,130 @@
----
-license: apache-2.0
----
+---
+license: apache-2.0
+tags:
+- video-moment-localization
+- point-supervised
+- vision-language
+- multimodal
+- pytorch
+---
+
+<a id="top"></a>
+<div align="center">
+  <h1>🚀 SG-SCI: Explicit Granularity and Implicit Scale Correspondence Learning for Point-Supervised Video Moment Localization</h1>
+
+  <p>
+    <b>Kun Wang</b><sup>1</sup>&nbsp;
+    <b>Hao Liu</b><sup>1</sup>&nbsp;
+    <b>Lirong Jie</b><sup>1</sup>&nbsp;
+    <b>Zixu Li</b><sup>1</sup>&nbsp;
+    <b>Yupeng Hu</b><sup>1</sup>&nbsp;
+    <b>Liqiang Nie</b><sup>2</sup>
+  </p>
+
+  <p>
+    <sup>1</sup>School of Software, Shandong University, Jinan, China<br>
+    <sup>2</sup>School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen, China
+  </p>
+</div>
+
+These are the official implementation, pre-trained model weights, and configuration files for **SG-SCI**, a novel framework explicitly designed to address explicit granularity alignment and implicit scale perception in Video Moment Localization (VML).
+
+🔗 **Paper:** [Accepted by ACM MM 2024](https://dl.acm.org/doi/10.1145/3664647.3680774)
+🔗 **GitHub Repository:** [iLearn-Lab/SG-SCI](https://github.com/iLearn-Lab/SG-SCI)
+
+---
+
+## 📌 Model Information
+
+### 1. Model Name
+**SG-SCI** (Semantic Granularity and Scale Correspondence Integration)
+
+### 2. Task Type & Applicable Tasks
+- **Task Type:** Point-supervised Video Moment Localization (VML) / Vision-Language / Multimodal Learning
+- **Applicable Tasks:** Localizing specific moments in untrimmed videos based on textual queries, utilizing only single-frame (point-level) annotations during training to reduce annotation costs.
+
+### 3. Project Introduction
+Existing point-supervised Video Moment Localization (VML) methods often struggle with explicit granularity alignment and implicit scale perception. **SG-SCI** introduces a Semantic Granularity and Scale Correspondence Integration framework to model the semantic alignment between video and text. This approach helps the model effectively enhance and utilize modal feature representations of varying granularities and scales.
+
+> 💡 **Method Highlight:** SG-SCI explicitly models semantic relations of different feature granularities (via the GCA module) and adaptively mines implicit semantic scales (via the SCL strategy). It fully supports end-to-end model training and multi-modal interaction using single-frame annotations.
+
+### 4. Training Data Source
+The model is evaluated on standard VML datasets using point-level supervision:
+- **Charades-STA**
+- **TACoS**
+*(Splits produced by [ViGA](https://github.com/r-cui/ViGA/tree/master))*
+
+---
+
+## 🚀 Usage & Basic Inference
+
+### Step 1: Prepare the Environment
+Clone the GitHub repository and set up the Conda environment (evaluated on Python 3.7 and PyTorch 1.10.0):
+```bash
+git clone https://github.com/iLearn-Lab/SG-SCI.git
+cd SG-SCI
+```
+```bash
+conda create --name sg-sci python=3.7 -y
+source activate sg-sci
+```
+
+```bash
+conda install pytorch=1.10.0 cudatoolkit=11.3.1 -y
+pip install numpy scipy pyyaml tqdm
+```
+
+
+
+### Step 2: Download Model Weights & Data
+1. **Pre-trained Checkpoints:** Download the model checkpoints and place them in your designated `LOGGER_PATH`.
+2. **Datasets:** Ensure the Charades-STA and TACoS datasets are properly structured in the `src/dataset/` directory according to the splits provided by ViGA.
+
+### Step 3: Run Training & Evaluation
+
+**Training from Scratch:**
+Depending on the dataset you want to train on, run the following commands:
+
+#### For TACoS
+python -m experiment.train --task tacos
+
+#### For Charades-STA
+python -m experiment.train --task charadessta
+
+
+**Evaluation (Eval):**
+Put the downloaded checkpoints in your `LOGGER_PATH`, then run:
+
+python -m src.experiment.eval --exp $LOGGER_PATH
+
+
+---
+
+
+## ⚠️ Limitations & Notes
+
+**Disclaimer:** This framework and its pre-trained weights are intended for **academic research purposes only**.
+- The model requires access to the original source datasets (Charades-STA, TACoS) for full evaluation. 
+- As a point-supervised method, while it significantly reduces annotation costs compared to fully-supervised methods, localization boundary precision may still be inherently limited by the single-frame nature of the ground-truth signals.
+
+---
+
+## 🤝 Acknowledgements & Contact
+
+- **Acknowledgement:** Thanks to the [ViGA](https://github.com/r-cui/ViGA/tree/master) open-source community for strong baselines and tooling. Thanks to all collaborators and contributors of this project.
+- **Contact:** If you have any questions, feel free to contact me at `khylon.kun.wang@gmail.com`.
+
+---
+
+## 📝⭐️ Citation
+
+If you find our work or this repository useful in your research, please consider citing our paper:
+
+
+@inproceedings{wang2024explicit,
+  title={Explicit granularity and implicit scale correspondence learning for point-supervised video moment localization},
+  author={Wang, Kun and Liu, Hao and Jie, Lirong and Li, Zixu and Hu, Yupeng and Nie, Liqiang},
+  booktitle={Proceedings of the 32nd ACM International Conference on Multimedia},
+  pages={9214--9223},
+  year={2024}
+}

charades/config.yaml

@@ -0,0 +1,45 @@
+charadessta:
+  batch_size: 512
+  clip_frames:
+  - 8
+  epoch: 30
+  feature_dim: 1024
+  feature_dir: data/charadessta/i3d
+  inter_loss: 0.1
+  intra_loss: 0.1
+  moment_length_factors:
+  - 0.25
+  - 0.3
+  - 0.35
+  momentum: 0.3
+  overlapping_factors:
+  - 0.0
+  - 0.1
+  - 0.2
+  - 0.3
+  - 0.4
+  - 0.5
+  - 0.6
+  - 0.7
+  - 0.8
+  - 0.9
+  pooling_func: max_pooling
+  sigma_factor: 0.3
+  stride: 4
+  video_feature_len: 128
+dataset_name: charadessta
+exp_dir: log
+gpu: '1'
+model:
+  dim: 512
+  dropout: 0.1
+  glove_path: data/glove.840B.300d.txt
+  momentum_alpha: 0.7
+  n_layers: 2
+  temp: 0.1
+  topk: 1
+seed: 0
+train:
+  clip_norm: 1.0
+  dev: false
+  init_lr: 0.0001

charades/model_best.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1a763be0003fed60cf8311959e1080076acdc789aefa82d52de4ad17d1f30ac1
+size 98963819

tacos/config.yaml

@@ -0,0 +1,50 @@
+tacos:
+  batch_size: 64
+  clip_frames:
+  - 32
+  epoch: 80
+  feature_dim: 4096
+  feature_dir: data/tacos/c3d
+  inter_loss: 0.1
+  intra_loss: 0.1
+  moment_length_factors:
+  - 0.05
+  - 0.1
+  - 0.15
+  - 0.2
+  - 0.25
+  - 0.3
+  - 0.35
+  - 0.4
+  momentum: 0.3
+  overlapping_factors:
+  - 0.0
+  - 0.1
+  - 0.2
+  - 0.3
+  - 0.4
+  - 0.5
+  - 0.6
+  - 0.7
+  - 0.8
+  - 0.9
+  pooling_func: mean_pooling
+  sigma_factor: 1.0
+  stride: 16
+  video_feature_len: 512
+train:
+  clip_norm: 1.0
+  dev: false
+  init_lr: 0.0001
+dataset_name: tacos
+exp_dir: log
+gpu: '0'
+model:
+  dim: 512
+  dropout: 0.1
+  glove_path: data/glove.840B.300d.txt
+  momentum_alpha: 0.7
+  n_layers: 2
+  temp: 0.1
+  topk: 1
+seed: 0

tacos/model_best.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:91aa6f88d84be4641fa24d4335c347c10738e56ac5fdc6a777ae5764fe821695
+size 106066283