first commit

README.md

@@ -1,3 +1,37 @@
 ---
 license: mit
 ---
+
+## Model Summary
+
+Video-CCAM-9B is a Video-MLLM built on [Yi-1.5-9B-Chat](https://huggingface.co/01-ai/Yi-1.5-9B-Chat) and [SigLIP SO400M](https://huggingface.co/google/siglip-so400m-patch14-384).
+
+## Usage
+
+Inference using Huggingface transformers on NVIDIA GPUs. Requirements tested on python 3.10：
+```
+torch==2.1.0
+torchvision==0.16.0
+transformers==4.40.2
+peft==0.10.0
+```
+
+## Inference
+
+Please refer to [Video-CCAM](https://github.com/QQ-MM/Video-CCAM) on inference and evaluation.
+
+### Video-MME
+
+|#Frames.|32|96|
+|:-:|:-:|:-:|
+|w/o subs|50.0|50.6|
+|w subs|53.1|54.9|
+
+## Acknowledgement
+
+* [xtuner](https://github.com/InternLM/xtuner): Video-CCAM-4B is trained using the xtuner framework. Thanks for their excellent works!
+* [Yi-1.5-9B-Chat](https://huggingface.co/01-ai/Yi-1.5-9B-Chat): Great language models developed by [01.AI](https://www.lingyiwanwu.com/).
+* [SigLIP SO400M](https://huggingface.co/google/siglip-so400m-patch14-384): Outstanding vision encoder developed by Google.
+
+## License
+The model is licensed under the MIT license.

llm_adapter/README.md

@@ -0,0 +1,202 @@
+---
+library_name: peft
+base_model: /group/40006/jaronfei/models/Yi-1.5-9B-Chat
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.10.0

llm_adapter/adapter_config.json

@@ -0,0 +1,34 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "/group/40006/jaronfei/models/Yi-1.5-9B-Chat",
+  "bias": "none",
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 256,
+  "lora_dropout": 0.05,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 512,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "k_proj",
+    "gate_proj",
+    "down_proj",
+    "o_proj",
+    "up_proj",
+    "v_proj",
+    "q_proj"
+  ],
+  "task_type": "CAUSAL_LM",
+  "use_dora": false,
+  "use_rslora": false
+}

llm_adapter/adapter_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:18a9d65175fb8c08c79c0a36c4789737318dc11b511dc643ee1e89083ff4e0c8
+size 3485701322

projector/config.json

@@ -0,0 +1,38 @@
+{
+  "architectures": [
+    "CCAMModel"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "auto_map": {
+    "AutoConfig": "configuration_ccam_projector.CCAMConfig",
+    "AutoModel": "modeling_ccam_projector.CCAMModel"
+  },
+  "cross_attention_frequency": 1,
+  "encoder_hidden_size": 1152,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 1152,
+  "initializer_range": 0.02,
+  "intermediate_size": 4096,
+  "layer_norm_eps": 1e-12,
+  "max_position_embeddings": 512,
+  "model_type": "ccam_projector",
+  "num_attention_heads": 18,
+  "num_hidden_layers": 1,
+  "num_query_tokens": 1024,
+  "output_size": 4096,
+  "pad_token_id": 0,
+  "position_embedding_type": "absolute",
+  "query_attn_mask_type": "full",
+  "spatial_pos_embed_type": "none",
+  "spatial_resolution": [
+    1,
+    1
+  ],
+  "temporal_pos_embed_type": "none",
+  "temporal_resolution": 16,
+  "torch_dtype": "float32",
+  "transformers_version": "4.40.2",
+  "visual_attn_mask_type": "ccam",
+  "vocab_size": 30522
+}

projector/configuration_ccam_projector.py

@@ -0,0 +1,42 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+"""
+================================================
+@author: Jaron
+@time: 2024/02/20 16:37:16
+@email: fjjth98@163.com
+@description: different projector in Video-LLM
+================================================
+"""
+
+from transformers.models.blip_2.configuration_blip_2 import Blip2QFormerConfig
+
+
+class CCAMConfig(Blip2QFormerConfig):
+    model_type = 'ccam_projector'
+    _auto_class = 'AutoConfig'
+
+    def __init__(
+        self, 
+        spatial_pos_embed_type: str = 'learnable',      # ['none', 'learnable', 'cosine']
+        spatial_resolution: tuple[int, int] = (1, 1),   # (H, W)
+        temporal_pos_embed_type: str = 'learnable',
+        temporal_resolution: int = 0,                   # T
+        num_query_tokens: int = 512,
+        visual_attn_mask_type: str = 'ccam',        # ['ccam', 'full']
+        query_attn_mask_type: str = 'full',       # ['causal', 'full']
+        num_hidden_layers=1,
+        cross_attention_frequency=1,
+        output_size=4096,   # llm dimension
+        encoder_hidden_size=1024, # visual dimension
+        hidden_size=1024,
+        vocab_size=30522, num_attention_heads=16, intermediate_size=4096, hidden_act="gelu", hidden_dropout_prob=0.1, attention_probs_dropout_prob=0.1, max_position_embeddings=512, initializer_range=0.02, layer_norm_eps=1e-12, pad_token_id=0, position_embedding_type="absolute", **kwargs):
+        super().__init__(vocab_size, hidden_size, num_hidden_layers, num_attention_heads, intermediate_size, hidden_act, hidden_dropout_prob, attention_probs_dropout_prob, max_position_embeddings, initializer_range, layer_norm_eps, pad_token_id, position_embedding_type, cross_attention_frequency, encoder_hidden_size, **kwargs)
+        self.spatial_pos_embed_type = spatial_pos_embed_type
+        self.spatial_resolution = spatial_resolution
+        self.temporal_pos_embed_type = temporal_pos_embed_type
+        self.temporal_resolution = temporal_resolution
+        self.num_query_tokens = num_query_tokens
+        self.visual_attn_mask_type = visual_attn_mask_type
+        self.query_attn_mask_type = query_attn_mask_type
+        self.output_size = output_size

projector/modeling_ccam_projector.py

@@ -0,0 +1,203 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+"""
+================================================
+@author: Jaron
+@time: 2024/02/20 16:21:56
+@email: fjjth98@163.com
+@description: QFormer projector, convert image and video into fixed-length tokens
+================================================
+"""
+
+import math
+import torch
+import torch.nn as nn
+from torch.nn.functional import interpolate
+from transformers.models.blip_2.modeling_blip_2 import Blip2QFormerModel, Blip2QFormerEncoder
+
+from .configuration_ccam_projector import CCAMConfig
+
+
+class SimpleQFormerOutput(nn.Module):
+    # replace last residual MLP with normal MLP
+    def __init__(self, config):
+        super().__init__()
+        self.dense = nn.Linear(config.intermediate_size, config.output_size)
+
+    def forward(self, hidden_states: torch.Tensor, input_tensor: torch.Tensor = None) -> torch.Tensor:
+        return self.dense(hidden_states)
+
+
+class SimpleQFormerIdentity(nn.Module):
+    # just to replace the first attention module with identity, since it is useless
+ 
+    def forward(self, hidden_states: torch.Tensor, *args, **kwargs) -> torch.Tensor:
+        return hidden_states,
+
+
+class CCAMModel(Blip2QFormerModel):
+    _auto_class = 'AutoModel'
+    config_class = CCAMConfig
+    base_model_prefix = 'model'
+    supports_gradient_checkpointing = True
+ 
+    def __init__(self, config: CCAMConfig):
+        super(Blip2QFormerModel, self).__init__(config)
+        self.gradient_checkpointing = False
+        self.config = config
+        self.num_query_tokens = config.num_query_tokens
+        self.visual_attn_mask_type = config.visual_attn_mask_type
+
+        self.layernorm = nn.LayerNorm(config.hidden_size, eps=config.layer_norm_eps)
+        self.dropout = nn.Dropout(config.hidden_dropout_prob)
+        self.encoder = Blip2QFormerEncoder(config)
+        self.encoder.layer[0].attention = SimpleQFormerIdentity()   # replace the 1st attention module with identity
+        self.encoder.layer[-1].output_query = SimpleQFormerOutput(config)
+
+        # initialize query tokens
+        self.query_tokens = nn.Parameter(torch.zeros(1, config.num_query_tokens, config.hidden_size))
+
+        # initialize pos embed
+        self.spatial_pos_embed = self._create_pos_embed(*config.spatial_resolution, type=config.spatial_pos_embed_type)     # (H, W, C)
+        self.temporal_pos_embed = self._create_pos_embed(config.temporal_resolution, type=config.temporal_pos_embed_type)   # (T, C)
+
+        # initialize query attn mask
+        if config.query_attn_mask_type == 'full':
+            self.query_attn_mask = None
+        elif config.query_attn_mask_type == 'causal':
+            query_attn_mask = torch.ones(self.num_query_tokens, self.num_query_tokens)
+            q = torch.arange(self.num_query_tokens)
+            query_attn_mask.masked_fill_(q > q[:, None], 0)
+            self.query_attn_mask = query_attn_mask[None]
+        else:
+            raise NotImplementedError(f'Do not support {self.query_attn_mask} query_attn_mask')
+
+        self.post_init()
+
+    def _create_pos_embed(self, *size: int, type: str = 'none') -> torch.Tensor:
+        C = self.config.encoder_hidden_size
+        if type == 'none':
+            pos_embed = None
+        elif type == 'learnable':
+            pos_embed = nn.Parameter(.02 * torch.randn(*size, C))
+        elif type == 'cosine':
+            total_len = 1
+            for i in size:
+                total_len *= i
+            raw = torch.outer(torch.arange(total_len), torch.exp(torch.arange(0, C, 2) * (-math.log(10000.) / C)))
+            pos_embed = nn.Parameter(torch.stack((raw.sin(), raw.cos()), dim=-1).view(*size, C), requires_grad=False)
+        else:
+            raise NotImplementedError(f'Do not support {type} position embeddings')
+        return pos_embed
+
+    def get_attn_mask(self, embeddings: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor]:
+        """Get visual_attn_mask and query_attn_mask if needed 
+            embeddings (torch.Tensor): (B, T, L, C)
+        """
+        B, T, L, _ = embeddings.size()
+        device = embeddings.device
+
+        # visual attn mask only work for videos
+        if T > 1:
+            if self.visual_attn_mask_type == 'ccam':
+                base_attn_mask = torch.ones(T, T, device=device)
+                t = torch.arange(T, device=device)
+                base_attn_mask.masked_fill_(t > t[:, None], 0)
+                visual_attn_mask = torch.cat((
+                    torch.kron(
+                        base_attn_mask,
+                        torch.ones(self.num_query_tokens // T, L, device=device)
+                    ),
+                    torch.ones(self.num_query_tokens % T, T * L, device=device)
+                ), dim=0)[None].expand(B, -1, -1)
+            elif self.visual_attn_mask_type == 'full':
+                visual_attn_mask = None
+            else:
+                raise NotImplementedError(f'Do not support {self.visual_attn_mask_type} attn_mask')
+        else:
+            visual_attn_mask = None
+
+        if self.query_attn_mask is None:
+            query_attn_mask = None
+        else:
+            query_attn_mask = self.query_attn_mask.expand(B, -1, -1)
+
+        return visual_attn_mask, query_attn_mask
+
+    def batch_forward_no_spatial(self, visual_embeds: torch.Tensor) -> torch.Tensor:
+        """Batch forward without spatial mask position embeddings
+
+        Args:
+            visual_embeds (torch.Tensor): (B, T, L, C)
+
+        Returns:
+            torch.Tensor: (B, Q, C)
+        """
+        B, T, _, C = visual_embeds.size()
+        query_embeds = self.query_tokens.expand(B, -1, -1)
+        visual_attn_mask, query_attn_mask = self.get_attn_mask(visual_embeds)
+
+        # add temporal position embeddings
+        if self.temporal_pos_embed is not None:
+            if T == self.temporal_pos_embed.size(0):
+                pos_embed = self.temporal_pos_embed
+            elif T == 1:
+                pos_embed = 0. * self.temporal_pos_embed[:1]     # for deepspeed 
+            else:
+                pos_embed = interpolate(
+                    self.temporal_pos_embed.T[None],    # (1, C, t)
+                    size=(T,),
+                    mode='linear',
+                    align_corners=False
+                )[0].T      # (T, C)
+            visual_embeds = visual_embeds + pos_embed.view(1, T, 1, C)
+        visual_embeds = visual_embeds.flatten(1, 2)
+
+        return super().forward(
+            query_embeds=query_embeds,
+            attention_mask=query_attn_mask,
+            encoder_hidden_states=visual_embeds,
+            encoder_attention_mask=visual_attn_mask
+        )[0]
+
+    def forward(self, visual_embeds: torch.Tensor, split_sizes: list[int], unmasked_ids: torch.LongTensor = None):
+        """
+            visual_embeds (torch.Tensor): (T, L, C)
+            split_sizes (list[int]): [t0, t1, ...] sum_i ti=T
+            unmasked_ids (torch.LongTensor): If provided, should be in the shape of (T, L) whose value v 0<=v<=HW-1
+            output_attentions (_type_, optional): _description_. Defaults to None.
+            output_hidden_states (_type_, optional): _description_. Defaults to None.
+            return_dict (_type_, optional): _description_. Defaults to None.
+        """
+        _, L, C = visual_embeds.size()
+
+        # add spatial position embeddings
+        if self.spatial_pos_embed is not None:
+            pos_embed = self.spatial_pos_embed.view(-1, C)  # (H*W, C)
+            if unmasked_ids is None:
+                pos_embed = pos_embed.view(1, L, C)     # if not provided, L must equals to H*W
+            else:
+                pos_embed = pos_embed[unmasked_ids]     # (T, L, C)
+            visual_embeds = visual_embeds + pos_embed
+
+        # all inputs in this batch has the same t
+        if len(set(split_sizes)) == 1:
+            visual_embeds = visual_embeds.view(len(split_sizes), split_sizes[0], L, C)
+            output = self.batch_forward_no_spatial(visual_embeds)
+        else:
+            visual_embeds = visual_embeds.split(split_sizes, dim=0)
+            # group visual_embeds accoding to the number of frames
+            output, group_visual_embeds = [None] * len(split_sizes), {}
+            for idx, (embed, t) in enumerate(zip(visual_embeds, split_sizes)):
+                if t in group_visual_embeds:
+                    group_visual_embeds[t][0].append(idx)
+                    group_visual_embeds[t][1].append(embed)
+                else:
+                    group_visual_embeds[t] = [[idx], [embed]]
+            for idx, embeds in group_visual_embeds.values():
+                cur_output = self.batch_forward_no_spatial(torch.stack(embeds, dim=0))
+                for i, j in enumerate(idx):
+                    output[j] = cur_output[i]
+            output = torch.stack(output, dim=0)
+
+        return output

projector/pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:35ba667d54cfc5ced0976b551cc12d4d521ee9e0f36516c1c8c48805439520d0
+size 112011169

visual_encoder_adapter/README.md

@@ -0,0 +1,202 @@
+---
+library_name: peft
+base_model: /group/40006/jaronfei/models/siglip-so400m-patch14-384
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.10.0

visual_encoder_adapter/adapter_config.json

@@ -0,0 +1,36 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": {
+    "base_model_class": "SiglipVisionModel",
+    "parent_library": "xtuner.model.modules.visual_encoder.factory"
+  },
+  "base_model_name_or_path": "/group/40006/jaronfei/models/siglip-so400m-patch14-384",
+  "bias": "none",
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 16,
+  "lora_dropout": 0.05,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 64,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "k_proj",
+    "fc1",
+    "v_proj",
+    "q_proj",
+    "fc2",
+    "out_proj"
+  ],
+  "task_type": null,
+  "use_dora": false,
+  "use_rslora": false
+}

visual_encoder_adapter/adapter_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:10774c93bff897b58b71457a023e37f2e6f1b06b943115e490ede1326eaac0ab
+size 142627250