| # Evaluating Anime Models Systematically - Basics |
|
|
| I was trying to refine my character models when I realized how I've been making models is really inefficient. |
| It typically goes like, tweak some configs or data, try some random prompts, see if they look okay. |
| It should be helpful to establish a well-defined procedure. |
| Then it's apparent that to evaluate fine-tuned models, knowing and quantifying how the base models perform as a baseline is essential. |
| So here I am, trying to evaluate base models. |
|
|
| I collected 1000 random prompts from Danbooru posts from 2021-2022 with the query `chartags:0 -is:child -rating:e,q order:random score:>=10 filetype:jpg,png,webp ratio:0.45..2.1` |
| and generated 1000 640x640 images with them for each of 3 widely-used anime models: |
| [animefull-latest](https://huggingface.co/deepghs/animefull-latest), |
| [Counterfeit-V3.0](https://civitai.com/models/4468?modelVersionId=57618), [MeinaMix_V11](https://huggingface.co/Meina/MeinaMix_V11). |
|
|
|
|
| A model can be evaluated over a number of aspects: fidelity, text-image alignment, aesthetics, diversity. Let's go through them one by one. |
|
|
| ## Fidelity |
|
|
| Generated images should be indistinguishable from real ones. They should make sense and not contain obvious errors such as extra limbs, mutated fingers, glitches or random blobs. |
| In literature, it's common to use metrics based on distribution distance, such as FID and IS. I calculated the KID score of the 3 sets of images against the 1000 real images. |
|
|
| | model | KID (lower better) | |
| |---|--| |
| | animefull-latest | 0.01192 | |
| | Counterfeit-V3.0 | 0.01807 | |
| | MeinaMix_V11 | 0.01345 | |
| |
| It seems like that KID does not align with human evaluation, which would generally rate animefull-latest as the worst one. |
| This is kind of expected, since models with strong style would have a different image feature distribution than random real images. |
| |
| I also tried multimodal LLMs, including GPT-4V and LLaVA, and unfortunately find them quite useless. GPT-4V is supposedly SOTA, but it's clear that it quite useless at spotting generation errors. |
| |
| |
|  |
| |
| |
|  |
| |
| So currently I can't find a process that computes a fidelity score for anime models. Have to wait for someone to train a specialized model for now. |
| |
| |
| ## Text-Image Alignment |
| |
| Generated images should not contradict the text prompts. A popular metric is the CLIP score, which is the cosine similarity of the projected CLIP embeddings. |
| There's also [PickScore_v1](https://huggingface.co/yuvalkirstain/PickScore_v1) which is fine-tuned on human preference data. |
| These are not well-suited for anime models due to how different Booru tagging is from regular images. |
| |
| Model using booru-tag prompts can be evaluated with a tagger. Specifically, I used [wd-v1-4-moat-tagger-v2](https://huggingface.co/SmilingWolf/wd-v1-4-moat-tagger-v2) with a threshold of 0.35. |
| A tag accuracy score can be defined as `#{prompted tags correctly reproduced}/#{prompted tags}`. The accuracy is macro-averaged over all images. Here are the scores: |
| |
| | model | tag accuracy (higher better) | |
| |---|--| |
| | animefull-latest | 0.464328 | |
| | Counterfeit-V3.0 | 0.434574 | |
| | MeinaMix_V11 | 0.375389 | |
|
|
| It can be seen that fine-tunes or merges may produce nicer images but at the cost of controllability. |
|
|
| ## Aesthetics |
|
|
| Images should be pretty. While this is generally subjective, there are models that give an aesthetic score, either averaged from many people's preferences or personalized. |
| There are CLIP based models ([aesthetic-predictor](https://github.com/LAION-AI/aesthetic-predictor), [improved-aesthetic-predictor](https://github.com/christophschuhmann/improved-aesthetic-predictor)) |
| and some custom models ([anime-aesthetic](https://huggingface.co/spaces/skytnt/anime-aesthetic-predict), [cafe_aesthetic](https://huggingface.co/cafeai/cafe_aesthetic)). |
|
|
| I tested averaged improved-aesthetic-predictor and anime-aesthetic: |
|
|
| | model | improved-aesthetic-predictor (higher better) | anime-aesthetic (higher better) | |
| |---|--|-| |
| | animefull-latest | 6.124954 | 0.639767 | |
| | Counterfeit-V3.0 | 6.359464 | 0.789190 | |
| | MeinaMix_V11 | 6.474662 | 0.829989 | |
| |
| The two scores appears to agree. |
| |
| Interestingly, GPT-4V does a reasonable job at this. |
|  |
| |
| ## Diversity |
| |
| Even with the same prompt, given different random seeds, generated images should not be repetitive. |
| There's this DIV score defined in the [Dreambooth paper](https://arxiv.org/pdf/2208.12242.pdf), which calculates image similarity with LPIPS. |
| For this particular set of images, this metric is not applicable, and I will leave it to a future update. |
| |
| ## Conclusions |
| |
| It's possible to programmatically generate some numbers given a base model. We can use the numbers as a proxy of the model's overall performance. |
| |
| |
| ## Miscellaneous notes |
| |
| I used diffusers and 13 images from animefull-latest came out as solid black images for unknown reasons even with the safety checker disabled and single precision VAE. |
| These images and their counterparts were excluded in metrics calculation. |
| |
| The images and prompts can be found [here](https://huggingface.co/datasets/gustproof/sd-data/tree/main/db1k). |
| |
| It's possible that some models perform better with special configs, but for simplicity I kept them the same. |
| |
| The code for image generation and metrics is quite messy so I'm will not upload it right now, but feel free to ask questions or give suggestions. |
| |
| I probably would create a fidelity model eventually if no one does, but it will take a while. |
| |
| |
| |
| Prompts with more tags have lower tag accuracy |
|  |
| |
| The affect of tag position is measurable albeit less pronounced. The trend at the pos 20-25 may be due to the 77-token limit wraparound. |
|  |
| |
| The next post will be about evaluating character models. |
| |
| |
