README.md

metadata

license: other
pipeline_tag: image-to-image
tags:
  - android
  - neural-graphics
  - gaming
  - graphics
language:
  - en

Neural Super Sampling (NSS)

Neural Super Sampling (NSS) is an innovative, efficient network for temporal super sampling on mobile devices. Content rendered at 540p can be upscaled to 1080p, resulting in up to 50% GPU savings. With our retraining tools content creators and game studios can build derivatives of the model suited to artwork style and performance requirements.

🎥 Neural Super Sampling Demo

Model Details

Neural Super Sampling (NSS) is a parameter prediction model for real-time temporal super sampling developed by Arm, optimized for execution on Neural Accelerators (NX) in mobile GPUs. It enables high-resolution rendering at a lower compute cost by reconstructing high-quality output frames from low-resolution temporal inputs. NSS is particularly suited for mobile gaming, XR, and other power-constrained graphics use cases.

• Developed by: Arm Limited
• Model type: Temporal image super sampling
• License: Other
• Repository: Neural Graphics Model Gym
• Blog: How Neural Super Sampling Works
• Paper: Mobile Neural Super Sampling
• Quickstart with ML extensions for Vulkan®: ML extensions for Vulkan® Quickstart Guide
• Quickstart with Unreal: Neural Super Sampling Quickstart Guide for NSS integration into Unreal Engine

NSS is under active development with regular updates planned. As we increase the size and diversity of the training dataset we expect to see significant quality improvements. Follow Arm to stay up to date on the latest releases.

The model is released under Arm's AI Model Community License which allows NSS to be retrained on datasets captured from your own content. Future releases of the Neural Graphics Model Gym will provide the tools to capture and convert content for use in (re)retraining.

Uses

NSS can be directly integrated into graphics pipelines using ML extensions for Vulkan®. See included ML SDK for Vulkan scenario for the simplest way to evaluate the model. The scenario includes the necessary pre- and post-processing compute shaders along with a single frame worth of input data.

The recommended way of integrating the model into a graphics pipeline is by using the VGF Library from the ML SDK for Vulkan.

NSS is released under a permissive license designed to foster innovation in the graphics industry and provide differentiation to content creators.

Direct Use

NSS has been integrated into Unreal Engine via the NSS Plugin for Unreal Engine. See our quick start guide for step-by-step instructions on how to use NSS in Unreal® Engine.

Out-of-Scope Use

• Not suited for non-temporal tasks such as a standalone image upsampling

Bias, Risks, and Limitations

• Requires accurate motion vectors and frame history for stable output
• May underperform in extremely low framerate scenarios (<10 FPS) with fast camera movement
• Padding of the input is needed if input dimensions are not divisible by 8

Recommendations

For ultra-low-FPS use cases, reduce the camera speed, acceleration, or both so that the relative motion between frames mimics the application running at a higher frame rate.

How to Get Started with the Model

This repository contains pre-trained weights and a compiled NSS model in VGF format, ready for integration with Vulkan applications.

The included Scenario demonstrates full execution of the model on a Vulkan compute-capable system. An Emulation Layer is provided to implement ML Extensions for Vulkan where it is not supported by the native Vulkan driver.

Download and Prepare the Scenario

These instructions will work on both Windows and Linux. On Windows, we strongly recommend the use of PowerShell over cmd.exe ("Command Prompt").

1. Ensure that Git LFS is installed. See the project's website for more details.

2. Clone the NSS model repository from Hugging Face:

git clone https://huggingface.co/Arm/neural-super-sampling
cd neural-super-sampling
git lfs pull

Install the Scenario Runner and Emulation Layers

1. Follow the steps in the Python documentation's "Create a new virtual environment" guide. We will assume below that you installed this environment into a .venv subdirectory.

2. Follow the steps in the Python documentation's "Activate a virtual environment" guide.

   Note for Windows users: It is not necessary to determine the location of the Python interpreter as described in this guide. For reference, however, the correct PowerShell command to do this is gcm python rather than the given where python.

3. Install Arm's Scenario Runner and Emulation Layer into your new virtual environment:

   pip install ai-ml-sdk-scenario-runner ai-ml-emulation-layer-for-vulkan
   

4. The Emulation Layer's documentation explains how its environment variables should be configured on Linux and on Windows. The following is a simplified summary, assuming that you are at the root of the NSS model repository.

   • On Linux, enter these commands into Bash:

     EMULATION_LAYER_DIR=$(find $PWD/.venv -name emulation_layer -type d)
     export LD_LIBRARY_PATH="$EMULATION_LAYER_DIR/deploy/lib:$LD_LIBRARY_PATH"
     export VK_LAYER_PATH="$EMULATION_LAYER_DIR/deploy/share/vulkan/explicit_layer.d:$VK_LAYER_PATH"
     export VK_INSTANCE_LAYERS="VK_LAYER_ML_Graph_Emulation:VK_LAYER_ML_Tensor_Emulation"
     

   • On Windows, enter these commands into PowerShell (cmd.exe requires a different syntax):

   $env:VK_LAYER_PATH="$pwd\.venv\Lib\site-packages\emulation_layer\deploy\bin;$env:VK_LAYER_PATH"
   $env:VK_INSTANCE_LAYERS="VK_LAYER_ML_Graph_Emulation;VK_LAYER_ML_Tensor_Emulation"
   

Run the Scenario

• Execute the Scenario Runner:

  scenario-runner --scenario scenario/scenario.json --output out
  

• Output images are encoded as B10G11R11_UFLOAT. This format is common for framebuffers but not widely supported by image viewers. Use RenderDoc to view these images.

Training and Evaluation

For background on NSS architecture and training read our blog: How Neural Super Sampling Works

Training and evaluation details, including model architecture code, training pipeline, and test configurations, are available at:

• Model training code: https://github.com/arm/neural-graphics-model-gym
• Examples and tutorials: https://github.com/arm/neural-graphics-model-gym-examples
• Sample dataset: https://huggingface.co/datasets/Arm/neural-graphics-dataset

🔎 Model Explorer VGF extension

The VGF extension to Model Explorer provides a simple interface to visualize model and analyse VGF composition.

License

• The license for the model source code can be found here.
• The license for the content of this repository can be found here

More Information

🧑‍🔬 More technical details about the model can be found in the NSS Guide.

👩🏽‍💻 Our Neural Graphics Development Kit contains engine plugins, model training tools, code examples and extensive developer documentation.

🙋🏻‍♀️ For questions or feedback please start a discussion

Trademark notice

• Arm® is a registered trademark of Arm Limited (or its subsidiaries) in the US and/or elsewhere.
• Python® is a registered trademark of the Python Software Foundation.
• Vulkan® is a registered trademark of the Khronos® Group.
• Windows® is a trademark of the Microsoft group of companies.