Skip to content

Commit

Permalink
update docs
Browse files Browse the repository at this point in the history
  • Loading branch information
julienkay committed Dec 21, 2023
1 parent 9bfc7a2 commit 0db4790
Show file tree
Hide file tree
Showing 5 changed files with 60 additions and 44 deletions.
3 changes: 2 additions & 1 deletion docs/docfx.json
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,8 @@
"globalMetadata": {
"_appFaviconPath": "images/favicon.ico",
"_appLogoPath": "images/doji.png",
"_appLogoUrl": "https://www.doji-tech.com/"
"_appLogoUrl": "https://www.doji-tech.com/",
"_disableContribution": true
},
"fileMetadataFiles": [],
"template": [
Expand Down
Binary file added docs/images/banner.webp
Binary file not shown.
4 changes: 4 additions & 0 deletions docs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,4 +2,8 @@

[Midas][1] unlocks real-time monocular depth estimation with MiDaS models running in Unity Sentis.

To get started, please see the [Getting Started](manual/getting-started.md) page. The package also provides a number of samples that can be imported via the Package Manager.

![banner](./images/banner.webp)

[1]: https://assetstore.unity.com/packages/slug/268501
58 changes: 37 additions & 21 deletions docs/manual/getting-started.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,50 +2,66 @@

This page will cover the basics of using the Midas package.


## Basic Usage

To start, create a new 'Midas' instance:

```CSharp
using UnityEngine;
var midas = new Midas();
```

public class DepthEstimationExample : MonoBehaviour {
public Texture inputImage;
public Midas midasInstance;
Then you can pass in a Texture2D to @Midas.Midas.EstimateDepth(UnityEngine.Texture,System.Boolean)

void Start() {
// Initialize Midas with default parameters
midasInstance = new Midas();
```CSharp
// Estimate depth from an input texture
var predictedDepth = midas.EstimateDepth(inputImage);

}
// ... use predictedDepth as needed
```

void EstimateDepth() {
// Estimate depth from input texture
midasInstance.EstimateDepth(inputImage);
Finally, when you're done you should call 'Dispose()' to properly free up native memory resources.

// Access depth result for further processing or visualization
RenderTexture depthResult = midasInstance.Result;
```CSharp
midas.Dispose();
```

// Use depthResult as needed
You can reuse the same 'Midas' isntance for multiple inferences. But if you only need it once, you might want to use the 'using' statement, which automatically Dispose() for you:

}
```CSharp
using (Midas midas = new Midas()) {
// your code
}
```

## Accessing Depth Results
A simple example on how to use the library can also be found in the 'Basic Sample' that can be imported via the Package Manager.

The depth estimation results are stored in the Result property of the Midas class. This property is a RenderTexture containing the estimated depth.

## Choosing a Model Type

Use the ModelType property to set the desired Midas model. Default is midas_v21_small_256.
Changing the model type automatically disposes of the existing model and initializes the new one.

```CSharp
var midas = new Midas(ModelType.dpt_beit_large_384);
```

Changing the model type on an existing 'Midas' instance automatically disposes of the existing model and initializes the new one.

```CSharp
midas.ModelType = ModelType.midas_v21_small_256;
```


## Choosing a Backend

Use the Backend property to set the desired backend for model execution (GPUCompute by default).
Changing the backend automatically disposes of the existing model and initializes the new one.

## Normalizing Depth (Optional)
```CSharp
midas.Backend = BackendType.CPU;
```


Set the NormalizeDepth property to true if you want to normalize the estimated depth values between 0 and 1. This is mainly useful for visualization.
## Normalizing Depth

A simple example on how to use the library can be found in the 'Basic Sample' that can be imported via the Package Manager.
Set the NormalizeDepth property to true if you want to normalize the estimated depth values between 0 and 1. This is mainly useful for visualization.
39 changes: 17 additions & 22 deletions docs/manual/models.md
Original file line number Diff line number Diff line change
@@ -1,39 +1,34 @@
# MiDaS Models

There are variety of different MiDaS models available. To be able to use them with Unity Sentis, the official models were converted to ONNX using [this colab notebook][colab]. You'll find links to the pretraine models in ONNX format below.
There are variety of different MiDaS models available. To be able to use them with Unity Sentis, the official models were converted to ONNX using [this colab notebook][colab]. You'll find links to the pretrained models in ONNX format below.


## Overview

| Model Type | Size | Version |
|-------------------------------|----------|-----------|
| [midas_v21_small_256][1] | 63 MB | 2.1 |
| [midas_v21_384][2] | 397 MB | 2.1 |
| [dpt_beit_large_512][3] | 1.34 GB | 3.1 |
| [dpt_beit_large_384][4] | 1.34 GB | 3.1 |
| [dpt_beit_base_384][5] | 450 MB | 3.1 |
| [dpt_swin2_large_384][6] | 832 MB | 3.1 |
| [dpt_swin2_base_384][7] | 410 MB | 3.1 |
| [dpt_swin2_tiny_256][8] | 157 MB | 3.1 |
| [dpt_swin_large_384][9] | 854 MB | 3.1 |
| [dpt_next_vit_large_384][10] | 267 MB | 3.1 |
| [dpt_levit_224][11] | 136 MB | 3.0 |
| [dpt_large_384][12] | 1.27 GB | 3.0 |
| Model Type | Size | MiDaS Version |
|-------------------------------|----------|-----------------|
| [midas_v21_small_256][1] | 63 MB | 2.1 |
| [midas_v21_384][2] | 397 MB | 2.1 |
| [dpt_beit_large_512][3] | 1.34 GB | 3.1 |
| [dpt_beit_large_384][4] | 1.34 GB | 3.1 |
| [dpt_beit_base_384][5] | 450 MB | 3.1 |
| [dpt_swin2_large_384][6] | 832 MB | 3.1 |
| [dpt_swin2_base_384][7] | 410 MB | 3.1 |
| [dpt_swin2_tiny_256][8] | 157 MB | 3.1 |
| [dpt_swin_large_384][9] | 854 MB | 3.1 |
| [dpt_next_vit_large_384][10] | 267 MB | 3.1 |
| [dpt_levit_224][11] | 136 MB | 3.0 |
| [dpt_large_384][12] | 1.27 GB | 3.0 |


## Usage

To keep the package size reasonable, only the midas_v21_small_256 model is included with the package when downloading from the Asset Store. To use other models they have to be downloaded to your development system first.
To keep the package size reasonable, only the @Midas.ModelType.midas_v21_small_256 model is included with the package when downloading from the Asset Store. To use other models you have to downloaded them first.

When you create an instance of the [`Midas`](~/api/Midas.Midas.yml) class you can pass the [`ModelType`](~/api/Midas.ModelType.yml) in the constructor to choose which model to use. In the Unity Editor if you're about to use a model that is not yet present, it will automatically be downloading if you wish to do so.
When you create an instance of the [`Midas`](~/api/Midas.Midas.yml) class you can pass the [`ModelType`](~/api/Midas.ModelType.yml) in the constructor to choose which model to use. In the Unity Editor if you're about to use a model that is not yet present, it can automatically be downloading if you wish to do so.

Otherwise you can always manually download the ONNX models from the links above and place them inside the Resources/ONNX folder.

Choose the appropriate model type based on your requirements, considering factors such as model size, architecture, and performance.
Utilize the provided methods in the ModelExtensions class to obtain download links, file names, and resource paths for the selected model.

For more information on each model type, refer to the script comments in the ModelType enum and the ModelExtensions class.


## Which Model To Use

Expand Down

0 comments on commit 0db4790

Please sign in to comment.