How can I directly use the functions of RealSR without calling through the CLI?

[laverne01]

I tried creating a convert function in JNI (which calls the process function of RealSR) and used it in Kotlin. However, the output image after upscaling is just black. I would appreciate your advice on what might be going wrong.

JNIEXPORT jobject JNICALL Java_com_example_realsrandroid_ImageUpscale_convert( JNIEnv *env, jobject /* this */, jobject bitmapIn, jobject assetManager) { ncnn::Mat in_img = ncnn::Mat::from_android_bitmap(env, bitmapIn,ANDROID_BITMAP_FORMAT_RGBA_8888); ncnn::Mat out_img = ncnn::Mat(in_img.w * scale, in_img.h * scale, in_img.c, (size_t) in_img.elemsize, in_img.elempack); realsr.process(in_img, out_img); out_img.to_android_bitmap(env, bitmapIn, ANDROID_BITMAP_FORMAT_RGBA_8888); return bitmapIn; }

I have initialized RealSR on it.

realsr.scale = 4; realsr.tilesize = 200; realsr.prepadding = 10; AAssetManager *mgr = AAssetManager_fromJava(env, assetManager); realsr.load(mgr);

[RobbWatershed]

I've buried hours into that problem and didn't find a working solution for retrieving the output bitmap in memory from NCNN's native environment to my app's Java/Kotlin environment.

I eventually ended up :

• C++ side : writing a PNG file to storage once the image has been rescaled
• Kotlin side : reading the resulting PNG file from storage

Here's the Kotlin side of the JNI interface I made : https://github.com/avluis/Hentoid/blob/master/app/ai-upscale/src/main/java/me/devsaki/hentoid/ai_upscale/AiUpscaler.kt

• dataIn : input Bitmap; passed in memory
• outPath : path to the PNG file to save (points to the app's private temp storage)

If you find anything more elegant, I'd be super glad to know 🙃

[laverne01]

@RobbWatershed Thank you, this helps me a lot!

[tumuyan]

Calling functions directly with jni instead of writing to a file is difficult.
I tried but failed, so that's why I'm using the current solution

[laverne01]

Calling the function directly makes the code look better. I will try it out and will message back if I have a solution. Thank you.

[RobbWatershed]

FYI, if that might be of any help, I've made my AI rescale module a standalone project : https://github.com/RobbWatershed/AI_Upscale

[laverne01]

Thank you! I was able to directly call the function with JNI without using the CLI. The next thing I’m working on is using GFPGAN or Codeformer for face enhancement on Android. Do you have any ideas or documentation for this?

[RobbWatershed]

The next thing I’m working on is using GFPGAN or Codeformer for face enhancement on Android. Do you have any ideas or documentation for this?

Nope, sorry. I'm more into drawn image enhancement.

[QQHEZhangyubin]

谢谢！我能够直接使用 JNI 调用该函数，而无需使用 CLI。我接下来要做的是使用 GFPGAN 或 Codeformer 在 Android 上进行人脸增强。你对此有什么想法或文档吗？

I'm curious about how you achieved this. I also tried to call directly through JNI, but the processed image was black. Here is my code:

// 模型加载函数
extern "C"
JNIEXPORT jboolean JNICALL
Java_com_example_machine_1learning_1ncnn_NativeLibNCNN_initSuperResolutionModelJNI(
JNIEnv *env,
jobject thiz,
jstring parm_file_path,
jstring bin_file_path) {

// 获取模型文件路径
const char *paramfilePath = env->GetStringUTFChars(parm_file_path, JNI_FALSE);
const char *binfilePath = env->GetStringUTFChars(bin_file_path, JNI_FALSE);


// 初始化 GPU
ncnn::create_gpu_instance();
int gpu_id = ncnn::get_default_gpu_index();

// 如果之前已经有 RealSR 实例，先释放
if (g_realsr) {
    delete g_realsr;
    g_realsr = nullptr;
}

// 创建 RealSR 实例
g_realsr = new RealSR(gpu_id, 0, 1); // tta_mode=0, num_threads=1

// 加载模型
std::string param_path = std::string(paramfilePath);
std::string model_path_bin = std::string(binfilePath);

if (g_realsr->load(param_path, model_path_bin)) {
    LOGE("Failed to load model");
    delete g_realsr;
    g_realsr = nullptr;
    ncnn::destroy_gpu_instance();
    // 释放 Java 字符串
    env->ReleaseStringUTFChars(parm_file_path, paramfilePath);
    env->ReleaseStringUTFChars(bin_file_path, binfilePath);
    return JNI_FALSE;
}

// 设置参数
g_realsr->scale = 4;
g_realsr->tilesize = 8;
g_realsr->prepadding = 100;

// 释放 Java 字符串
env->ReleaseStringUTFChars(parm_file_path, paramfilePath);
env->ReleaseStringUTFChars(bin_file_path, binfilePath);

return JNI_TRUE;

}

====================================================================================

// 图片处理函数
extern "C"
JNIEXPORT jstring JNICALL
Java_com_example_machine_1learning_1ncnn_NativeLibNCNN_runSuperResolutionModelJNI(
JNIEnv *env,
jobject thiz,
jstring inputPath) {

// 检查是否已经加载了模型
if (!g_realsr) {
    LOGE("Model is not loaded. Call nativeLoadModel first.");
    return JNI_FALSE;
}

// Convert Java strings to C strings
const char *input_path = env->GetStringUTFChars(inputPath, nullptr);

LOGI("Processing image: %s", input_path);
std::string srcImgPathCPP(input_path);

// 加载输入图像
int w, h, c;
unsigned char* pixeldata = stbi_load(input_path, &w, &h, &c, 0);
if (!pixeldata) {
    LOGE("Failed to load input image");
    env->ReleaseStringUTFChars(inputPath, input_path);
    return JNI_FALSE;
}

ncnn::Mat inImage;
if (c == 3)
    inImage = ncnn::Mat::from_pixels(pixeldata, ncnn::Mat::PIXEL_RGB, w, h);
else if (c == 4)
    inImage = ncnn::Mat::from_pixels(pixeldata, ncnn::Mat::PIXEL_RGBA, w, h);
else {
    fprintf(stderr, "unsupported channel number %d\n", c);
    stbi_image_free(pixeldata);
    return JNI_FALSE;
}

// ncnn::Mat outImage;
int outWidth = w * g_realsr->scale;
int outHeight = h * g_realsr->scale;
ncnn::Mat outImage = ncnn::Mat(outWidth, outHeight, (size_t) c, c);
// 处理图像
__android_log_print(ANDROID_LOG_INFO, "RealSR", "ppppppp");
g_realsr->process(inImage, outImage);

// 在当前目录下创建一个super-resolution-output.png文件
std::string outputImagePath =
        srcImgPathCPP.substr(0, srcImgPathCPP.find_last_of('.')) + "_super-resolution-output.jpg";

cv::Mat image;
image = cv::Mat(outImage.h, outImage.w, CV_8UC3, outImage.data); // 3通道图像
cv::cvtColor(image, image, cv::COLOR_RGB2BGR);
cv::imwrite(outputImagePath, image);

// 释放 Java 字符串
env->ReleaseStringUTFChars(inputPath, input_path);

return env->NewStringUTF(outputImagePath.c_str());

}