[Android] Error running a transformer decoder-based neural network #8665
Comments
|
Can you please include your dev env and commit hash on main, in case developers need to reproduce the issue? |
guangy10
added
the
module: android
guangy10
added
the
need-user-input
|
Hi @adonnini Currently the demo app doesn't copy java sources (modules) (e.g. NativePeer) but use the AAR instead and the AAR contains .so and .jar (which contains all compiled java file) And you can include AAR in your project by |
|
Hi @guangy10 ,
|
|
@kirklandsign Thanks. P.S. Is the information in your message anywhere in the documentation? Could you please send me a link, if there is? Thanks |
|
Hi, sorry about this. I couldn't find setup.sh I the llama demo directory. I didn't realize that the llama set-up applies also to non llama apps. Does it?Again sorry about my clumsiness Thanks.On Feb 25, 2025 7:25 AM, Hansong ***@***.***> wrote:
Hi @adonnini
Currently the demo app doesn't copy java sources (modules) (e.g. NativePeer) but use the AAR instead https://github.com/pytorch/executorch/blob/f1a7d60e823104d1753b114656f798334fc728f3/examples/demo-apps/android/LlamaDemo/setup.sh#L21
and the AAR contains .so and .jar (which contains all compiled java file)—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>
kirklandsign left a comment (pytorch/executorch#8665)
Hi @adonnini
Currently the demo app doesn't copy java sources (modules) (e.g. NativePeer) but use the AAR instead https://github.com/pytorch/executorch/blob/f1a7d60e823104d1753b114656f798334fc728f3/examples/demo-apps/android/LlamaDemo/setup.sh#L21
and the AAR contains .so and .jar (which contains all compiled java file)
—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>
|
mergennachin
added
the
module: user experience
|
I apologize in advance for the (very) long message. Please read it to the end. I tried running setup.sh from both the LlamaDemo and ExecutorchDemo directories. They both failed.
I checked. There is no
the In setting up exeutorch I followed the instruction in However, I must have done something wrong, not sure what. I am not familiar shell scripting. I did not make any changes to the two scripts. As a workaround, I ran Android Studio reports the following error:
in this line
Note that there is a line like this in I also checked the contents of EValue.class in executorch.aar. It does contain a However, application build was successful. When I ran the application, execution still failed when attempting to load the module. Please see the error log below. Is the Please let me know what I should do next. Thanks ERROR LOG |
Thank you for reporting the issue with detailed steps for repro. The Android apps are built and running continuously in our CI, so likely what you see is due to some setup issue. To help us better understand your setup, you can attach you dev env by running the following commands: |
|
Hi @adonnini
Please run from root directory. For prebuilt AAR, maybe use this 0.5 release? Docs: https://pytorch.org/executorch/0.5/android-prebuilt-library.html#using-prebuilt-libraries BTW the AAR is same for both llama and non-llama. The docs above is not updated. I will update docs. |
@adonnini Could you please run the script and attach the outputs? |
|
@kirklandsign thanks for the quick response. All set wrt exeuctorch.aar The .aar I downloaded with the workaround I used is the correct one. Thanks for confirming that. Now, it's a question of resolving the error in the subject line of this issue |
|
@guangy10 Sorry!! Here is the output. Please let me know if I messed up again. Thanks Please let me know if you need additional information to resolve the ``NoSuchMethodError``` |
guangy10
added
triaged
|
Hi @adonnini Let's confirm the SHA of executorch.aar is Also I have the following deps in build.gradle.kts |
|
My dependencies I will change the dependencies to match yours and change the executorch.aar using this https://pytorch.org/executorch/0.5/android-prebuilt-library.html#using-prebuilt-libraries Thanks |
|
@kirklandsign I changed the dependencies and downloaded the aar using the instructions in https://pytorch.org/executorch/0.5/android-prebuilt-library.html#using-prebuilt-libraries I have no way to verify the SHA of the aar I downloaded using the instructions above. BTW, the actual download command from the docs is:
Is there a release newer than My dependencies in build.gradle Application compile fails with the following error Cannot resolve method 'toTensor' in 'EValue' |
|
I did some more digging. The ExecuTorchDemo android application builds successfully. No The executorch dependencies in the build.gradle files of my application and ExecuTorchDemo are the same. A far as I can tell, the main differences are:
If I add this dependency to build.gradle in my application, gradle sync fails with the following error: If I remove the exclude clause, and just use |
|
Hi @adonnini Thank you for your updates! Sorry would you mind using https://github.com/pytorch/executorch/tree/main/extension/benchmark/android/benchmark as an example, instead of ExecuTorchDemo? I used |
I probably saw this kind of error from android studio in the past but I can finally compile. Is it android studio code analysis, or real compilation error? |
0.6 not yet available. |
|
@kirklandsign Thanks! |
|
@kirklandsign I also tried Android Studio restart invalidating caches and deleted the build folder for app. made no difference. So, I am stuck. Not quite. I made the following change: The error I am not familiar enough with Tensor constructs to determine how accessing the data at location Thanks |
|
Hi @adonnini
However, seems that |
|
May I ask what kind of model and input did you use? |
|
Seems that your input is a tensor right? May I ask what is inside arrDataPytorch? |
|
I do have
the model itself is a transformer decoder-based neural network. (https://arxiv.org/abs/2210.04095). |
|
for do you need the last 1 dimension in Does the model need a 3-D input? |
|
Also, let's do a really fast sanity check, what about you use |
|
@kirklandsign Thanks again for your help. I really appreciate it. I changed to 2-d input, just in case it made a difference although for my model I am pretty sure I do need 3-d input. Unfortunately, the result is the same, Please see the error log below. The sanity check you suggested worked (see output of first I also added output to verify that What do you think is going on? Thanks ERROR LOG |
|
@kirklandsign This one may also be related (I opened it) |
|
So it means the model can run if all inputs are ones if the dimension of the input tensor is correct, unless there is a dynamic shape in the model (is this the case?) Do you know how you exported the model? What's the sample input when you export? |
|
Here is the code I use to export the model: Please note that the code above did produce a .pte file. Do you want me to send it to you? YES, there are dynamic shapes in the models I use. I don't know how many times your colleagues have asked me this question! There is a long history of (a few) issues I created related to the models I use and dynamic shapes, etc., etc. I don't know how any changes I made to the model export code. Frankly, at least in some instances it looked like the change I was asked to make was just for the sake of trying something to see what would happen (not taking in consideration my time) Unfortunately, all of the issues I created except for (#2163) are still open. I am still waiting for someone/anyone to get back to me. I think that if you put yourself in my place, you would feel at least a little exasperated. Did you take a look at the issue which I thought might possibly be related to the latest problem I have run into (perhaps with the exception of #2163)? Thanks |
Does it mean the model takes two parameters? Do you know what is the type and dimensions of Basically we need |
I need your help. I work in Java not Python.
AND where
Given the above, it might not be possible for me to use Executorch to run my model for inference in my Android application. Please let me know what you think. Thanks |
|
To make sure things work, |
|
Basically you need to transform If you have an idea how to do it in python or c++, it is the same in java.
|
|
A couple of things:
Frankly, I am getting close to the point of giving up and look for alternatives. I know you are very busy and am grateful for you support. Thanks |
|
Below you will find a first attempt at Java code to create Two unanswered (unsolved) questions:
A couple of things to note about the code below: a) It is not done b) I may need to define a I would appreciate your feedback. In my opinion, unless I am really missing something point 1) above may be a show stopper. Thanks JAVA CODE TO PRODUCE X, XDICT |
Lines 16 to 28 in 461d61d Therefore I guess when you export the model, it's as if
|
kirklandsign
changed the title
|
Thanks. I just want to make sure I understand correctly what you say about the Input can be (float[][], constant) where constant is of type from the list you referred to in your comment above. Is this right? Where can I find the documentation about this? Thanks |
|
I mean this is how you exported the model right? Here Therefore, the input for |
And see https://www.geeksforgeeks.org/variable-arguments-varargs-in-java/ |
|
This repository contains my updated model (python) I am familiar with x_dict is the "dictionary" containing all the configuration variables (the map where each key is a variable name). I will need to figure out how to implement it in Java. I'll need to figure out whether your guess "(my guess is it's not really a used variable, but a export-time const)." is correct. Thanks again for your help. |
|
I tried the following: Execution failed producing the error log you will find below. When I ran the only difference in the error log was this line Doesn't this mean that the error reported in the error log below is not related to the number of parameters in the forward call? What am I missing? Thanks ERROR LOG |
The way torch.export works (and therefore ExecuTorch) is it flattens all data structures. Heres a doc explaining the concept if you are unfamiliar (it's from JAX but the principal is the same in PyTorch) https://docs.jax.dev/en/latest/pytrees.html. What this practically means is that the key set of your dict needs to be fixed/static. The value's shapes can still be dynamic afaik if that matters for your model. Then at runtime you will need to manually flatten your structure before converting the values to EValues and passing to ET. I see this stack is about android so we probably don't have java bindings for our cpp flattening lib today. https://github.com/pytorch/executorch/tree/main/extension/pytree |
|
Thanks for the information/explanation. In my application, I do what you say should be done: For my model the export works as far as I can tell as the process produces a .pte file The shapes in my model are dynamic. The key set of the dict xdict is fixed/static. However, it is not a single constant. It'a a Map<String,String>. This is where I am stuck. When I export the model, the inputs are the input data x and xdict, and everything seems to work (the.pte is produced) When I run Module.forward in my Android application, after loading the model, the problem is that unless I am mistaken EValue.from (the forward parameter type) does not take a Map as an argument. I am not sure what I should do next. |
|
My understanding is we need to implement flattening a dict to support this. |
|
@kirklandsign OK. But doesn't the fact that the 0x12 error occur also when I use two parameters in the forwad call seem to indicate that the error is not related to the fact that I did not use xdict? When I used two paramters in the forward call the failure should have occurred when actually running for inference. That was not the case. Please I would like to understand what is wrong with my reasoning. If what I say is correct, then what I do next is under my control, once the 0x12 error issue is resolved. Thanks |
Correct, that's the current issue. @JacobSzwejbka should we
|
Hi,
@kirklandsign I have been developing on Android for more than 15 years. I am familiar with dependencies etc., etc. The issue 8636 had to do with packaging.
I guess you did not see my last comment in
#8636
Here it is again:
Hi Update #2
I worked around the classNotFound exception by creating a java.org.pytorch.executorch package containing all the source files from
executorch/extension/android/src/main/java/org/pytorch/executorch
Now execution fails producing the error log reported below.
as in the android demo app
At this point, I am at a loss as to what I am doing wrong and what I should try next.
Should I add and android module to my project just as there is one in the android demo app. In the android demo app, the android module contains all the modules necessary to use the executorch runtime (e.g. NativePeer) in the application.
I think the issue may be related to multi-project set-up which I do not use in my application (although I do have multiple modules within the project.
Generally speaking, I still have the sense that the set-up to use the executorch runtime on Android applications could/should be simpler, for example not requiring the use of the wrapper modules in
org.pytorch.executorchPlease let me know if you need additional information.
Thanks
ERROR LOG
cc @kirklandsign @cbilgin @mergennachin @byjlw
The text was updated successfully, but these errors were encountered: