Port marian to WASM for inference

[abhi-agg]

Description

This PR enables capability to build and use marian as a WASM module for doing inference.
It is related to issues: #23

List of changes:

• Enable building only decoder (only for inference)
• Added onnxjs submodule (to provide a wasm compatible SGEMM routine)
• Updates intgemm submodule to use fast Integer 8-bit inference + simd shuffle pattern on WASM
• Updates SentencePiece submodule (to build stuff that is required just for inference)
• Build scripts (local machine and on Docker)
• Github workflow for WASM builds

Added dependencies: onnxjs submodule

How to test

To run and test, follow the README

Checklist

• I have tested the code manually
• I have run regression tests
• I have read and followed CONTRIBUTING.md
• I have updated CHANGELOG.md

[ugermann]

Before submitting pull requests please make sure that

• all changes are properly documented in CHANGELOG.md
• all regression tests pass
• all files with changes have clutter / commented-out code removed (or contain a convincing explanation why the commented-out code should be kept around

Without at least these three conditions having been met, chances of approval are very slim.

[XapaJIaMnu]

Yes, as this is only usable if we use MKL (which we can't)

…

On Tue, Mar 9, 2021 at 6:19 PM abhi-agg ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In src/tensors/cpu/prod.cpp <#24 (comment)>: > + //} + /* @XapaJIaMnu <https://github.com/XapaJIaMnu> Just to be sure, I can remove this dead code <https://github.com/browsermt/marian-dev/blob/wasm/src/tensors/cpu/prod.cpp#L203-L280> as well? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#24 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAPO5VPPE4JUUJJ53PZTQGTTCZREDANCNFSM4YS7KLFA> .

[XapaJIaMnu]

Left 2 comments.
Could you please answer all @ugermann's concerns about the disabled exceptions before merging.

[ugermann]

Could you please answer all @ugermann's concerns about the disabled exceptions before merging.

Also, all regression tests should pass before a merge into master, and all changes need to be documented in CHANGELOG.md. Otherwise, there's no chance in hell we'll ever get this past the upstream code masters for an upstream merge.

[abhi-agg]

@ugermann @kpu @XapaJIaMnu I have made lot of improvements, answered most of your concerns in the comments and added commits to resolve them. I just need to run regression tests now. It would be great if you can review again. It looks in much better shape now 👍

[abhi-agg]

I have rebased this PR on master and ran the regression tests on:

1. Latest master (commit)
2. On this PR

I am getting the exact same status for both of them which is following:

Failed:
  - tests/examples/unit-tests/test_unit_tests.sh
  - tests/scorer/scores/test_scores_cpu.sh  

1st test is failing because I have CUDA-11 and cusparseSgemmi is not available in CUDA VERSION >= 11 (see the logs below). I believe this will pass for CUDA-10.

2nd test failure is already known.

For the sake of completeness, I am sharing the failure logs for the 1st test case 👍

Failure logs for test_unit_tests.sh

+ set -e
+ cd /data/rw/pit/src/build-original
+ env CTEST_OUTPUT_ON_FAILURE=1 ctest --force-new-ctest-process
Test project /data/rw/pit/src/build-original
   Start 1: graph_tests
1/6 Test #1: graph_tests ......................   Passed    0.36 sec
   Start 2: operator_tests
2/6 Test #2: operator_tests ...................***Exception: Child aborted  0.99 sec
[2021-03-22 17:28:32] Error: cusparseSgemmi is not available in CUDA VERSION >= 11.
[2021-03-22 17:28:32] Error: Aborted from cusparseStatus_t marian::gpu::cusparseSgemmiEx(cusparseHandle_t, int, int, int, int, const float*, const float*, int, const float*, const int*, const int*, const float*, float*, int) in /data/rw/pit/src/src/tensors/gpu/prod.cpp:367

[CALL STACK]
[0x55cb46eed9c5]    marian::gpu::  CSRProd  (IntrusivePtr<marian::TensorBase>,  std::shared_ptr<marian::Allocator>,  IntrusivePtr<marian::TensorBase> const&,  IntrusivePtr<marian::TensorBase> const&,  IntrusivePtr<marian::TensorBase> const&,  IntrusivePtr<marian::TensorBase> const&,  bool,  bool,  float) + 0x1d65
[0x55cb4507057d]                                                       + 0x40d57d
[0x55cb450a1fa7]    std::_Function_handler<void (),marian::CSRDotNodeOp::forwardOps()::{lambda()#1}>::  _M_invoke  (std::_Any_data const&) + 0x207
[0x55cb4514f6d1]    marian::Node::  forward  ()                        + 0x211
[0x55cb45062d3b]    marian::ExpressionGraph::  forward  (std::__cxx11::list<IntrusivePtr<marian::Chainable<IntrusivePtr<marian::TensorBase>>>,std::allocator<IntrusivePtr<marian::Chainable<IntrusivePtr<marian::TensorBase>>>>>&,  bool) + 0x22b
[0x55cb450647bc]    marian::ExpressionGraph::  forwardNext  ()         + 0x23c
[0x55cb44f9a197]    void  tests  <float>(marian::DeviceType,  marian::Type) + 0xb5b7
[0x55cb44e9db8f]    Catch::RunContext::  invokeActiveTestCase  ()      + 0x2f
[0x55cb44ead18d]    Catch::RunContext::  runCurrentTest  (std::__cxx11::basic_string<char,std::char_traits<char>,std::allocator<char>>&,  std::__cxx11::basic_string<char,std::char_traits<char>,std::allocator<char>>&) + 0x47d
[0x55cb44ec2030]    Catch::RunContext::  runTest  (Catch::TestCase const&) + 0x1a0
[0x55cb44ec8b5c]    Catch::Session::  runInternal  ()                  + 0xa3c
[0x55cb44ec9ada]    Catch::Session::  run  ()                          + 0x1a
[0x55cb44e69186]    main                                               + 0x86
[0x7f13b6a18b97]    __libc_start_main                                  + 0xe7
[0x55cb44e9539a]    _start                                             + 0x2a


~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
run_operator_tests is a Catch v2.10.1 host application.
Run with -? for options

-------------------------------------------------------------------------------
Expression graph supports basic math operations (gpu)
 csr-dot product
-------------------------------------------------------------------------------
/data/rw/pit/src/src/tests/units/operator_tests.cpp:456
...............................................................................

/data/rw/pit/src/src/tests/units/operator_tests.cpp:456: FAILED:
 {Unknown expression after the reported line}
due to a fatal error condition:
 SIGABRT - Abort (abnormal termination) signal

===============================================================================
test cases:  1 |  0 passed | 1 failed
assertions: 81 | 80 passed | 1 failed


   Start 3: rnn_tests
3/6 Test #3: rnn_tests ........................   Passed    0.79 sec
   Start 4: attention_tests
4/6 Test #4: attention_tests ..................   Passed    0.69 sec
   Start 5: fastopt_tests
5/6 Test #5: fastopt_tests ....................   Passed    0.02 sec
   Start 6: utils_tests
6/6 Test #6: utils_tests ......................   Passed    0.02 sec

83% tests passed, 1 tests failed out of 6

Total Test time (real) =   3.03 sec

The following tests FAILED:
         2 - operator_tests (Child aborted)
Errors while running CTest

TL;DR: This PR doesn't introduce any additional regression test failures to current master.

@ugermann @kpu Please review the PR as all the changes that you requested are in. I would like to merge it asap.

[jerinphilip]

TL;DR: This PR doesn't introduce any additional regression test failures to current master.

It does not strictly suffice here. Plenty of (CMake) switches in this branch has previously led to failure in linkages (quantizer.cpp for example), from bergamot-translator switches. Someone trying to build shouldn't be able to mess with switches and get a non-working build. Normally Makefiles for separate architectures (Makefile, Makefile.mingw) etc are supposed to be loaded or unloaded as a whole, but this PR makes it goes through a lot many switches making it painful to review. I would personally be scared to branch off browsermt/marian-dev for development if this gets merged in (due to the sheer complexity it induces on the build files which will be tricky to edit after).

Your use case is only target WASM architecture (which you're testing and reporting), but the build logic change seems to generalize many other combinations - decoder only, no exceptions, etc etc.

I would like to merge it asap.

For all complaints of big PR elsewhere with even tinier set of changes, this is quite interesting.

[abhi-agg]

TL;DR: This PR doesn't introduce any additional regression test failures to current master.

It does not strictly suffice here. Plenty of (CMake) switches in this branch has previously led to failure in linkages (quantizer.cpp for example), from bergamot-translator switches. Someone trying to build shouldn't be able to mess with switches and get a non-working build. Normally Makefiles for separate architectures (Makefile, Makefile.mingw) etc are supposed to be loaded or unloaded as a whole, but this PR makes it goes through a lot many switches making it painful to review. I would personally be scared to branch off browsermt/marian-dev for development if this gets merged in (due to the sheer complexity it induces on the build files which will be tricky to edit after).

Your use case is only target WASM architecture (which you're testing and reporting), but the build logic change seems to generalize many other combinations - decoder only, no exceptions, etc etc.

@jerinphilip The recent commits in this PR addresses your concerns 👍

For all complaints of big PR elsewhere with even tinier set of changes, this is quite interesting.

The changes in this PR has been a result of the wasm exploration that started 5 months ago. I had no option but to create a single PR as the technical debt is too much. Additionally, this PR doesn't add implementation in source files (as happened "elsewhere"). It mostly adds bunch of pre-processor directives (which are easier to review) and some cmake changes.

PS: I don't like this PR myself but I have no option here 👍

[kpu]

nit could have been outside the if statement

[kpu]

Fun fact: now that @XapaJIaMnu has binary working, you can get rid of cnpy entirely.

[abhi-agg]

Yes, but for compilation we will still have to keep this ifdef. Right?

[abhi-agg]

@kpu I made the final changes addressing all your concerns. A quick final review and approval will be much appreciated 👍

[abhi-agg]

Before submitting pull requests please make sure that

* all changes are properly documented in CHANGELOG.md

* all regression tests pass

* all files with changes have clutter / commented-out code removed (or contain a convincing explanation why the commented-out code should be kept around

Without at least these three conditions having been met, chances of approval are very slim.

@ugermann Could you please approve the PR as I already addressed all your review comments?

[ugermann]

I'd suggest a squashed merge.

[abhi-agg]

This PR is also connected to the #5