[GPU] LSTMSequence and LSTMCell optimization #26767

michal-miotk · 2024-09-24T14:09:12Z

Details:

creating simple primitive for lstm_sequence to be faster than previous approach using many primitives
using oneDNN
based on commit c99ddc0 from 25732

Tickets:

146601

commit 232d272f11fbe65e82fa9787260a8b9d34b57d20 Author: michal-miotk <[email protected]> Date: Mon Jul 29 11:17:47 2024 +0000 wip commit e642ca3 Author: michal-miotk <[email protected]> Date: Sun Jul 28 22:08:24 2024 +0000 wip commit c6b74d3 Author: michal-miotk <[email protected]> Date: Fri Jul 26 14:10:26 2024 +0000 wip commit 0451429 Author: michal-miotk <[email protected]> Date: Thu Jul 25 20:35:11 2024 +0000 wip3

commit 1164592 Author: michal-miotk <[email protected]> Date: Tue Aug 6 09:25:45 2024 +0000 wip commit 8b2c049 Author: michal-miotk <[email protected]> Date: Tue Aug 6 09:24:02 2024 +0000 wip commit 886b412 Author: michal-miotk <[email protected]> Date: Mon Aug 5 14:59:14 2024 +0000 wip commit 08fb207 Author: michal-miotk <[email protected]> Date: Sun Aug 4 20:21:38 2024 +0000 wip, errors on half commit 125884d Author: michal-miotk <[email protected]> Date: Sat Aug 3 23:59:58 2024 +0000 wip commit af4f209 Author: michal-miotk <[email protected]> Date: Fri Aug 2 17:58:38 2024 +0000 wip commit 12626fc Author: michal-miotk <[email protected]> Date: Fri Aug 2 10:52:15 2024 +0000 wip commit dfdd052 Author: michal-miotk <[email protected]> Date: Thu Aug 1 15:38:41 2024 +0000 wip commit 54ee912 Author: michal-miotk <[email protected]> Date: Thu Aug 1 11:01:55 2024 +0000 only bfyx layout commit 240fe4a Author: michal-miotk <[email protected]> Date: Thu Aug 1 10:34:45 2024 +0000 two outputs from prim commit bc775be Author: michal-miotk <[email protected]> Date: Wed Jul 31 22:13:14 2024 +0000 wip commit d1cfd60 Author: michal-miotk <[email protected]> Date: Wed Jul 31 22:07:06 2024 +0000 wip commit 7d18884 Author: michal-miotk <[email protected]> Date: Wed Jul 31 19:19:04 2024 +0000 begin of handling reverse commit 39f64af Author: michal-miotk <[email protected]> Date: Wed Jul 31 15:37:06 2024 +0000 betterbetter commit 67b3c9a Author: michal-miotk <[email protected]> Date: Wed Jul 31 13:12:39 2024 +0000 better commit 6ded5aa Author: michal-miotk <[email protected]> Date: Wed Jul 31 10:12:31 2024 +0000 wip commit 1ccdacc Author: michal-miotk <[email protected]> Date: Tue Jul 30 23:07:21 2024 +0000 wip commit ab1307c Author: michal-miotk <[email protected]> Date: Tue Jul 30 22:00:50 2024 +0000 test passed commit bc65969 Author: michal-miotk <[email protected]> Date: Tue Jul 30 15:37:20 2024 +0000 wip commit 03cbf57 Author: michal-miotk <[email protected]> Date: Tue Jul 30 15:15:06 2024 +0000 only 2 outputs commit fd5f3dc Author: michal-miotk <[email protected]> Date: Tue Jul 30 14:47:12 2024 +0000 wip commit 939d23c Author: michal-miotk <[email protected]> Date: Tue Jul 30 11:34:56 2024 +0000 wip commit 2bb561f Author: michal-miotk <[email protected]> Date: Tue Jul 30 09:28:03 2024 +0000 added to binary buffer commit 1ef83ff Author: michal-miotk <[email protected]> Date: Mon Jul 29 22:30:57 2024 +0000 not works

…tion only in gpu plugin

This reverts commit 2b6ad65.

This reverts commit 75d0b67.

p-durandin · 2024-11-07T08:02:12Z

build_jenkins

src/plugins/intel_gpu/src/graph/lstm_cell.cpp

src/plugins/intel_gpu/src/graph/lstm_seq.cpp

src/plugins/intel_gpu/src/graph/impls/onednn/lstm_seq_onednn.cpp

src/plugins/intel_gpu/src/graph/program.cpp

src/plugins/intel_gpu/src/runtime/execution_config.cpp

src/plugins/intel_gpu/src/graph/graph_optimizer/post_optimize_weights.cpp

Signed-off-by: Michal Miotk <[email protected]>

vladimir-paramuzov · 2024-10-10T08:57:23Z

src/plugins/intel_gpu/src/graph/graph_optimizer/post_optimize_lstm_weights.cpp

+
+namespace cldnn {
+
+post_optimize_lstm_weights::post_optimize_lstm_weights(reorder_factory& rf_ref)


Please make it a part of common post_optimize_weights pass

vladimir-paramuzov · 2024-10-10T08:57:37Z

src/plugins/intel_gpu/src/graph/graph_optimizer/post_optimize_lstm_weights.cpp

+// function which prepares given primitive for weights optimization
+template<typename T>
+void post_optimize_lstm_weights::optimize_lstm_weights(T& node, program& p) {
+    //auto offsets = get_weights_bias_offset(node);


vladimir-paramuzov · 2024-10-10T08:59:37Z

src/plugins/intel_gpu/src/graph/layout_optimizer.cpp

+    }
+}
+
+void reorder_factory::get_weights_split(primitive_id input_id,


IMO, reorder_factory class doesn't look like a good place for such kind of transformation. Could it be a part of post_optimize_weights pass?

vladimir-paramuzov · 2024-11-13T04:44:12Z

src/plugins/intel_gpu/include/intel_gpu/graph/program.hpp

@@ -267,6 +267,9 @@ struct program {
                             const ExecutionConfig& config,
                             std::shared_ptr<ov::threading::IStreamsExecutor> task_executor,
                             bool is_internal);
+
+    static bool has_lstm(topology const& topology);


Looks like it's not used.

vladimir-paramuzov · 2024-11-13T04:49:53Z

src/plugins/intel_gpu/include/intel_gpu/graph/topology.hpp

@@ -18,6 +18,9 @@ typedef std::map<primitive_id, std::shared_ptr<primitive>> topology_map;
 struct topology {
 public:
    using ptr = std::shared_ptr<topology>;
+
+    bool lstm_present = false;


Looks like a hack to be honest. As I can see, the only usage of that is setting use_onednn = true if lstm_present, so I think you can just iterate over primitives somewhere in program::init_graph() to check if rnn primitive is present

vladimir-paramuzov · 2024-11-13T05:47:55Z

src/plugins/intel_gpu/src/graph/lstm_seq.cpp

+
+template<typename ShapeType>
+std::vector<layout> lstm_seq_inst::calc_output_layouts(lstm_seq_node const& node, kernel_impl_params const& impl_param) {
+    auto desc = impl_param.typed_desc<lstm_seq>();


vladimir-paramuzov · 2024-11-13T05:48:18Z

src/plugins/intel_gpu/src/graph/lstm_seq.cpp

+//
+#include "lstm_seq_inst.h"
+#include "primitive_type_base.h"
+#include "intel_gpu/runtime/error_handler.hpp"


Please avoid using of this header

vladimir-paramuzov · 2024-11-13T05:49:24Z

src/plugins/intel_gpu/src/graph/lstm_seq.cpp

+        second_out_fmt = node.get_preferred_output_fmt(1);
+        third_out_fmt = node.get_preferred_output_fmt(2);
+    }
+    return {cldnn::layout{ShapeType{lstm_batch_size, 1, lstm_seq_length, lstm_hidden_size}, input_layout_x.data_type, first_out_fmt}, \


IIRC, this is not correct shapes for bidirectional LSTM. Is should probably be num_directions or something like that

vladimir-paramuzov · 2024-11-13T05:50:35Z

src/plugins/intel_gpu/src/graph/program.cpp

+    if (_engine.get_device_info().supports_immad) {
+        _config.set_property(ov::intel_gpu::use_onednn(true));
+    }


I think this can be handled once in ExecutionConfig::apply_user_properties

vladimir-paramuzov · 2024-11-13T05:56:56Z

src/plugins/intel_gpu/src/kernel_selector/cl_kernels/lstm_cell_and_seq_bfyx.cl

+                        #ifdef SEQUENCE
+                            INPUT1_TYPE_VEC initial_block = READ_VEC(0, &initial_hidden_state[INPUT1_GET_INDEX(b, 0, j*VEC_SIZE, 0)]);
+                            INPUT4_TYPE_VEC r_block = READ_VEC(0, &R[INPUT4_GET_INDEX(0, weight_idx, j*VEC_SIZE, 0)]);
+                        #else
+                            INPUT1_TYPE_VEC initial_block = READ_VEC(0, &initial_hidden_state[INPUT1_GET_INDEX(b, j*VEC_SIZE, 0, 0)]);
+                            INPUT4_TYPE_VEC r_block = READ_VEC(0, &R[INPUT4_GET_INDEX(weight_idx, j*VEC_SIZE, 0, 0)]);


I wonder if that kind of code can be unified for cell and sequence? Looks like the only difference is indexing, so maybe you can do something like this:

#ifdef SEQUENCE #define GET_IN0_IDX(b, f, y) INPUT1_GET_INDEX(b, f, y, 0) #else #define GET_IN0_IDX(b, f, y) INPUT1_GET_INDEX(b, y, 0, 0) #endif INPUT1_TYPE_VEC initial_block = READ_VEC(0, &initial_hidden_state[GET_IN0_IDX(b, 0, j*VEC_SIZE, 0)]); ...

…g onednn on template Signed-off-by: Michal Miotk <[email protected]>

Signed-off-by: Michal Miotk <[email protected]>

michal-miotk added 30 commits July 18, 2024 11:36

compiles lstm_seq

9ce143a

more kernel args

027f991

bigger proper run chances

c191c58

19jul

d461e66

inference works

01fa2ac

in middle of implementation

1f017fd

problems with inputs get element in kernel

5787c7d

not compile

837db22

wipx

d4ce531

wip

19c268e

solved problem with too much inputs kernel

f5273bc

wip

d50b3be

more changes

63a8dfd

wip

f54ecc1

wip

3748a11

wip

fae772a

proper shape for 2 outputs

c00ff8a

cleaning

31fcb79

Merge branch 'master' into lstm2

4b16eef

updated to new primitive_base api, disabled lstm to tensor transforma…

dcad182

…tion only in gpu plugin

now it should compile on windows, changed kernel name

d6aeb54

deleted cell, deleted input_forget

9688f63

generic primitive

5003d47

fix compilation problem, smaller lws

5937b14

wip

8b31a91

wip, not resolved fail on dynamic

2ff5a7c

fixed failing dynamic test

2d9e5c6

change name cldnn::rnn -> cldnn::lstm_seq

702e941

michal-miotk added 17 commits October 28, 2024 21:15

only lstmseq decomposition

2b6ad65

little refactor

8e1d36c

micro refactor - deleted one line

bd81512

Merge branch 'master' into lstm_with_onednn

e4fffa5

deleting skip of deconvolution_random_test

aea04bd

Revert "only lstmseq decomposition"

7af4e1e

This reverts commit 2b6ad65.

updated onednn

75d0b67

oooq only for lstmSeq

acd6369

Revert "updated onednn"

0430ee3

This reverts commit 75d0b67.

Merge branch 'master' into lstm_with_onednn

122ad96

fix caching tests

a54f49a

decomposition for seq_len == 1

79ef461

refactor rnn

31313c9

wip

d3ec29b

Merge branch 'master' into lstm_with_onednn

09b9283

minor refactor

bde6fb6

in_order que when support immad in tests

fcee4af

sshlyapn reviewed Nov 8, 2024

View reviewed changes

michal-miotk added 7 commits November 10, 2024 23:50

reordering output done in query

320c0d9

Signed-off-by: Michal Miotk <[email protected]>

flags for onednn, added missing loader and saver statement

c7ccf49

Signed-off-by: Michal Miotk <[email protected]>

cleaning

421eb89

Signed-off-by: Michal Miotk <[email protected]>

cleaning

c6e160f

Signed-off-by: Michal Miotk <[email protected]>

fix error, added more check if device supports immad

0adaa9b

deleting unnecessary variables

fee5623

Signed-off-by: Michal Miotk <[email protected]>

fix PrimitiveTypeTest

3482f33

vladimir-paramuzov reviewed Nov 13, 2024

View reviewed changes

michal-miotk added 3 commits November 14, 2024 14:19

move functions from reorder factory to post_optimize_weights, enablin…

e174f1a

…g onednn on template Signed-off-by: Michal Miotk <[email protected]>

updated indexing in kernel

06f394a

Signed-off-by: Michal Miotk <[email protected]>

deleted redundant check of immad

483e637

Signed-off-by: Michal Miotk <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[GPU] LSTMSequence and LSTMCell optimization #26767

[GPU] LSTMSequence and LSTMCell optimization #26767

michal-miotk commented Sep 24, 2024 •

edited

Loading

p-durandin commented Nov 7, 2024

vladimir-paramuzov Oct 10, 2024

michal-miotk Nov 13, 2024

vladimir-paramuzov Oct 10, 2024

michal-miotk Nov 13, 2024

vladimir-paramuzov Oct 10, 2024

michal-miotk Nov 15, 2024

vladimir-paramuzov Nov 13, 2024

michal-miotk Nov 15, 2024

vladimir-paramuzov Nov 13, 2024

michal-miotk Nov 15, 2024

vladimir-paramuzov Nov 13, 2024

michal-miotk Nov 14, 2024

vladimir-paramuzov Nov 13, 2024

michal-miotk Nov 15, 2024

vladimir-paramuzov Nov 13, 2024

vladimir-paramuzov Nov 13, 2024

michal-miotk Nov 15, 2024

vladimir-paramuzov Nov 13, 2024

michal-miotk Nov 15, 2024


		namespace cldnn {

		post_optimize_lstm_weights::post_optimize_lstm_weights(reorder_factory& rf_ref)

[GPU] LSTMSequence and LSTMCell optimization #26767

Are you sure you want to change the base?

[GPU] LSTMSequence and LSTMCell optimization #26767

Conversation

michal-miotk commented Sep 24, 2024 • edited Loading

Details:

Tickets:

p-durandin commented Nov 7, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

michal-miotk commented Sep 24, 2024 •

edited

Loading