different output at exported model

[kspook]

@emedvedev, in my case, it doesn't work when I run exported model including Korean recognition. T.T
In the exported model, Korean return good results, but number and English Alphabet showed different result.

Test & predict show the good results .
Do you have solution?

[emedvedev]

There's not much to go on, so could you maybe provide more information? I may not be able to help, since it's your own fork with significant changes, but maybe someone else would take a look.

Does your Korean fork include the latest master changes? What's the correct fork/branch to look at? How exactly are you exporting the model? Do you have different results for the exact same image with predict and with the same model on the same stage of training when exported? What are the exact results, ideally with full logs?

Unless your issue includes all that, it would be very hard to see where the issue comes from.

[kspook]

thank you for your response.

my information as below;
master version : https://github.com/kspook/aocrKR.git
checkpoint: https://drive.google.com/open?id=1tPHoE1gK-AUjYBqbSCFbSc6lRnBh7WI5
exported model : https://drive.google.com/open?id=1j--Jb2XlyZXHFwTtWR3NBx2EsDKgyIZ-
(https://drive.google.com/open?id=1QPoZQZuLbLmdLq1LGK9saQ4eKMvkOVFA
for better result)
test : https://github.com/kspook/aocrKR/tree/master/dataset
predict : https://github.com/kspook/aocrKR/tree/master/dataset/test-img3/
test curl : https://github.com/kspook/aocrKR/tree/master/aocr36

1. logic for Korean as below : Korean training (tensorflow serving included) #126 (comment)
   I tried to train Korean words again.
   For example, '신','한' was converted '853863'.
   that is, 85('신')+3+86('한')+3
   FYI, I omitted 1,2,3 for index of all input.
   Later, I put serial index (0-40) for training of characters in the word
   If I put 36+3 as index of '신', 37+3 as '한' --> [1, 39, 40, 2 ],

2. export file : I changed 'stdin' --> arguement.

test for 1.jpeg : in case curl, 1.jpeg showed different result.

(py37) kspook@ml004:~/aocrKR$ python aocr36/main.py predict data/test-img3/1.jpeg
2019-06-08 12:42:18.288443: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-06-08 12:42:18.308877: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2397220000 Hz
2019-06-08 12:42:18.309086: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x5637712d7920 executing computations on platform Host. Devices:
2019-06-08 12:42:18.309162: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): ,
2019-06-08 12:42:18,309 root INFO phase: predict
2019-06-08 12:42:18,310 root INFO model_dir: checkpoints
2019-06-08 12:42:18,314 root INFO load_model: True
2019-06-08 12:42:18,314 root INFO output_dir: results
2019-06-08 12:42:18,314 root INFO steps_per_checkpoint: 0
2019-06-08 12:42:18,314 root INFO batch_size: 1
2019-06-08 12:42:18,315 root INFO learning_rate: 1.000000
2019-06-08 12:42:18,315 root INFO reg_val: 0
2019-06-08 12:42:18,315 root INFO max_gradient_norm: 5.000000
2019-06-08 12:42:18,315 root INFO clip_gradients: True
2019-06-08 12:42:18,315 root INFO max_image_width 260.000000
2019-06-08 12:42:18,315 root INFO max_prediction_length 18.000000
2019-06-08 12:42:18,316 root INFO channels: 1
2019-06-08 12:42:18,316 root INFO target_embedding_size: 10.000000
2019-06-08 12:42:18,316 root INFO attn_num_hidden: 128
2019-06-08 12:42:18,316 root INFO attn_num_layers: 2
2019-06-08 12:42:18,316 root INFO visualize: False
filename, data/test-img3/1.jpeg
2019-06-08 12:42:24,658 root INFO Result: OK. 0.83 L

(py37) kspook@ml004:~/aocrKR$ python aocr36/main.py test data/testingk.tfrecords
2019-06-08 12:45:57.344504: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-06-08 12:45:57.351414: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2397220000 Hz
2019-06-08 12:45:57.351623: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x55e8a7637170 executing computations on platform Host. Devices:
2019-06-08 12:45:57.351705: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): ,
2019-06-08 12:45:57,352 root INFO phase: test
2019-06-08 12:45:57,352 root INFO model_dir: checkpoints
2019-06-08 12:45:57,353 root INFO load_model: True
2019-06-08 12:45:57,353 root INFO output_dir: results
2019-06-08 12:45:57,353 root INFO steps_per_checkpoint: 0
2019-06-08 12:45:57,353 root INFO batch_size: 1
2019-06-08 12:45:57,353 root INFO learning_rate: 1.000000
2019-06-08 12:45:57,354 root INFO reg_val: 0
2019-06-08 12:45:57,354 root INFO max_gradient_norm: 5.000000
2019-06-08 12:45:57,354 root INFO clip_gradients: True
2019-06-08 12:45:57,354 root INFO max_image_width 260.000000
2019-06-08 12:45:57,358 root INFO max_prediction_length 18.000000
2019-06-08 12:45:57,358 root INFO channels: 1
2019-06-08 12:45:57,358 root INFO target_embedding_size: 10.000000
2019-06-08 12:45:57,358 root INFO attn_num_hidden: 128
2019-06-08 12:45:57,359 root INFO attn_num_layers: 2
2019-06-08 12:45:57,359 root INFO visualize: False

n,c, 5 5
n,c, 55 5
c, c_idx, n, label, label[0], label[1] 3 55 55 (4, '하') 4 하
c, c_idx, n, label, label[0], label[1] 3 55 55 (5, '나') 5 나
c, c_idx, n, label, label[0], label[1] 3 55 55 (6, '우') 6 우
c, c_idx, n, label, label[0], label[1] 3 55 55 (7, '리') 7 리
c, c_idx, n, label, label[0], label[1] 3 55 55 (8, '국') 8 국
c, c_idx, n, label, label[0], label[1] 3 55 55 (9, '민') 9 민
c, c_idx, n, label, label[0], label[1] 3 55 55 (44, '신') 44 신
c, c_idx, n, label, label[0], label[1] 3 55 55 (45, '한') 45 한
c, c_idx, n, label, label[0], label[1] 3 55 55 (46, '기') 46 기
c, c_idx, n, label, label[0], label[1] 3 55 55 (47, '업') 47 업
c, c_idx, n, label, label[0], label[1] 3 55 55 (48, '농') 48 농
c, c_idx, n, label, label[0], label[1] 3 55 55 (49, '협') 49 협
c, c_idx, n, label, label[0], label[1] 3 55 55 (54, '0') 54 0
c, c_idx, n, label, label[0], label[1] 3 55 55 (55, '1') 55 1
1
c, c_idx, n, label, label[0], label[1] 3 55 55 (56, '2') 56 2
c, c_idx, n, label, label[0], label[1] 3 55 55 (57, '3') 57 3
c, c_idx, n, label, label[0], label[1] 3 55 55 (58, '4') 58 4
c, c_idx, n, label, label[0], label[1] 3 55 55 (59, '5') 59 5
c, c_idx, n, label, label[0], label[1] 3 55 55 (64, '6') 64 6
c, c_idx, n, label, label[0], label[1] 3 55 55 (65, '7') 65 7
c, c_idx, n, label, label[0], label[1] 3 55 55 (66, '8') 66 8
c, c_idx, n, label, label[0], label[1] 3 55 55 (67, '9') 67 9
c, c_idx, n, label, label[0], label[1] 3 55 55 (68, 'A') 68 A
c, c_idx, n, label, label[0], label[1] 3 55 55 (69, 'B') 69 B
c, c_idx, n, label, label[0], label[1] 3 55 55 (74, 'C') 74 C
c, c_idx, n, label, label[0], label[1] 3 55 55 (75, 'D') 75 D
c, c_idx, n, label, label[0], label[1] 3 55 55 (76, 'E') 76 E
c, c_idx, n, label, label[0], label[1] 3 55 55 (77, 'F') 77 F
c, c_idx, n, label, label[0], label[1] 3 55 55 (78, 'G') 78 G
c, c_idx, n, label, label[0], label[1] 3 55 55 (79, 'H') 79 H
c, c_idx, n, label, label[0], label[1] 3 55 55 (84, 'I') 84 I
c, c_idx, n, label, label[0], label[1] 3 55 55 (85, 'J') 85 J
c, c_idx, n, label, label[0], label[1] 3 55 55 (86, 'K') 86 K
c, c_idx, n, label, label[0], label[1] 3 55 55 (87, 'L') 87 L
c, c_idx, n, label, label[0], label[1] 3 55 55 (88, 'M') 88 M
c, c_idx, n, label, label[0], label[1] 3 55 55 (89, 'N') 89 N
c, c_idx, n, label, label[0], label[1] 3 55 55 (94, 'O') 94 O
c, c_idx, n, label, label[0], label[1] 3 55 55 (95, 'P') 95 P
c, c_idx, n, label, label[0], label[1] 3 55 55 (96, 'Q') 96 Q
c, c_idx, n, label, label[0], label[1] 3 55 55 (97, 'R') 97 R
c, c_idx, n, label, label[0], label[1] 3 55 55 (98, 'S') 98 S
c, c_idx, n, label, label[0], label[1] 3 55 55 (99, 'T') 99 T
c, c_idx, n, label, label[0], label[1] 3 55 55 (444, 'U') 444 U
c, c_idx, n, label, label[0], label[1] 3 55 55 (445, 'V') 445 V
c, c_idx, n, label, label[0], label[1] 3 55 55 (446, 'W') 446 W
c, c_idx, n, label, label[0], label[1] 3 55 55 (447, 'X') 447 X
c, c_idx, n, label, label[0], label[1] 3 55 55 (448, 'Y') 448 Y
c, c_idx, n, label, label[0], label[1] 3 55 55 (449, 'Z') 449 Z
c, c_idx, n, label, label[0], label[1] 3 55 55 (454, '-') 454 -
2019-06-08 12:46:02,786 root INFO Step 2 (0.095s). Accuracy: 0.00%, loss: 0.895335, perplexity: 2.44816, probability: 83.24% 0% (L vs 1) ./dataset/test-img3/1.jpeg

(base) kspook@ml004:~$ curl -X POST http://localhost:9002/v1/models/aocr:predict -H 'cache-control: no-cache' -H 'content-type: application/json -d '{
"signature_name": "serving_default",
"inputs":
{
"input": { "b64": "/9j/4AAQSkZJRgABAQAASABIAAD/4QBYRXhpZgAATU0AKgAAAAgAAgESAAMAAAABAAEAAIdpAAQAAAABAAAAJgAAAAAAA6ABAAMAAAABAAEAAKACAAQAAAABAAAAOaADAAQAAAABAAAAHAAAAAD/7QA4UGhvdG9zaG9wIDMuMAA4QklNBAQAAAAAAAA4QklNBCUAAAAAABDUHYzZjwCyBOmACZjs+EJ+/8AAEQgAHAA5AwEiAAIRAQMRAf/EAB8AAAEFAQEBAQEBAAAAAAAAAAABAgMEBQYHCAkKC//EALUQAAIBAwMCBAMFBQQEAAABfQECAwAEEQUSITFBBhNRYQcicRQygZGhCCNCscEVUtHwJDNicoIJChYXGBkaJSYnKCkqNDU2Nzg5OkNERUZHSElKU1RVVldYWVpjZGVmZ2hpanN0dXZ3eHl6g4SFhoeIiYqSk5SVlpeYmZqio6Slpqeoqaqys7S1tre4ubrCw8TFxsfIycrS09TV1tfY2drh4uPk5ebn6Onq8fLz9PX29/j5+v/EAB8BAAMBAQEBAQEBAQEAAAAAAAABAgMEBQYHCAkKC//EALURAAIBAgQEAwQHBQQEAAECdwABAgMRBAUhMQYSQVEHYXETIjKBCBRCkaGxwQkjM1LwFWJy0QoWJDThJfEXGBkaJicoKSo1Njc4OTpDREVGR0hJSlNUVVZXWFlaY2RlZmdoaWpzdHV2d3h5eoKDhIWGh4iJipKTlJWWl5iZmqKjpKWmp6ipqrKztLW2t7i5usLDxMXGx8jJytLT1NXW19jZ2uLj5OXm5+jp6vLz9PX29/j5+v/bAEMACAYGBwYFCAcHBwkJCAoMFA0MCwsMGRITDxQdGh8eHRocHCAkLicgIiwjHBwoNyksMDE0NDQfJzk9ODI8LjM0Mv/bAEMBCQkJDAsMGA0NGDIhHCEyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMv/dAAQABP/aAAwDAQACEQMRAD8AyYYwgG6rIIAHBqCLHFQeVLHP57SHPmhdmeApOK5bHKaKnjjjtzT1ljjLhnXKj5gD0rMaVxdtCsch2zA7ieADVqK1X7VMxX5y2Dk9QQD/AFq0hWLkc0fmhFbJKhvwNTxMHGQehway9LVUnl9D9xic/KKuW88e+ZVJyGLdO2KZJahZZYhIDgH9Kk2j+8KpWcu7chRhhmIJHByc1fwP8igR/9DKhIwNtI1oHuPM3tsyG8vtkd6bExAFW0Y4rmRx3AwD5gR94hjz3FXLqMJcOP4gFB+oAH9KhjG48+lWLg7rxye8hqhNkSRhQqqoAAwKsKVHtxjIHvUP3nGalQYFMBwOCD/+qpd/1pgUcU/NAH//2Q==" }
}
}'
{
"outputs": {
"probability": 0.356208,
"output": "Q"
}

[kspook]

In addition to test 1.jpeg the above posting, I will continue for 'shin'
it shows the same result.

1. for test
   2019-06-08 13:17:19,422 root INFO Step 6 (0.095s). Accuracy: 16.67%, loss: 3.153957, perplexity: 23.4286, probability: 91.60% 0% (T vs 신)

2. predict
   (py37) kspook@ml004:~/aocrKR$ python aocr36/main.py predict dataset/test-img3/shin.jpeg
   2019-06-08 13:18:12.320031: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
   2019-06-08 13:18:12.326873: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2397220000 Hz
   2019-06-08 13:18:12.327057: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x556c9f033c20 executing computations on platform Host. Devices:
   2019-06-08 13:18:12.327136: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): ,
   2019-06-08 13:18:12,327 root INFO phase: predict
   2019-06-08 13:18:12,328 root INFO model_dir: checkpoints
   2019-06-08 13:18:12,328 root INFO load_model: True
   2019-06-08 13:18:12,328 root INFO output_dir: results
   2019-06-08 13:18:12,328 root INFO steps_per_checkpoint: 0
   2019-06-08 13:18:12,328 root INFO batch_size: 1
   2019-06-08 13:18:12,328 root INFO learning_rate: 1.000000
   2019-06-08 13:18:12,329 root INFO reg_val: 0
   2019-06-08 13:18:12,329 root INFO max_gradient_norm: 5.000000
   2019-06-08 13:18:12,329 root INFO clip_gradients: True
   2019-06-08 13:18:12,329 root INFO max_image_width 260.000000
   2019-06-08 13:18:12,329 root INFO max_prediction_length 18.000000
   2019-06-08 13:18:12,329 root INFO channels: 1
   2019-06-08 13:18:12,329 root INFO target_embedding_size: 10.000000
   2019-06-08 13:18:12,330 root INFO attn_num_hidden: 128
   2019-06-08 13:18:12,332 root INFO attn_num_layers: 2
   2019-06-08 13:18:12,333 root INFO visualize: False
   filename, dataset/test-img3/shin.jpeg
   2019-06-08 13:18:18,207 root INFO Result: OK. 0.92 T

3. curl.
   (base) kspook@ml004:~$ curl -X POST \

http://localhost:9002/v1/models/aocr:predict
-H 'cache-control: no-cache'
-H 'content-type: application/json'
-d '{
"signature_name": "serving_default",
"inputs":
{
"input": { "b64": "/9j/4AAQSkZJRgABAQEAeAB4AAD/4QAiRXhpZgAATU0AKgAAAAgAAQESAAMAAAABAAEAAAAAAAD/2wBDAAIBAQIBAQICAgICAgICAwUDAwMDAwYEBAMFBwYHBwcGBwcICQsJCAgKCAcHCg0KCgsMDAwMBwkODw0MDgsMDAz/2wBDAQICAgMDAwYDAwYMCAcIDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAz/wAARCAA+AD4DASIAAhEBAxEB/8QAHwAAAQUBAQEBAQEAAAAAAAAAAAECAwQFBgcICQoL/8QAtRAAAgEDAwIEAwUFBAQAAAF9AQIDAAQRBRIhMUEGE1FhByJxFDKBkaEII0KxwRVS0fAkM2JyggkKFhcYGRolJicoKSo0NTY3ODk6Q0RFRkdISUpTVFVWV1hZWmNkZWZnaGlqc3R1dnd4eXqDhIWGh4iJipKTlJWWl5iZmqKjpKWmp6ipqrKztLW2t7i5usLDxMXGx8jJytLT1NXW19jZ2uHi4+Tl5ufo6erx8vP09fb3+Pn6/8QAHwEAAwEBAQEBAQEBAQAAAAAAAAECAwQFBgcICQoL/8QAtREAAgECBAQDBAcFBAQAAQJ3AAECAxEEBSExBhJBUQdhcRMiMoEIFEKRobHBCSMzUvAVYnLRChYkNOEl8RcYGRomJygpKjU2Nzg5OkNERUZHSElKU1RVVldYWVpjZGVmZ2hpanN0dXZ3eHl6goOEhYaHiImKkpOUlZaXmJmaoqOkpaanqKmqsrO0tba3uLm6wsPExcbHyMnK0tPU1dbX2Nna4uPk5ebn6Onq8vP09fb3+Pn6/9oADAMBAAIRAxEAPwD+f+iiigAoooxQAUUFSpwRgjgijHFABRRRQAUDrRQDg0Adb8FPg14i/aA+Lnh3wX4V0+TUvEXii+i0/T7ZASZZXYKOBk4HUnHABPQV+gf/AAV1/wCDdjxN/wAEsP2WPCvxKm8XWniq31DU00bW7ZYvK/s+4lR3gCf31IRgT2YCvVf+DPv9jCP4y/txeIfirqtu0mn/AAt04rYF0BU6hcgopOejLHvxz/FX6d/8F4f2APFX/BYrwjD8N/hd8aPAtjqPwxmfVNc8Ezyia6ur11xbvPIkha3AjdggdNhZm+bg4APzL/4NXP8Agmj8C/8AgodpXxmPxY8Lr4s1PwzLpgsYZrmSEQwz+cTIuwglt0RB9iPWvgv/AILBfAbwn+zH/wAFIfiv4H8DWbab4T0DVvs+m2rOX8iMRplcnk/Pu61+mX/Bqn8PfG37EP8AwV9+MXwR8faRLoXiq38MvbalZzSZ8t7eaCaKSMj5ZElimDowyGjcMDg5r87v+C7GoR6l/wAFZPjdLGwkVfEUsZKnIBAAI+vtQB8jUUUUAFKpww+vcUlFAH6If8Edf+C/viL/AIJF/CXxt4X0jwDo/jL/AISi7jvrN7y+NotlKqFWL7Iy0i9CFyPQEV84Rf8ABSr4waN+2f4g+PvhnxhqHhH4j+IdRm1Ca+02TCHzCB5DxvuSSEJgbZFZTjkGvn6t74Z+CG+JXxI0Dw7HcQ2ja5qNtpyzyHCQmaVI97ew3ZPsKAPvj/gnp+2b8XfhJ8SPiL+2J4n1zUPEmvxxJpc+rarKZbjUmLwiRVwynZGiwxbR+7HnRKUIIFfCvxm+KGtfHb4n+JPHHiCZrzWvFOpT6nfTY4M0zl2+gycAegr+qD9p/wD4Ia/s46Z+xX8Mfhv478YHwD8P/hmVv9Xnhu4LFvEs4Xa0k8j5bBeR24ycsuclQa/In/gvP+1r+yPP8CvCnwJ/Zb8K6TDY+F9VbUdT17T4CsdywQp5ZmfMlwxPzFi2B2FAH5U0UUUAFFFFAADUlrO9rcxyRyNFJGwZXVipQg5BBHIx61HRQB6b8cf2w/id+0rFZp498eeKPFUVjEIoYdQvnkjiUYACrnHYGvNSVx17cd6ZRQAUUUUAf//Z"
}
}
}'
{
"outputs": {
"probability": 0.915982,
"output": "T"
}

• Try this to check exact Korean recognition.
  https://drive.google.com/open?id=1QPoZQZuLbLmdLq1LGK9saQ4eKMvkOVFA

[emedvedev]

I'm going to leave this issue open for a while in case someone wants to jump in, but as I've mentioned, I don't have the capacity to debug forks, unfortunately. Hopefully someone will be able to help or you'll be able to figure it out :)

[kspook]

@emedvedev , for your information, the exported model can't return Korean recognition when I use 'stdin' in predict() at main.py to get image file.

Now I can't get number recognition well when I changed argument style to get image.

[kspook]

@emedvedev, I finally found out the difference b/ saved_model and script is bug of your code.

If I train with the basic settings, it returns the same result b/ the saved model and script.
But with the full-ascii training, it returns the different result.

code : https://github.com/kspook/aocr_fullascii.git
saved model/ checkpoints : https://drive.google.com/open?id=1FsRonnoMG9cS9npIbob5F2Pgu29JyAzi