Unicode read / write support

[doppio-wacharin]

This PR was created based on the discussion on the following issues => #70

Initially, I

• created 2 more keywords which are Write Unicode and Write Bear Unicode
• Modify the decoding when get text from the screen, from unicode_escape decoding to utf-8 decoding

[samuelpcabral]

Hi @doppio-wacharin I pull your branch and run the tests in PUB400.. the ones in "atest/mainframe.robot" but got a lot of failures, like that: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa7 in position 41: invalid start byte

log.zip

The problem is in py327.py line 473..
return cmd.data[0].decode("utf-8")
if return to: "unicode_escape" all the tests passes.. but I can´t be sure if the Thai chars works, because pub400 don´t have support for them... I try to use

Write Unicode Bare กขค

but the screen in pub400 show me like that:

[doppio-wacharin]

Hi @doppio-wacharin I pull your branch and run the tests in PUB400.. the ones in "atest/mainframe.robot" but got a lot of failures, like that: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa7 in position 41: invalid start byte

log.zip

The problem is in py327.py line 473.. return cmd.data[0].decode("utf-8") if return to: "unicode_escape" all the tests passes.. but I can´t be sure if the Thai chars works, because pub400 don´t have support for them... I try to use

Write Unicode Bare กขค

but the screen in pub400 show me like that:

I don't have pub400 but let me try to test it in the hercules, thx

[samuelpcabral]

You can test in PUB400 without create a user... here is the configurations: https://github.com/MarketSquare/Robot-Framework-Mainframe-3270-Library/wiki/Keyword-Tests

[doppio-wacharin]

I have created a simple script for this. and this seams working.

But now the atest seams broken. I'm checking whether it is because of my change or something else.

[doppio-wacharin]

The failure in the atest is my mistake T_T ( I tried to run all of this on macbook )
All pass on the ubuntu docker

I think all are good now.
Do you have anything else to be adjusted? or any concerns?
@samuelpcabral

[samuelpcabral]

@doppio-wacharin as soon I have some free time I take a look.. thanks

[samuelpcabral]

Sorry for the late @doppio-wacharin I just have time to see now.. I checkout your branch again.. but still having issues in reading with utf-8

Traceback (most recent call last): File "C:\Merkle\repositories\Mainframe-3270-Library\Mainframe3270\__init__.py", line 197, in run_keyword return DynamicCore.run_keyword(self, name, args, kwargs) File "C:\Python39\lib\site-packages\robotlibcore.py", line 173, in run_keyword return self.keywords[name](*args, **(kwargs or {})) File "C:\Merkle\repositories\Mainframe-3270-Library\Mainframe3270\keywords\assertions.py", line 33, in page_should_contain_string result = self.mf.search_string(txt, ignore_case) File "C:\Merkle\repositories\Mainframe-3270-Library\Mainframe3270\py3270.py", line 480, in search_string line = self.string_get(ypos + 1, 1, self.model_dimensions["columns"]) File "C:\Merkle\repositories\Mainframe-3270-Library\Mainframe3270\py3270.py", line 473, in string_get return cmd.data[0].decode("utf-8") UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa7 in position 41: invalid start byte

See my log.html from mainframe.robot execution...
log.zip

If you want to chat and try to see what could be happening.. you can send me a DM in robot framework slack: https://robotframework.slack.com

[samuelpcabral]

The mainframe.robot tests are not passing

[KUGDev]

@samuelpcabral the tests are failing due to this character

Running Open Connection ${HOST} extra_args=["-utf8"] instead does the thing, but I'm unsure whether the tests should be changed or should it be the code itself changed in some other way.
Theoretically, there may be other terminals that do not support UTF-8, but in that case the changes would have to be done much deeper. But from the POV of the existing solution, more than half of the code is already UTF-8 oriented. Also, it would make the most of sense to put "-utf8" arg as a default during establishing a connection in this case.

[samuelpcabral]

Hello @doppio-wacharin sorry for the late response.. we use utf-8 as you can see in py3270.py
I think should be some deep problem.. but unfortunately I don´t have access in a terminal with your language for debug...

let me know if I can do something to help.