Improve documentation on pyChapel return types

[buddha314]

From the Misc. Page we can see that primitives can be returned from a Chapel routine into Python. This isn't explicitly stated. Is there a way to return non-primitives? How would I return arrays or sparse arrays, for instance? What about JSON or blobs?

This could be useful for sending data back and forth. I'm sure there are reasonable limits I don't understand.

[lydia-duncan]

Only certain types are supported across the Python-Chapel divide. In the current implementation, we've had to explicitly mention certain types, so you won't see every type Chapel or Python could support. While arrays can be passed to Chapel functions, they cannot currently be returned from them (but you will see modifications made to arrays you pass in to Chapel functions)

[lydia-duncan]

It is an action item to support arrays of more than just the one element type, to return them, and to support arbitrary structures defined within Chapel and Python

[buddha314]

In the example below, Chapel is modifying and np.ndarray.

@Chapel()
def quant(date=np.ndarray, bid=np.ndarray, ask=np.ndarray, voodoo=np.ndarray):
    """
    var future = 200;
    voodoo = ask-bid;
    for i in 1..future {
        voodoo += (ask-bid + ask-bid  + ask-bid + ask-bid
                   +ask-bid + ask-bid + ask-bid + ask-bid
                  ) / 8;
    }
    voodoo = voodoo / future;
    """
    return None

Is the Python view of the "voodoo" variable modified?

[buddha314]

Also, I'm trying to send an array into a routine defined in an sfile. I need an example of the syntax. This failed:

  export
  proc flip_bit(v: [?D] int) {
    v[0] = -v[0];
    return v;
  }

with error

(pychapel) buddha314@notes:~/pychapel/tmp$ python call.py
/tmp/temp-lnjLSb.chpl:10: In function 'flip_bit':
/tmp/temp-lnjLSb.chpl:11: internal error: CAL0126 chpl Version 1.16.0 pre-release (c3663a54f0)

Internal errors indicate a bug in the Chapel compiler ("It's us, not you"),
and we're sorry for the hassle.  We would appreciate your reporting this bug --
please see http://chapel.cray.com/bugs.html for instructions.  In the meantime,
the filename + line number above may be useful in working around the issue.

g++: error: /tmp/tmpEB8cif.a: No such file or directory
Traceback (most recent call last):
  File "call.py", line 12, in <module>
    shout_out()
  File "/home/buddha314/.virtualenvs/pychapel/local/lib/python2.7/site-packages/pych/extern.py", line 212, in wrapped_f
    raise MaterializationError(self)
pych.exceptions.MaterializationError: Failed materializing ({'anames': [],
 'atypes': [],
 'bfile': None,
 'dec_fp': '/home/buddha314/pychapel/tmp/response.chpl',
 'dec_hs': '7ecfac2d168f3423f7104eeb38057ac3',
 'dec_ts': 1501804367,
 'doc': None,
 'efunc': None,
 'ename': 'shout_out',
 'lib': 'sfile-chapel-7ecfac2d168f3423f7104eeb38057ac3-1501804367.so',
 'module_dirs': [],
 'pfunc': <function shout_out at 0x7fe326f428c0>,
 'pname': 'shout_out',
 'rtype': None,
 'sfile': '/home/buddha314/pychapel/tmp/response.chpl',
 'slang': 'chapel',
 'source': None}).

[lydia-duncan]

re: the voodoo variable - I believe Python will see any changes made to it, and since it is an argument to the function, will be able to work with the actual used after the call has completed. But you should double check that to be certain.

Hmm, I don't think we have any tests of sending an array to a function defined in an sfile. I suspect that the ?D domain is causing issues, since Chapel arguments to functions that Python calls must be fully concrete and ?D or no domain at all makes the argument generic. That really shouldn't be an internal error (and is a problem on the Chapel side). Perhaps try specifying the domain exactly for the call?

[lydia-duncan]

And if you wouldn't mind, please open an issue on the Chapel repository about exporting a function with a generic array argument causing an internal error instead of a user facing one :)

[lydia-duncan]

Maybe we will be able to allow the exporting of functions with generic arguments when our separate compilation story gets implemented

[buddha314]

Sorry, just saw this, should I still open that ticket? I'm not sure I understand it, but I'm happy to parrot your words.

[lydia-duncan]

Thanks!

[buddha314]

Do you have a basic example of sending an integer between python and chapel? Something like

import os
from pych.extern import Chapel

currentloc = os.path.dirname(os.path.realpath(__file__))
@Chapel(sfile=os.path.join(currentloc + '/response.chpl'))
def flip_bit(v=int):
    return int

if __name__=="__main__":
    u = 71
    w = flip_bit(u)
    print(w)

And

module Response{
  export
  proc flip_bit(v: int) :int {
    v = -v;
    return v;
  }
}

This example isn't working, but I can't tell from the docs how to send in v

Thanks!

[lydia-duncan]

The interaction of these two files seems to work for me: python file and chapel file

I suspect the difference has something to do with wrapping your chapel code in a module, since you would normally need a use statement to access its contents (due to the name of the module being different from the name of the file - Chapel is case sensitive). You could either remove the module declaration or make it lower case (i.e. module response)

[lydia-duncan]

Note that this might interplay with issue #10

[buddha314]

Changing to lower case module response yields:

/tmp/temp-d4UnJ0.chpl:4: In function 'flip_bit':
/tmp/temp-d4UnJ0.chpl:5: error: illegal lvalue in assignment
g++: error: /tmp/tmpbUDg_C.a: No such file or directory
Traceback (most recent call last):
  File "call.py", line 15, in <module>
    w = flip_bit(u)
  File "/home/buddha314/.virtualenvs/pychapel/local/lib/python2.7/site-packages/pych/extern.py", line 212, in wrapped_f
    raise MaterializationError(self)
pych.exceptions.MaterializationError: Failed materializing ({'anames': ['v'],
 'atypes': [<type 'int'>],
 'bfile': None,
 'dec_fp': '/home/buddha314/pychapel/tmp/response.chpl',
 'dec_hs': '7ecfac2d168f3423f7104eeb38057ac3',
 'dec_ts': 1502208167,
 'doc': None,
 'efunc': None,
 'ename': 'flip_bit',
 'lib': 'sfile-chapel-7ecfac2d168f3423f7104eeb38057ac3-1502208167.so',
 'module_dirs': [],
 'pfunc': <function flip_bit at 0x7fc042d4a938>,
 'pname': 'flip_bit',
 'rtype': <type 'int'>,
 'sfile': '/home/buddha314/pychapel/tmp/response.chpl',
 'slang': 'chapel',
 'source': None}).

Removing declaration of module so response.chpl is now

  export
  proc flip_bit(v: int) :int {
    v = -v;
    return v;
  }

Brings me this when I do python call.py

/tmp/temp-fb2D85.chpl:2: In function 'flip_bit':
/tmp/temp-fb2D85.chpl:3: error: illegal lvalue in assignment
g++: error: /tmp/tmpbJWVBW.a: No such file or directory
Traceback (most recent call last):
  File "call.py", line 15, in <module>
    w = flip_bit(u)
  File "/home/buddha314/.virtualenvs/pychapel/local/lib/python2.7/site-packages/pych/extern.py", line 212, in wrapped_f
    raise MaterializationError(self)
pych.exceptions.MaterializationError: Failed materializing ({'anames': ['v'],
 'atypes': [<type 'int'>],
 'bfile': None,
 'dec_fp': '/home/buddha314/pychapel/tmp/response.chpl',
 'dec_hs': '7ecfac2d168f3423f7104eeb38057ac3',
 'dec_ts': 1502208246,
 'doc': None,
 'efunc': None,
 'ename': 'flip_bit',
 'lib': 'sfile-chapel-7ecfac2d168f3423f7104eeb38057ac3-1502208246.so',
 'module_dirs': [],
 'pfunc': <function flip_bit at 0x7f0a18ab0938>,
 'pname': 'flip_bit',
 'rtype': <type 'int'>,
 'sfile': '/home/buddha314/pychapel/tmp/response.chpl',
 'slang': 'chapel',
 'source': None}).

[lydia-duncan]

Oh, right. That's because you're trying to assign to the argument, when the argument's default intent is const in (meaning it can't be overwritten).

[lydia-duncan]

Should we be having this conversation on stackoverflow?

[buddha314]

Hey, that's fine with me! Okay, I'll start one and post it here. Good idea!

[buddha314]

HIT IT! Note to future readers, solution is on SO