Upstream to ob-julia?

[tecosaur]

Hello!

I've come across this because I just have tried using ob-julia and it's completely non-functional 🙁. For instance, it tries to use readcsv and writecsv which simply don't exist any more. It looks like it was last updated in 2014 😓.

There's an open call for org babel maintainers (Please help by becoming a maintainer for an Org Babel file), and the work you've done here seems like a great step forward!

I'd love it if the go-to experience of Julia in Org ... worked, so would you be willing to consider making this the new ob-julia?

[frederic-santos]

Hi!
Thanks for letting me know! Actually, I think that the same proposition should be made to the maintainer(s) of ob-julia-vterm, which is more stable right now. (Or, at least, I regard it as the most promising solution for a good Julia support in Org mode.)
ob-ess-julia works for a very basic use, but relies on ESS which has some long-known and unfixed bugs or glitches for Julia support. Thus, it inherits all those glitches and bugs, and unfortunately, I cannot fix them on my side. Julia does not seem to be the priority of the ESS team (which I fully understand), so that ob-ess-julia probably won't become as stable as ob-julia-vterm before a pretty long time.
Note that, however, I would be glad to be part of a team of maintainers for ob-julia (there is enough work for a whole team of maintainers!), but starting from ob-ess-julia might not be the best option available right now. :)

[tecosaur]

It sounds like what's "best", particularly long-term, isn't clear at this point. What is clear to me is that this would be a massive upgrade in functionality compared to the currently severely outdated ob-julia (which also uses ess currently, just with many more (breaking) issues that you say you have here).

I think it's great that you've made a package for this, but frankly I'm guessing the main thing people do when trying to use Julia with babel is

#+begin_src julia
"oh no, this doesn't work"
#+end_src

Even just replacing the current ob-julia with a snapshot of the state here would be a very, very welcome improvement IMO.

[frederic-santos]

I would be okay if the biggest blocking issue in ob-ess-julia could be solved. This issue (I think) is not acceptable for an "official" ob backend. If I find a way to solve it, I think that ob-ess-julia will be almost acceptable for a reliable use. I'll try to fix that soon.

But really, you should also give a try to ob-julia-vterm ;)

[tecosaur]

Cool! The vterm approach looks interesting, but as the current approach (and what others use, like R) is based on ess while this package seems like a straight upgrade, there would be a bunch of other hoops that would make vterm less easy to swallow as part of Org - for example requiring dynamic module support (which isn't supported by the minimum Emacs version for Org. There's an argument this should be bumped, but that's another matter).

For an example of how broken the current ob-julia is, 1+1 in a session block currently doesn't work 😞.

[frederic-santos]

Hi again!
Just two little news:

1. You might also want to give a look at https://git.nixo.xyz/nixo/ob-julia, which has advanced features.
2. The paths I try to explore to overcome the ESS limitations seem to face... Julia limitations. 😄 That's kind of my worst programming experience so far: I have the strange feeling that all the necessary infrastructure is buggy, illogical or overcomplicated in some way (really, writing an ob-R.el is so much easier than an ob-julia.el, that's crazy). That's really frustrating, I must admit (and I'm too much of a beginner in both Julia and Elisp to overcome this in an elegant way). That's why I think ob-julia-vterm, starting almost from scratch, is probably the best option. nixo's ob-julia also relies on ESS, and in some way, he did manage to overcome most of the limitations I struggle with thanks to a virtuosity in the Julia side I clearly don't have ;) I would really like to help and contribute, but I feel being the only one maintainer of such a project is too difficult for me at this stage.

[tecosaur]

I hadn't seen https://git.nixo.xyz/nixo/ob-julia before. That looks interesting, thanks for pointing it out.

Perhaps we can pull some of nixo's innovations into this package? I'm happy to try to have a look at the Julia/Elisp side of things since I'm quite familiar with Elisp and while I'm currently new to Julia I'm going to be using it a lot. I am rather busy though, so that may not be able to happen soon.

[frederic-santos]

I hadn't seen https://git.nixo.xyz/nixo/ob-julia before. That looks interesting

It is! ;) Nixo used to collaborate with the original author of ob-julia, so that his solution is quite stable right now (although it does not seem to be under active development recently). But it's a strong candidate for an official ob-julia.

Also, many thanks for your comments and enthusiasm ;)

[tecosaur]

I've emailed Nicolò with a link to this issue, asking if he might be interested in collaborating on Org's ob-julia 🙂

[nico202]

Hello everybody (I'm nixo)
@tecosaur thanks for writing me

I'd really like to collaborate on ob-julia and finally help providing good upstream support!

I've not checked out ob-ess-julia nor ob-julia-vterm yet, so I don't know what's the "state of the art" yet. I'll check them out, read open issues to be sure we all are on the same page :) Are there "only" those three (including mine) or are there other implementations that might be worth checking out?

[frederic-santos]

Hi Nicolò!
There are other solutions (based on jupyter kernels or julia-snail). They are good on their own, but their heavy framework and dependencies do not seem to be a good idea for a babel support.

As concerns ob-ess-julia, it works for many basic aspects, but it is inferior to your ob-julia for two main reasons:

1. there is no async evaluation
2. I struggle with a blocking issue (that is, essentially, that org-babel-comint-with-output function cannot be used for :results output, as it is used in ob-R.el for example, because there is no busy detection of inferior Julia processes by ESS). Thus, I also tried to just capture the output of code block eval with org-babel-comint-eval-invisibly-and-wait-for-file, write them into temp files, and then insert the contents of this temp file as results, but once again there are too many complications for an early beginner in Julia, so it seems to be another dead end. I'm basically stuck with this issue, at least, as long as busy detection is not allowed by ESS. On your side, you have overcome this same issue by doing some impressive intricate work I don't fully understand :).

I may try to help by working with you for some aspects (documentation, minor code improvements, etc.), but ob-ess-julia is way behind your ob-julia at this point. ;)

And, finally, ob-julia-vterm works just fine: I couldn't find any major issue with it.

[tecosaur]

Thank you @nico202 and @frederic-santos for being interested in this! I'd love to see great support for Julia in Org 😃, and it's great to see we share this desire.

I'm not sure what the best approach would be, between:

1. Brushing up ob-ess-julia and then submitting it for upstreaming
2. Brushing up Nicolò's ob-julia and submitting it for upstreaming
3. Trying to use what's been learned in ob-ess-julia and ob-julia to make a third version for upstreaming
4. Something else

ob-julia-vterm seems to work well, the major caveat with that as I see it is what I mentioned earlier,

while this package seems like a straight upgrade [to Org's ob-julia], there would be a bunch of other hoops that would make vterm less easy to swallow as part of Org - for example requiring dynamic module support (which isn't supported by the minimum Emacs version for Org. There's an argument this should be bumped, but that's another matter).

Perhaps (2) or (3) may be the best bet? What do you think?

[frederic-santos]

Perhaps (2) or (3) may be the best bet? What do you think?

I would say exactly the same! :)

[frederic-santos]

And although it's almost useless now, the newest release of ob-ess-julia closes (I think) the most annoying bug I mentioned earlier. Say hallelujah for my savior on JuliaDiscourse ;)

See you soon, gentlemen!

[nico202]

Perhaps (2) or (3) may be the best bet? What do you think?
Agreed!

I'm proceeding with a refactor of the code, making it a bit more readable, adding comments and removing old leftovers. In the meanwhile I was wondering if you have any thoughts about julia-repl. Maybe it's better than ess. No dependencies, and the ?help work inside emacs. Would it be better or should we stick to ess?

[tecosaur]

I'm guessing the main concern with julia-repl may be windows users (https://github.com/tpapp/julia-repl#note-for-windows-users). Other than that, I don't know how it compares and so have nothing else to add.

[tecosaur]

Oh, it looks like 5 months ago an Alberto Ramos volunteered to maintain Org's ob-julia. I think this may be @ramos.
So, we have 3-4 people who want to make this work 😀 that's fantastic!

[frederic-santos]

Great news! I would be happy to be part of this dream team ;) Clearly, I'm the least comptent in Julia programming, but I can help the main maintainer(s) with documentation, examples, and minor (boring) stuff like this. Thanks @tecosaur for your work on this!

[nico202]

We have a big team now :)
I've started cleaning my part of the code here. Please mind that even if I'm working on master, I plan to rebase things in the future, so history might change.

I'd need help on writing the README and adding things on the TODO list.

This code is now slightly better, with more docstrings, and the only known bug is the use of :var that depends on async blocks (still trying to figure out what to do in those cases).

Would you mind trying it out and reporting bugs and missing features (as well as suggesting which packages we should support out of the box)?

Thanks!

[tecosaur]

Brilliant @nico202! I'll try it out shortly. How about I open issues for any of the features/packages that I think we should support?

[tecosaur]

I also want to add, I'm feeling really good about this. I think we could make some really nice Org/Julia integrations happen here 😀.

[nico202]

Sure, feel free to open as many issues you want :) Also, for each type we want to support, we should discuss how it should be formatted

[tecosaur]

Oh @nico202 I'm currently going through the standard library and looking at type representations. For chatting about this sort of thing do you think it could be nice to use an IM platform? I get the feeling there will be a lot of little things mentioned and it may be nicer that a lot of really small issues / a long chain here.

Just a thought.

[tecosaur]

Actually, this isn't as much as I thought. Issues should be fine 🙂

[xianwenchen]

Thank you all for the contributions! Does this mean that at one point I shall use https://github.com/nico202/ob-julia instead of ob-ess-julia?

By the way, is there a ob julia library that supports graphic output in Org mode document?

[frederic-santos]

Thank you all for the contributions! Does this mean that at one point I shall use https://github.com/nico202/ob-julia instead of ob-ess-julia?

You'll probably find a smoother experience and a better stability there, yep. I'll still maintain ob-ess-julia (unless @tecosaur or @nico202 prefer that I remove it from Melpa to avoid confusion between those implementations in the future), at least for bug fixes. But it will probably never give you the advanced features provided by ob-julia :)

[tecosaur]

Just a little call for help here for more visibility: if anyone has thoughts on nico202/ob-julia#8 (comment) that would be much appreciated.