Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

http://www.essepuntato.it/lode service down #16

Open
lewismc opened this issue Sep 14, 2018 · 12 comments
Open

http://www.essepuntato.it/lode service down #16

lewismc opened this issue Sep 14, 2018 · 12 comments

Comments

@lewismc
Copy link

lewismc commented Sep 14, 2018

Hi @essepuntato @seralf @nicholascar I love this service and have been using it to generate a cache visualizations of the rather large SWEET ontology suite, the visualizations can be seen at https://esipfed.github.io/stc/sweet_lode/sweetAll.html

As I began iterating through the generation of visualizations for all of the files in SWEET, after a while I encountered the following responses

<html><head><title>Apache Tomcat/6.0.35 - Error report</title><style><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}--></style> </head><body><h1>HTTP Status 500 - </h1><HR size="1" noshade="noshade"><p><b>type</b> Exception report</p><p><b>message</b> <u></u></p><p><b>description</b> <u>The server encountered an internal error () that prevented it from fulfilling this request.</u></p><p><b>exception</b> <pre>javax.servlet.ServletException: Servlet execution threw an exception
</pre></p><p><b>root cause</b> <pre>java.lang.OutOfMemoryError: Java heap space
</pre></p><p><b>note</b> <u>The full stack trace of the root cause is available in the Apache Tomcat/6.0.35 logs.</u></p><HR size="1" noshade="noshade"><h3>Apache Tomcat/6.0.35</h3></body></html>

...

java.lang.OutOfMemoryError: Java heap space

I therefore fear that I may have crashed your service.
I wanted to say that I was not executing these requests in parallel or within a short time of each other. Each request was submitted literally minutes apart.
Apologies, if I crashed the service.
@nicholascar
Copy link
Contributor

Hi Lewis, We have a v2 up at http://lode2.linked.data.gov.au that incorporates a few updates to the HTML generated. @essepuntato and I have discussed folding those changes back into the main codebase but, for now, there are two services in case one goes down and you are caught off guard.

@lewismc
Copy link
Author

lewismc commented Sep 18, 2018

Thanks @nicholascar :)

@lewismc
Copy link
Author

lewismc commented Sep 18, 2018

@nicholascar there are several improvements in the data.gov.au service you pointed me to so thank you for that. I just used the service to render the following file https://github.com/ESIPFed/sweet/blob/master/src/sweetAll.ttl which is essentially just a list of imports.
Again, I'm running in to issues when executing the following request

http://lode2.linked.data.gov.au/extract?imported=true&webvowl=true&url=https://raw.githubusercontent.com/ESIPFed/sweet/master/src/sweetAll.ttl

Proxy Error

The proxy server received an invalid response from an upstream server.
The proxy server could not handle the request POST /extract.

Reason: Error reading from remote server

Apache/2.4.18 (Ubuntu) Server at lode2.linked.data.gov.au Port 80

Any advice on better use of the service? Thanks in advance.

@nicholascar
Copy link
Contributor

Hmmm... neither LODE version can handle your hugh import command! It's clearly just timing out. I ran this Python script to merge all the graphs:

import requests
import rdflib

gg = rdflib.Graph()

with open('imports.txt', 'r') as f:
    for l in f.readlines():
        r = requests.get(l.strip(), headers={'Accept': 'text/turtle'})
        g = rdflib.Graph().parse(data=r.content.decode('utf-8'), format='turtle')
        print(len(g))
        gg += g

print(len(gg))

with open('all.ttl', 'w') as f:
    f.write(gg.serialize(format='turtle').decode('utf-8'))

I ran that and then ran LODE2 on all.ttl but that's timing out too. I think SWEET is just too big!

You might have to template this one out: I have, for large SKOS vocabs, just used a simple Python script with the rdflib + Jinja2 templating modules. Creating a template for Concept, Collection & ConceptSchema for SKOS works well due to the limited set of classes and properties for them. If SWEET uses only a few basic annotation properties, this approach would be reasonable I think (template for Class, Property, Ontology with basic relations.

I may, if you lead, help create such a template-based approach as an alternate to this LODE XSLT approach...

@lewismc
Copy link
Author

lewismc commented Sep 19, 2018

Thanks for the above @nicholascar
This is an issue indeed and yes I agree that both SWEET's large nature and highly connected characteristics seem to be bogging down the service. This is no surprise as ontology editors e.g. Protege, also struggle depending on availability of memory on host machine.
I do not have cycles for coding up the LODE templating right now, however we may be able to consider it for a future GSoC or ESIP Labs project... for example ESIP Labs CfP opens in November. Do you envisage the LODE templating approach to eventually replace the XSLT architecture?

On a separate thread, regarding the LODE deployment you have at http://lode2.linked.data.gov.au, are you guys developing that code in an open forum or elsewhere? I've am not seeing any branches in this repository so a bit puzzled as to which code I should be looking at. Thanks

@nicholascar
Copy link
Contributor

OK, I'll see how I go with a templating thing. It would be very different approach to LODE of course but I think there are a couple of reasons to implement something like that. Large ontology size is one reason but another is for allowing people to override parts: we could implement a series of templates that transform certain RDF into HTML but then if someone wanted something done differently - perhaps catering for a particular class in a particular way - then they could either add a template for that or override an existing one. If wedo this, it will be within the Python rdflib world (https://github.com/rdflib) since we mainly use Python for RDF handling and there's quite an ecosystem of tools we're familiar with there.

The code for LODE2 is at https://github.com/CSIRO-enviro-informatics/LODE and we're continuing to develope it as we have a continuing need for standardised ways of documenting many Australian govt. ontologies. We will probably make a fork of LODE2 that has Aust. Govt. logos, style, and so on already embedded. We could go down the templating path here though... No decisions yet! I suspect we will have a hard requirement to choose to either make that Aus Govt fork or make templates by December, based on our project work.

@essepuntato
Copy link
Owner

Hi all,

Sorry for being late in answering. Great to see that @nicholascar has already addressed all the issues raised. We would need to work, indeed, in integrating the LODE2 version in the main repository, indeed, at certain point. I just need to figure out when I have a bunch of spare time (ah!) for doing so.

@np00
Copy link

np00 commented Feb 18, 2019

LODE (1) is down: "internal server error" :/

http://www.essepuntato.it/lode

@nicholascar
Copy link
Contributor

http://lode2.linked.data.gov.au is working if you need LODE in a hurry, although Silvio's pretty quick at getting his system back up usually!

@np00
Copy link

np00 commented Feb 20, 2019

yeah thanks for the pointer. I found it, but somehow it also does not work for me :(
#20

@nicholascar
Copy link
Contributor

@np00 do you mean http://lode2.linked.data.gov.au isn't working for you? Just a bit hard to tell in this thread.

If so, just drop Edmond an email ([email protected]) and explain the bug you're seeing and he'll fix it within a day since he actively maintains the tool. We do know about the service maxing out if very large ontologies are loaded.

@np00
Copy link

np00 commented Feb 21, 2019

Correct, somehow lode2 does not work with my ontology. I created an issue for my problem with lode2: #20 I will drop him a mail about it.

lode1 seems to not work at all.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@lewismc @essepuntato @nicholascar @np00