Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CoLic throwing path error #102

Open
altsalt opened this issue Dec 7, 2021 · 4 comments
Open

CoLic throwing path error #102

altsalt opened this issue Dec 7, 2021 · 4 comments

Comments

@altsalt
Copy link

altsalt commented Dec 7, 2021

Hello, I am participating in the MSR Hackathon and have attempted to install Graal a few times to no avail. Most recently, I set up a clean Debian 11 VM to verify that it wasn't due to prior system configuration, incorrect permissions, etc. Alas, the same error is appearing:

Python 3.9.2 (default, Feb 28 2021, 17:03:44) 
[GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from graal.backends.core.colic import CoLic
>>> repo_uri = 'https://github.com/chaoss/grimoirelab-graal'
>>> repo_dir = 'grimoirelab-graal'
>>> cl = CoLic(uri=repo_uri, git_path=repo_dir)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.9/dist-packages/graal-0.2.9-py3.9.egg/graal/backends/core/colic.py", line 79, in __init__
    if not GraalRepository.exists(exec_path):
  File "/usr/local/lib/python3.9/dist-packages/graal-0.2.9-py3.9.egg/graal/graal.py", line 411, in exists
    return os.path.exists(dest)
  File "/usr/lib/python3.9/genericpath.py", line 19, in exists
    os.stat(path)
TypeError: stat: path should be string, bytes, os.PathLike or integer, not NoneType
>>> 

I have tried with both a blank directory and having pre-created the grimoirelab-graal folder (not that this should be a blocker, but looking for anything that might be...)

Any pointers or guidance would be very much appreciated. Thanks!

@altsalt
Copy link
Author

altsalt commented Dec 7, 2021

Related issues:

Also, while this issue is referencing the git path, it may be relevant.

@altsalt
Copy link
Author

altsalt commented Dec 7, 2021

I've spent hours trying to get graal and scancode to play nice to no avail. I think this is largely due to my python path setup, but given that I have a fresh VM that only being used for this, there is likely an issue with the build instructions.

My latest attempted has graal being built into scancode-toolkit/bin using --prefix. This command is then successful:
(scancode-toolkit) $ graal colic https://github.com/chaoss/grimoirelab-graal --exec-path /home/salt/MSR-Magna_Carta/scancode-toolkit/bin/scancode --category code_license_scancode > ./scancode_test

However, as a script it continues to throw a ModuleNotFoundError. I've tried loading it directly via /home/salt/scancode-toolkit/bin/python3 test.py with or without the virtualenv active. I've also tried a variety of sys.path.append calls. I'm just not sure what I'm missing here....

The test script

#!/home/salt/MSR-Magna_Carta/scancode-toolkit/bin/python3

#import sys
#sys.path.append('/home/salt/MSR-Magna_Carta/scancode-toolkit')
#sys.path.append('/home/salt/MSR-Magna_Carta/scancode-toolkit/bin')
#sys.path.append('/home/salt/MSR-Magna_Carta/scancode-toolkit/lib')
#sys.path.append('/home/salt/MSR-Magna_Carta/scancode-toolkit/python3.9/site-packages/')
#sys.path.append('/home/salt/MSR-Magna_Carta/scancode-toolkit/python3.9/site-packages/graal-0.2.9-py3.9.egg/graal/')

from graal.backends.core.colic import CoLic

# URL for the git repo to analyze
repo_uri = 'https://github.com/chaoss/grimoirelab-graal'

# directory where to mirror the repo
repo_dir = 'grimoirelab-graal'

# CoLic object initialization
cl = CoLic(uri=repo_uri, git_path=repo_dir)

# fetch all commits
commits = [commit for commit in cl.fetch()]

with open("graal_test_commits", "w") as f:
  f.write(commits)

Some of the commands and errors which have gone by

(scancode-toolkit) $ graal colic https://github.com/chaoss/grimoirelab-graal --exec-path /home/salt/MSR-Magna_Carta/scancode-toolkit/etc/scripts/scancli.py  --category code_license_scancode_cli > ./scancode_cli_test
[2021-12-07 11:40:16,258] - Starting the quest for the Graal.
ERROR: Could not open requirements file: [Errno 2] No such file or directory: 'requirements.txt'
[2021-12-07 11:40:16,592] - Error!: Command '['/home/salt/MSR-Magna_Carta/scancode-toolkit/configure']' returned non-zero exit status 1.
[2021-12-07 11:40:16,593] - Quest completed.

@valeriocos
Copy link
Member

valeriocos commented Dec 8, 2021

Hi @altsalt,

I followed the steps here https://github.com/chaoss/grimoirelab-graal#how-to-installcreate-the-executables, section ScanCode

cd /tmp
git clone https://github.com/nexB/scancode-toolkit.git
cd scancode-toolkit
git checkout -b test_scancli 96069fd84066c97549d54f66bd2fe8c7813c6b52
./scancode --help
pip install simplejson execnet

The execution from command line and python were OK

  • command line
graal colic https://github.com/chaoss/grimoirelab-graal --git-path /tmp/test --exec-path /tmp/scancode-toolkit/etc/scripts/scancli.py --category code_license_scancode_cli
  • python
from graal.backends.core.colic import CoLic

# URL for the git repo to analyze
repo_uri = 'https://github.com/chaoss/grimoirelab-graal'

# directory where to mirror the repo
repo_dir = '/tmp/grimoirelab-graal'

# CoLic object initialization
cl = CoLic(uri=repo_uri, git_path=repo_dir, exec_path='/tmp/scancode-toolkit/etc/scripts/scancli.py')

# fetch all commits
commits = [commit for commit in cl.fetch(category='code_license_scancode_cli')] <--- this call may take a lot of time, you may want to save each commit in the file below.

with open("graal_test_commits", "w+") as f:
  f.write(commits)

Hope this helps

EDIT:

Also the category code_license_scancode seems to work fine

  • command line
graal colic https://github.com/chaoss/grimoirelab-graal --git-path /tmp/test --exec-path /tmp/scancode-toolkit/scancode --category code_license_scancode
  • python
from graal.backends.core.colic import CoLic

# URL for the git repo to analyze
repo_uri = 'https://github.com/chaoss/grimoirelab-graal'

# directory where to mirror the repo
repo_dir = '/tmp/grimoirelab-graal'

# CoLic object initialization
cl = CoLic(uri=repo_uri, git_path=repo_dir, exec_path='/tmp/scancode-toolkit/scancode')

# fetch all commits
commits = []
for commit in cl.fetch(category='code_license_scancode'):
  commits.append(commit)
  print(commit)

with open("graal_test_commits", "w+") as f:
  f.write(commits)

@altsalt
Copy link
Author

altsalt commented Dec 8, 2021

An error has been thrown in step 1. I'll install the missing package and keep pushing through, but it may point to part of the problem.

salt@debianclivm:/tmp/scancode-toolkit$ ./scancode --help
* Configuring ScanCode for first use...
/tmp/scancode-toolkit/configure: line 24: python2.7: command not found
./scancode: line 114: /tmp/scancode-toolkit/bin/scancode: No such file or directory

FWIW, I had been building ScanCode from the main branch rather than checking one out.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants