[gitlabcomments] Add enricher to handle gitlab comments #881

vchrombie · 2020-05-26T21:33:03Z

This PR proposes a new gitlab enricher to handle gitlab comments. The old enricher is left in place to avoid breaking changes with existing customer dashboards.

Ref: chaoss/grimoirelab#208

vchrombie · 2020-05-26T21:35:23Z

The PR has a lot of pending work, I will be working on this. I will mark ready for review after completing some good work.

Please let me know if you are having any suggestions.
Thanks.

valeriocos · 2020-05-30T12:12:06Z

Hi @vchrombie , please ping me when I can review this PR. I had a look at the current status and the start looks promising :)

WRT tests (if this can be of any help), you should run Perceval and copy some docs to a test file, which will be finally put in the folder tests/data. The name is important since it is used to automatically identify the file and load its content (https://github.com/chaoss/grimoirelab-elk/blob/master/tests/base.py#L137)

If needed you can tweak the test data (e.g., changing value attributes) to make sure that all code is covered when running the tests.

vchrombie · 2020-06-04T13:13:06Z

Hi @valeriocos
I have completed the major part of the PR. I am yet to complete the __get_reactions function and I am working on that right now.

valeriocos · 2020-06-04T13:18:12Z

Thanks for the update @vchrombie

grimoire_elk/enriched/gitlabcomments.py

vchrombie · 2020-06-04T21:29:55Z

I will mark this PR ready-for-review once I complete the __get_reactions function too. 🚀

grimoire_elk/enriched/gitlabcomments.py

coveralls · 2020-06-06T21:37:10Z

Pull Request Test Coverage Report for Build 1227039055

0 of 0 changed or added relevant lines in 0 files are covered.
30 unchanged lines in 1 file lost coverage.
Overall coverage increased (+0.5%) to 82.719%

Files with Coverage Reduction	New Missed Lines	%
/home/runner/work/grimoirelab-elk/grimoirelab-elk/grimoire_elk/utils.py	30	66.54%

Totals
Change from base Build 1220934812:	0.5%
Covered Lines:	9042
Relevant Lines:	10931

💛 - Coveralls

valeriocos

Hi @vchrombie, thank you for advancing with this PR. I tested it on a battery of gitlab repos (taken from the gitlab:issue section at https://gitlab.com/Bitergia/c/gitlab/sources/-/blob/master/projects.json).

After fixing some minor errors (check comments), the execution was working fine.
I also checked that the url generated for comments work fine and allows to jump on the original comment on gitlab:

"sub_type" : "issue_comment",
          "body" : "@sabbap \"This merge request at least does something with the chevron until @JobV returns from his vacation when he can implement the scrolling.\"\r\n\r\n@JobV your call if this is good enough or not",
          "body_analyzed" : "@sabbap \"This merge request at least does something with the chevron until @JobV returns from his vacation when he can implement the scrolling.\"\r\n\r\n@JobV your call if this is good enough or not",
          "url" : "https://gitlab.com/gitlab-com/www-gitlab-com/issues/272#note_948621", <----
          "comment_updated_at" : "2015-03-13T00:16:20.314Z",
          "id" : "160818_merge_comment_948621",
          "grimoire_creation_date" : "2015-03-13T00:16:20.314000+00:00",
          "is_gitlab_issue_comment" : 1,

Can you add the tests and schema? Thanks

grimoire_elk/enriched/gitlabcomments.py

vchrombie · 2020-06-07T13:31:53Z

#881 (review)

Thanks for the review @valeriocos.
I have just updated the PR with the changes you have suggested.

Can you add the tests and schema?

Sure, I will be working on it and add them here soon.

vchrombie · 2020-06-09T19:35:21Z

Hi @valeriocos

Recent updates to the PR

I removed the reactions mapping as we discussed it is not needed and also implemented the new __get-reactions() function.
added the schema using the gitlabcomments-mrs/issues_enriched index.
added the tests, (highly need a review on this as I'm not really good with it).

I'm right now checking the coverage, will update the PR if needed.

Once the PR is perfect, I can squash the commits and rebase them with the latest changes too.

valeriocos

Hi @vchrombie , thank you for working on this PR. Overall it looks good, I left some minor comments.

I also successfully executed the enricher on a short list of gitlab repos.

grimoire_elk/enriched/gitlabcomments.py

schema/gitlabcomments_issues.csv

grimoire_elk/enriched/gitlabcomments.py

schema/gitlabcomments_merges.csv

tests/test_gitlabcomments.py

vchrombie · 2020-06-10T18:04:02Z

Thanks @valeriocos.
I left a lot of things pending, my bad.
I will get back to you after fixing them. 👍

valeriocos · 2020-06-10T18:15:22Z

No worries, thank you for working on this PR

vchrombie · 2020-06-11T17:38:37Z

schema/gitlabcomments_issues.csv

@@ -11,10 +11,10 @@ assignee_org_name,keyword,true,"Assignee organization name."
 assignee_user_name,keyword,true,"Assignee user name from SortingHat."
 assignee_uuid,keyword,true,"Assignee UUID from SortingHat."
 author_bot,boolean,true,"True if the given author is identified as a bot, from SortingHat profile."
-author_domain,keyword,true,"Domain associated to the author in SortingHat profile."


Hi @valeriocos
I was confused about this deletion.

I didn't see this (author_domain) in the mapping too, but assignee_domain and merged_by_domain are present.
Can you check test the enricher once and let me know if you have this?

The same deletion is present in the merge_request too.

*_domain fields are added by SortingHat. By default GitLab users don't have their emails exposed on the API. However, if you execute sortinghat with a matching algorithm that joins the identities based on username and email (the algorithms are available here), the email information can be collected. The *_domain fields should be filled and thus they will appear in the mapping.

Okay, got it. So, I will add them back to the schema then.

Perfect, let me know when I can review the PR, thanks!

vchrombie · 2020-06-11T19:33:09Z

Also, I have made a small script to generate schema file of an es index, generate-es-index-schema-py.
This was built on top of @animeshk08's script as it had some backlogs.

The script can be improved by adding the idea of default field and adding descriptions. Please leave your comments in the gist or you can open an issue here, vchrombie/grimoirelab-scripts. Feedback is highly welcomed. ^^

vchrombie · 2020-06-12T11:23:55Z

Hi @valeriocos
I have updated the PR as per your suggestions.
Can you review this when you are free?
Thanks.

valeriocos

Hi @vchrombie ,

The PR is almost ready to be merged, thank you for your work! Please check the comments (for the ones concerning the test coverage, try to address them if possible).

Please check the actions below too:

create a yaml file to provide a high-level description of this new addition (https://github.com/Bitergia/release-tools#changelog). As a reference you can check the PR for rocketchat
create an issue in https://github.com/Bitergia/release-tools#changelog to give feedback on the use of the changelog (you can check the issues 14 and 15 in that repo as a reference)
update the readme at https://github.com/chaoss/grimoirelab-elk#enriched-data
submit a PR to mordred to document how to execute the gitlabcomments: https://github.com/chaoss/grimoirelab-sirmordred#supported-data-sources-
squash the commits of this PR in just one
sync this branch with master

valeriocos · 2020-06-12T11:58:42Z

grimoire_elk/enriched/gitlabcomments.py

+                    "type": "text",
+                    "index": true
+               },
+               "id": {


Is this mapping needed?

grimoire_elk/enriched/gitlabcomments.py

valeriocos · 2020-06-12T12:01:41Z

tests/test_gitlabcomments.py

+
+        item = self.items[0]
+        eitem = enrich_backend.get_rich_item(item)
+        self.assertEqual(item['category'], 'issue')


can you add an assert to check the item_type? This applies also to the other items in this test

grimoire_elk/enriched/gitlabcomments.py

valeriocos · 2020-06-12T13:12:10Z

grimoire_elk/enriched/gitlabcomments.py

+            rich_mr.update(self.get_item_project(rich_mr))
+
+        if 'project' in item:
+            rich_mr['project'] = item['project']


Not covered by the tests

grimoire_elk/enriched/gitlabcomments.py

valeriocos · 2020-06-16T07:16:11Z

please @vchrombie ping me when the PR is ready for review, thanks

almereyda · 2021-04-26T21:32:53Z

Coming from outside and having dismissed GrimoireLab previously for lacking GitLab support, I would still like to see it support additional code forges other than GitHub, as per the arguments in inhttps://github.com/chaoss/grimoirelab/issues/284#issuecomment-827155190).

Is there something me as an outsider can contribute to the formal aspects of the PR, in so the remaining technical questions can be answered more easily? Speaking of:

rebase on master
update elk readme
add mordred supported data sources documentation (PR)
update tests for edge cases and error conditions

vchrombie · 2021-04-27T15:54:05Z

Hi @almereyda, thanks for your comment.

Coming from outside and having dismissed GrimoireLab previously for lacking GitLab support

Sorry about it, but the current GrimoireLab supports analyzing issues and merge requests from the gitlab projects (reference). This PR is an additional enricher which can analyze the comments of the gitlab issues and mrs. I couldn't complete it because I completely forgot about this.

Is there something me as an outsider can contribute to the formal aspects of the PR, in so the remaining technical questions can be answered more easily? Speaking of:

Thanks for offering help, I appreciate it. But, I think I can complete the work on this PR. I can pull some time during the coming weekend and complete the pending work.

This commit adds a new enricher to handle gitlab comments data. This is the github2 version for gitlab. The tests are added accordingly. Signed-off-by: Venu Vardhan Reddy Tekula <[email protected]>

vchrombie mentioned this pull request May 26, 2020

[enricher] Add gitlabcomments Enricher vchrombie/gsoc#4

Closed

vchrombie force-pushed the gitlabcomments branch from 4633b66 to 8774b57 Compare June 4, 2020 13:10

vchrombie commented Jun 4, 2020

View reviewed changes

grimoire_elk/enriched/gitlabcomments.py Show resolved Hide resolved

vchrombie commented Jun 4, 2020

View reviewed changes

grimoire_elk/enriched/gitlabcomments.py Outdated Show resolved Hide resolved

vchrombie commented Jun 4, 2020

View reviewed changes

grimoire_elk/enriched/gitlabcomments.py Outdated Show resolved Hide resolved

vchrombie commented Jun 4, 2020

View reviewed changes

grimoire_elk/enriched/gitlabcomments.py Outdated Show resolved Hide resolved

vchrombie force-pushed the gitlabcomments branch from fe4d7a7 to 27ff93b Compare June 4, 2020 21:14

vchrombie commented Jun 6, 2020

View reviewed changes

grimoire_elk/enriched/gitlabcomments.py Show resolved Hide resolved

vchrombie force-pushed the gitlabcomments branch 2 times, most recently from 95edbc8 to c61362e Compare June 6, 2020 21:19

vchrombie marked this pull request as ready for review June 6, 2020 22:11

valeriocos reviewed Jun 7, 2020

View reviewed changes

grimoire_elk/enriched/gitlabcomments.py Show resolved Hide resolved

grimoire_elk/enriched/gitlabcomments.py Outdated Show resolved Hide resolved

grimoire_elk/enriched/gitlabcomments.py Outdated Show resolved Hide resolved

vchrombie force-pushed the gitlabcomments branch 3 times, most recently from 705ed10 to 2d14167 Compare June 9, 2020 19:26

vchrombie force-pushed the gitlabcomments branch 3 times, most recently from 240e81d to c2a40cb Compare June 10, 2020 06:26

valeriocos reviewed Jun 10, 2020

View reviewed changes

vchrombie commented Jun 11, 2020

View reviewed changes

vchrombie force-pushed the gitlabcomments branch from 1e30dfe to 10bf53e Compare June 12, 2020 11:20

valeriocos reviewed Jun 12, 2020

View reviewed changes

vchrombie force-pushed the gitlabcomments branch from 10bf53e to b41798a Compare August 10, 2020 23:01

GeorgLink mentioned this pull request Oct 21, 2020

Add "reactions" as parameter on Issues_Closed chaoss/wg-evolution#378

Merged

vchrombie force-pushed the gitlabcomments branch 2 times, most recently from 78a9e7f to 84580a1 Compare September 11, 2021 15:46

[gitlabcomments] Add enricher to handle gitlab comments

d0bd0dc

This commit adds a new enricher to handle gitlab comments data. This is the github2 version for gitlab. The tests are added accordingly. Signed-off-by: Venu Vardhan Reddy Tekula <[email protected]>

vchrombie force-pushed the gitlabcomments branch from 84580a1 to d0bd0dc Compare September 12, 2021 18:02

vchrombie changed the title ~~[enriched-gitlabcomments] New enricher to handle gitlab comments~~ [gitlabcomments] Add enricher to handle gitlab comments Sep 12, 2021

vchrombie mentioned this pull request Sep 15, 2021

GitLab issues, wiki, CI support chaoss/grimoirelab-perceval#119

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[gitlabcomments] Add enricher to handle gitlab comments #881

[gitlabcomments] Add enricher to handle gitlab comments #881

vchrombie commented May 26, 2020

vchrombie commented May 26, 2020 •

edited

Loading

valeriocos commented May 30, 2020

vchrombie commented Jun 4, 2020

valeriocos commented Jun 4, 2020

vchrombie commented Jun 4, 2020 •

edited

Loading

coveralls commented Jun 6, 2020 •

edited

Loading

valeriocos left a comment

vchrombie commented Jun 7, 2020

vchrombie commented Jun 9, 2020 •

edited

Loading

valeriocos left a comment

vchrombie commented Jun 10, 2020

valeriocos commented Jun 10, 2020

vchrombie Jun 11, 2020

valeriocos Jun 12, 2020

vchrombie Jun 12, 2020

valeriocos Jun 12, 2020

vchrombie commented Jun 11, 2020

vchrombie commented Jun 12, 2020

valeriocos left a comment •

edited

Loading

valeriocos Jun 12, 2020

valeriocos Jun 12, 2020

valeriocos Jun 12, 2020

valeriocos commented Jun 16, 2020

almereyda commented Apr 26, 2021

vchrombie commented Apr 27, 2021 •

edited

Loading

[gitlabcomments] Add enricher to handle gitlab comments #881

Are you sure you want to change the base?

[gitlabcomments] Add enricher to handle gitlab comments #881

Conversation

vchrombie commented May 26, 2020

vchrombie commented May 26, 2020 • edited Loading

valeriocos commented May 30, 2020

vchrombie commented Jun 4, 2020

valeriocos commented Jun 4, 2020

vchrombie commented Jun 4, 2020 • edited Loading

coveralls commented Jun 6, 2020 • edited Loading

Pull Request Test Coverage Report for Build 1227039055

💛 - Coveralls

valeriocos left a comment

Choose a reason for hiding this comment

vchrombie commented Jun 7, 2020

vchrombie commented Jun 9, 2020 • edited Loading

valeriocos left a comment

Choose a reason for hiding this comment

vchrombie commented Jun 10, 2020

valeriocos commented Jun 10, 2020

vchrombie Jun 11, 2020

Choose a reason for hiding this comment

valeriocos Jun 12, 2020

Choose a reason for hiding this comment

vchrombie Jun 12, 2020

Choose a reason for hiding this comment

valeriocos Jun 12, 2020

Choose a reason for hiding this comment

vchrombie commented Jun 11, 2020

vchrombie commented Jun 12, 2020

valeriocos left a comment • edited Loading

Choose a reason for hiding this comment

valeriocos Jun 12, 2020

Choose a reason for hiding this comment

valeriocos Jun 12, 2020

Choose a reason for hiding this comment

valeriocos Jun 12, 2020

Choose a reason for hiding this comment

valeriocos commented Jun 16, 2020

almereyda commented Apr 26, 2021

vchrombie commented Apr 27, 2021 • edited Loading

vchrombie commented May 26, 2020 •

edited

Loading

vchrombie commented Jun 4, 2020 •

edited

Loading

coveralls commented Jun 6, 2020 •

edited

Loading

vchrombie commented Jun 9, 2020 •

edited

Loading

valeriocos left a comment •

edited

Loading

vchrombie commented Apr 27, 2021 •

edited

Loading