Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jira Cloud: Almost there! #26

Closed
KelSolaar opened this issue Sep 2, 2019 · 37 comments
Closed

Jira Cloud: Almost there! #26

KelSolaar opened this issue Sep 2, 2019 · 37 comments

Comments

@KelSolaar
Copy link

KelSolaar commented Sep 2, 2019

No description provided.

@KelSolaar
Copy link
Author

Seems like I managed to get rid of the history so I can put the updated content again:

Follow-up on Zendesk Ticket #105065. So I was looking at the jira_basic_auth branch and we started to patch the code in a similar way, actually a bit more so that our webserver works with HTTPS and our certificates. Here is a diff between our branch and v0.1_enhancements:

$ git diff v0.1_enhancements
diff --git a/sg_jira/bridge.py b/sg_jira/bridge.py
index f859509..953a174 100644
--- a/sg_jira/bridge.py
+++ b/sg_jira/bridge.py
@@ -67,7 +67,7 @@ class Bridge(object):

         self._jira = JiraSession(
             jira_site,
-            auth=(
+            basic_auth=(
                 jira_user,
                 jira_secret
             ),
@@ -212,7 +212,8 @@ class Bridge(object):

         :returns: A string with the username.
         """
-        return urllib.unquote_plus(self.jira.current_user())
+        # TODO: "JIRA.current_user()" returns None has "JIRA._myself" does not exist for us.
+        return urllib.unquote_plus(self.jira.current_user() or os.environ['SGJIRA_JIRA_USER'])

     @property
     def jira(self):
diff --git a/webapp.py b/webapp.py
index e54f65e..5743c40 100644
--- a/webapp.py
+++ b/webapp.py
@@ -477,7 +477,7 @@ def run_server(port, settings, keyfile=None, certfile=None):
     """
     httpd = Server(
         settings,
-        ("localhost", port), RequestHandler
+        ("", port), RequestHandler
     )
     if keyfile and certfile:
         # Activate HTTPS.

Essentially getting rid of "localhost" and see the TODO, this one is weird but the JIRA._myself attribute is undefined for us. Anyway, with that patch we are getting a bit further up and things start to talk together, unfortunately, there is an issue with the emailAddress field not existing:

sg-jira-bridge Log

$ jira-bridge
2019-09-02 14:15:15,765 INFO [bridge] Successfully read settings from ***/sg-jira-bridge/settings.py
2019-09-02 14:15:16,078 INFO [shotgun_session] Connected to https://***.shotgunstudio.com.
2019-09-02 14:15:16,704 INFO [jira_session] Connected to https://***.atlassian.net.
2019-09-02 14:15:20,334 INFO [webapp] *** - /jira2sg/default - "GET /jira2sg/default HTTP/1.1" 200 -
2019-09-02 14:15:20,827 ERROR [webapp] *** - /favicon.ico - code 404, message Not Found
2019-09-02 14:15:20,829 INFO [webapp] *** - /favicon.ico - "GET /favicon.ico HTTP/1.1" 404 -
2019-09-02 14:15:57,215 INFO [webapp] *** - /admin/reset - "POST /admin/reset HTTP/1.1" 200 -
2019-09-02 14:15:57,708 ERROR [bridge] 'emailAddress'
Traceback (most recent call last):
  File "***/sg-jira-bridge/sg_jira/bridge.py", line 344, in sync_in_shotgun
    handler = syncer.accept_jira_event(resource_type, resource_id, event)
  File "***/sg-jira-bridge/sg_jira/syncer.py", line 168, in accept_jira_event
    if user["emailAddress"].lower() == self.bridge.current_jira_username.lower():
KeyError: 'emailAddress'
2019-09-02 14:15:57,709 ERROR [webapp] *** - /jira2sg/default/Task/*** - code 500, message emailAddress
2019-09-02 14:15:57,710 INFO [webapp] *** - /jira2sg/default/Task/*** - "POST /jira2sg/default/Task/*** HTTP/1.1" 500 -
2019-09-02 14:15:57,898 ERROR [bridge] 'emailAddress'
Traceback (most recent call last):
  File "***/sg-jira-bridge/sg_jira/bridge.py", line 344, in sync_in_shotgun
    handler = syncer.accept_jira_event(resource_type, resource_id, event)
  File "***/sg-jira-bridge/sg_jira/syncer.py", line 168, in accept_jira_event
    if user["emailAddress"].lower() == self.bridge.current_jira_username.lower():
KeyError: 'emailAddress'
2019-09-02 14:15:57,898 ERROR [webapp] *** - /jira2sg/default/Task/*** - code 500, message emailAddress
2019-09-02 14:15:57,899 INFO [webapp] *** - /jira2sg/default/Task/*** - "POST /jira2sg/default/Task/*** HTTP/1.1" 500 -
2019-09-02 14:15:58,002 ERROR [bridge] 'emailAddress'
Traceback (most recent call last):
  File "***/sg-jira-bridge/sg_jira/bridge.py", line 344, in sync_in_shotgun
    handler = syncer.accept_jira_event(resource_type, resource_id, event)
  File "***/sg-jira-bridge/sg_jira/syncer.py", line 168, in accept_jira_event
    if user["emailAddress"].lower() == self.bridge.current_jira_username.lower():
KeyError: 'emailAddress'
2019-09-02 14:15:58,003 ERROR [webapp] *** - /jira2sg/default/Task/*** - code 500, message emailAddress
2019-09-02 14:15:58,003 INFO [webapp] *** - /jira2sg/default/Task/*** - "POST /jira2sg/default/Task/*** HTTP/1.1" 500 -
2019-09-02 14:15:58,109 ERROR [bridge] 'emailAddress'
Traceback (most recent call last):
  File "***/sg-jira-bridge/sg_jira/bridge.py", line 344, in sync_in_shotgun
    handler = syncer.accept_jira_event(resource_type, resource_id, event)
  File "***/sg-jira-bridge/sg_jira/syncer.py", line 168, in accept_jira_event
    if user["emailAddress"].lower() == self.bridge.current_jira_username.lower():
KeyError: 'emailAddress'
2019-09-02 14:15:58,109 ERROR [webapp] *** - /jira2sg/default/Task/*** - code 500, message emailAddress
2019-09-02 14:15:58,110 INFO [webapp] *** - /jira2sg/default/Task/*** - "POST /jira2sg/default/Task/*** HTTP/1.1" 500 -

shotgunEvents Log

CRITICAL:plugin.sg_jira_event_trigger.process_event:An error occured processing an event.

Traceback (most recent call last):
  File "***/shotgunEvents/src/shotgunEventDaemon.py", line ***0, in process
    self._callback(self._shotgun, self._logger, event, self._args)
  File "***/sg-jira-bridge/triggers/sg_jira_event_trigger.py", line 166, in process_event
    response.raise_for_status()
  File "***/***/python2.7/site-packages/requests/models.py", line 940, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
HTTPError: 500 Server Error: emailAddress for url: https://***/jira2sg/default/Task/***


Local variables at outer most frame in plugin:

{'dispatch_routes': {***: 'https://***/jira2sg/default',
                     ***: None,
                     ***: None},
 'entity_id': ***,
 'entity_type': 'Task',
 'event': {'attribute_name': 'sg_sync_in_jira',
           'created_at': datetime.datetime(2019, 9, 2, 14, 30, 2, tzinfo=<shotgun_api3.lib.sgtimezone.LocalTimezone object at 0x7fd9b3dad910>),
           'entity': {'id': ***,
                      'name': 'Jira Bridge - Test Tasks - 05',
                      'type': 'Task'},
           'event_type': 'Shotgun_Task_Change',
           'id': 1002601,
           'meta': {'attribute_name': 'sg_sync_in_jira',
                    'entity_id': ***,
                    'entity_type': 'Task',
                    'field_data_type': 'checkbox',
                    'in_create': True,
                    'new_value': True,
                    'old_value': False,
                    'type': 'attribute_change'},
           'project': {'id': ***, 'name': '***', 'type': 'Project'},
           'session_uuid': '***',
           'type': 'EventLogEntry',
           'user': {'id': ***,
                    'name': '*** ***',
                    'type': 'HumanUser'}},
 'logger': <logging.Logger object at 0x7fd9b316bf90>,
 'meta': {'attribute_name': 'sg_sync_in_jira',
          'entity_id': ***,
          'entity_type': 'Task',
          'field_data_type': 'checkbox',
          'in_create': True,
          'new_value': True,
          'old_value': False,
          'type': 'attribute_change'},
 'payload': {'entity_id': ***,
             'entity_type': 'Task',
             'meta': {'attribute_name': 'sg_sync_in_jira',
                      'entity_id': ***,
                      'entity_type': 'Task',
                      'field_data_type': 'checkbox',
                      'in_create': True,
                      'new_value': True,
                      'old_value': False,
                      'type': 'attribute_change'},
             'project': {'id': ***, 'name': '***', 'type': 'Project'},
             'session_uuid': '***',
             'user': {'id': ***,
                      'name': '*** ***',
                      'type': 'HumanUser'}},
 'project': {'id': ***, 'name': '***', 'type': 'Project'},
 'response': <Response [500]>,
 'sg': <shotgun_api3.shotgun.Shotgun object at 0x7fd9b3489410>,
 'sync_server_url': 'https://***/jira2sg/default',
 'sync_url': 'https://***/jira2sg/default/Task/***'}

The User email visibility setting in our Jira Cloud instance is set to Public.

@KelSolaar
Copy link
Author

@manneohrstrom : Feel free to re-open the ticket at your convenience and sorry for the fuss!

@manneohrstrom
Copy link
Contributor

Thanks Thomas! No fuss at all :) - I'll reopen it and will let the team know!

@manneohrstrom manneohrstrom reopened this Sep 9, 2019
@gplsteph
Copy link
Contributor

gplsteph commented Sep 9, 2019

hello Thomas, quick heads up that we issued a quick fix for one of your problems:
48bb634
Similar to your fix, but it seems better to not introduce a dependency to the environment variable.

Not being able to retrieve users email addresses will cause problems down the line: they are used to match Jira to Shotgun users (e.g. for assignments). We run a test setup with an admin user for the bridge and are able to get Jira users email address. I don't think we changed the email privacy setting anywhere. Could that be an option for you?

@KelSolaar
Copy link
Author

Not being able to retrieve users email addresses will cause problems down the line: they are used to match Jira to Shotgun users (e.g. for assignments).

Yeah that what I was gathering from the code. We will have to think about what it means from a security standpoint to have a admin user doing that.

@KelSolaar
Copy link
Author

By the way, you are testing with Jira Cloud right?

@gplsteph
Copy link
Contributor

gplsteph commented Sep 10, 2019 via email

@KelSolaar
Copy link
Author

Thanks! How hard would it be for you guys to try with a regular user? I'm not very comfortable with the requirement for it to be admin. I will also separately raise a ticket with Atlassian tomorrow to try understanding what is going on.

@KelSolaar
Copy link
Author

KelSolaar commented Sep 10, 2019

Update from our end, so it turns out that irrespective of the User email visibility setting in our Jira Cloud instance being set to Public, a logged user is able to get the emailAddress field through the API.

This is executed on our Droplet with the Python environment the Jira Bridge uses and the same user:

>>> from jira import JIRA
>>> jr = JIRA('https://***.atlassian.net', basic_auth=('***@***.com', '***'))
>>> jr.issue('***').fields.creator.emailAddress
u'***@***.com'
>>> jr.issue('***').fields.reporter.emailAddress
u'***@***.com'

It makes me rather confused over what is currently not working. Maybe I'm not using the Bridge correctly or as it was intended. The original logs I provided are generated upon creating a Task in Shotgun and ticking the Sync in Jira tickbox, my expectations would be that it would create a matching issue in Jira. Is this the case?

@KelSolaar
Copy link
Author

So looking at the event here: https://github.com/shotgunsoftware/sg-jira-bridge/blob/jira_basic_auth/sg_jira/syncer.py#L160

We don't have an emailAddress key:

{u'entity_id': ***, u'entity_type': u'Task', u'project': {u'type': u'Project', u'id': *, u'name': u'*'}, u'meta': {u'entity_id': *, u'attribute_name': u'color', u'in_create': True, u'entity_type': u'Task', u'old_value': None, u'field_data_type': u'color', u'new_value': u'pipeline_step', u'type': u'attribute_change'}, u'user': {u'type': u'HumanUser', u'id': *, u'name': u'*'}, u'session_uuid': u'*'},

My understanding it that this is actually an event from Shotgun, so I'm puzzled as to why it is being handled by a definition called accept_jira_event, not only that but the Shotgun field for the email address of a Person entity is email anyway.

@KelSolaar
Copy link
Author

KelSolaar commented Sep 15, 2019

Hi,

Still no luck with the master, not a surprise given it is the same code than what we are using. This morning, I tried to create a ticket in Jira and I get the following error in the webapp log:

2019-09-16 08:49:25,555 ERROR [bridge] 'name'
Traceback (most recent call last):
  File "***/sg-jira-bridge/sg_jira/bridge.py", line 342, in sync_in_shotgun
    handler = syncer.accept_jira_event(resource_type, resource_id, event)
  File "***/sg-jira-bridge/sg_jira/syncer.py", line 162, in accept_jira_event
    if user["name"].lower() == self.bridge.current_jira_username.lower():
KeyError: 'name'

It really does feel like the Jira Events processing code processes Shotgun Events and vice versa. I have checked again our configuration and I think we are doing everything as per your documentation.

@gplsteph
Copy link
Contributor

Hello, it seems your log is about accepting a Jira event, so not sure what you mean about Jira processing code processing SG event? To be sure, you can stop the SG event daemon, so you will only get Jira to SG events being processed.
Jira Cloud introduced non-backward compatible changes to comply with GDPR (in a super strict way). It seems the user name is not available in the Jira events payload anymore, and has been replaced with an accountId key. This could explain the name KeyError. Please note that Jira local servers are not affected by these GDPR changes..
https://developer.atlassian.com/cloud/jira/platform/deprecation-notice-user-privacy-api-migration-guide/

@KelSolaar
Copy link
Author

@gplsteph : Did you see the previous one though? #26 (comment)

@gplsteph
Copy link
Contributor

@KelSolaar: just double checked your SG event first log and this sounds strange:

'sync_server_url': 'https://***/jira2sg/default',
 'sync_url': 'https://***/jira2sg/default/Task/***'}

I would expect a sg2jira url here?

@KelSolaar
Copy link
Author

It is actually, the *** is the shadowed/redacted URL of our DO droplet.

@KelSolaar
Copy link
Author

Oh and thanks for taking some time to poke at this, I was getting a bit desperate :)

@gplsteph
Copy link
Contributor

Hmm, there is jira2sg in your url? If I remember well this value is pulled from a SG Project field? It should definitely be a sg2jira url to sync SG events to Jira.

@KelSolaar
Copy link
Author

KelSolaar commented Sep 16, 2019

Right, I see now, thanks! I haven't done the original setup so I totally overlooked that. I'm not at work currently I just re-read the docs and I can indeed see the two types of urls:

  • https://***/jira2sg
  • https://***/sg2jira

It is likely the explanation for what I was describing #26 (comment) I will test that first thing tomorrow.

Might be worth putting an emphasis about that in the documentation.

@KelSolaar
Copy link
Author

@gplsteph : Confirming that the Shotgun 2 Jira path is working now, we indeed have mis-configured field. The Jira 2 Shotgun path is still busted with same error than reported in #26 (comment)

I need to take a look at how the transitions are mapped. Importantly we still need to remove the "localhost" value for the server address so that it works from the web:

diff --git a/webapp.py b/webapp.py
index 1634a93..62f2ada 100644
--- a/webapp.py
+++ b/webapp.py
@@ -275,7 +275,7 @@ def run_server(port, settings, keyfile=None, certfile=None):
     """
     httpd = Server(
         settings,
-        ("localhost", port), RequestHandler
+        ("", port), RequestHandler
     )
     if keyfile and certfile:
         # Activate https

May I suggest for this to be parameterized please?

Is it also possible to put a warning/note in the documentation about the two urls paths, it is easy to overlook and we totally missed it, I did for sure when taking over the setup on our end.

Keep you posted on our progresses.

@KelSolaar
Copy link
Author

KelSolaar commented Sep 16, 2019

I have played a little bit more and here is a small report:

  1. Creating a Task in Shotgun creates an Issue in Jira as expected.
  2. Creating an Issue in Jira raises the exception noted in Jira Cloud: Almost there! #26 (comment), probably what this Known Limitation is about:

SG Jira Bridge does not create entities in Shotgun, it will only update existing ones. This means syncing can only initiate from Shotgun and not from Jira. If an entity in Shotgun that was synced with an issue in Jira is deleted, it will not currently be re-created in Shotgun.

  1. Adding a Note to the Shotgun Task creates a Reply in Jira as expected.
  2. Replying to a Note does not do anything as per the Known Limitations, this is arguably a big issue, and I reckon it might be better to just inline them in the Issue Replies:

Note Replies in Shotgun are not currently handled by SG Jira Bridge. Jira has no concept of replies, or threading in Comments. This workflow is being investigated.

  1. Replying in the Jira Issue does not create a Note for the Shotgun Task, no errors to be seen so I don't know if it is expected.

@unquietwiki
Copy link

@KelSolaar @gplsteph : the place I'm working at is looking to use this with Jira Cloud, and tasked me with the installation of this daemon. Should we wait for you all to resolve this, or is there some way that I and a fellow dev can pitch in? From my initial observation, I was hoping that we could just be using API keys from Shotgun & Jira Cloud, and not need to set up dummy users that would cost money to operate for a classic model we're not implementing; I am also interested if the webhook can be HTTPS, or not. Thanks!

@KelSolaar
Copy link
Author

I think we are close to be in a good place, the only really showstopper issue for now on my end is point 5 in my previous comment. I haven't had time to look at that yesterday nor will have today but being able to have the other path working is obviously mandatory, transistions mapping is also very important but we can at least do that manually.

The webapp we serve is through HTTPS indeed can see from the logs above. We are starting the webapp with the certificates passed to the ssl_context argument and the 443 port:

 --ssl_context /***/certificates/privkey.pem /***/certificates/fullchain.pem --port 443

@unquietwiki
Copy link

unquietwiki commented Sep 18, 2019

@KelSolaar I can see about checking out the code in your branch & also if you ran into #30 on my initial scan of this code. I'm not a Python guru, but willing to take a stab at it.

Edit: forgot to mention #31

@KelSolaar
Copy link
Author

@gplsteph : I'm trying to address my bullet point n. 5, I have set the verbose level to DEBUG in the bridge settings and when I create a comment on the Jira side I get these log messages:

2019-09-19 10:03:55,100 DEBUG [entity_issue_handler] Rejecting event with an unsupported webhook event 'comment_created': {u'comment': {u'body': u'Re-Pong\\!', ***', u'webhookEvent': u'comment_created'}
2019-09-19 10:03:55,101 DEBUG [note_comment_handler] Rejecting event with unsupported webhookEvent comment_created. Handler only accepts comment_updated events: {u'comment': {u'body': u'Re-Pong\\!', ***, u'webhookEvent': u'comment_created'}
2019-09-19 10:03:55,102 DEBUG [syncer] Event {u'comment': {u'body': u'Re-Pong\\!', ***, u'webhookEvent': u'comment_created'} was rejected by all handlers [<sg_jira.handlers.enable_syncing_handler.EnableSyncingHandler object at 0x7f58b63964d0>, <sg_jira.handlers.task_issue_handler.TaskIssueHandler o
ect at 0x7f58b6396450>, <sg_jira.handlers.note_comment_handler.NoteCommentHandler object at 0x7f58b6396490>]

Our webhook is setup as follows:

image

@gplsteph
Copy link
Contributor

@KelSolaar creating a comment from Jira side is not supported I think. If I remember well, only notes which have been synced from SG can be updated back from Jira to SG.

@KelSolaar
Copy link
Author

KelSolaar commented Sep 19, 2019

Thanks,

This is what I assumed when looking at that unit test case: https://github.com/shotgunsoftware/sg-jira-bridge/blob/master/tests/test_syncer.py#L1458

This is quite a major issue because we would expect to be able to answer back from Jira. Idea being that the production people don't have to leave Shotgun and the Devs don't have to leave Jira.

@gplsteph
Copy link
Contributor

Makes sense. A bit complicated since on the Jira side there is a unique comment and on the SG side you can have multiple notes and replies. Not sure what the workflow should be here...

@KelSolaar
Copy link
Author

I would maybe inline everything on the Jira end:

  • Shotgun Notes and Note Replies ---> Jira Comments
  • Jira Comments ---> Notes

It will then come down for people to get used to how the system works.

@unquietwiki
Copy link

Regarding the current codebase, I have things kinda working on my end with https://github.com/unquietwiki/sg-jira-bridge ; however, getting a JIRA_SHOTGUN_TYPE_FIELD issue. Most of the changes I did were automated per #31 ; but had to manually change some arg/karg stuff, since apparently Python3 behaves differently for that.

Is there a separate codebase I should be investigating, or anything else to do to follow along with this thread?

@KelSolaar
Copy link
Author

I would not expect to get Python 3 support before it is supported by the DCCs themselves, it would certainly be great but hard to achieve with Maya & co. following. What issue do you have exactly? There are so many changes on your fork that it is hard to understand what is going on.

@unquietwiki
Copy link

@KelSolaar when trying to set this up, I noticed shotgunsoftware/shotgunEvents#68 , which was Python 3 support for that side of things; except it appears to be an incomplete port. After seeing your thread on here regarding how the latest version of this was mostly working, I was curious if making a code update would help; especially since the latest Shotgun API does support it.

@KelSolaar
Copy link
Author

KelSolaar commented Sep 22, 2019

At this stage the bridge works for us, the problem is whether the current functionality is enough for our needs. I unfortunately don't think it is because we need to:

  • Have bi-directional messaging between Shotgun and Jira, cf. The bridge should be bi-directional. #33. The fact both applications do not have the same conversation flow is an issue but it could be worked around as mentioned above, i.e. inlining, but not having the capability to do the back and forth is a showstopper for us.
  • Being able to map the statuses between Shotgun and Jira, cf. How to map Shotgun Statuses to Jira Transitions? #29. Like the previous bullet point, this is mandatory core functionality and not having that is also a showstopper for us.

@gplsteph : I will bump the internal support ticket with those elements, thanks for your help here.

@gplsteph
Copy link
Contributor

gplsteph commented Sep 23, 2019 via email

@unquietwiki
Copy link

@gplsteph @KelSolaar I spent some more time on #31 tonight. Still getting the same API error I had a few days ago (missing Jira field), but the code is getting a better linting score, and I think I discovered some stuff in jira_session & shotgun_session that might be causing the incomplete data syncs.

@KelSolaar
Copy link
Author

KelSolaar commented Sep 24, 2019

Hi,

I'm not sure which incomplete data sync you are referring to but as mentioned in #26 (comment), the bridge essentially works, whatever is missing for us is functionality at this stage.

Cheers,

Thomas

@unquietwiki
Copy link

@KelSolaar there was a bunch of TODO & FIXME items in the session code, and the linter found bad code return practices (I saw some sloppy-looking if/return mixes). The people where I'm at right now don't need this anymore; if you or @gplsteph want to try my branch out, I'll file the release form & help out more as able (I need the practice anyway). Thanks.

@KelSolaar
Copy link
Author

Closing this one as the original issues are solved.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants