FAI-14070/FAI-14071: Add use_board_ownership flag to Jira Source #1908

matiaslcoulougian · 2025-02-04T13:34:33Z

Description

We are adding use_board_ownership flag, like the one we had on the feed, to be able to keep using it with 1 board per project.

Default behavior is use_board_ownership:true, as it was working as today.

When flag is set to false:

Board list is populated with project uids
Project-board relation created 1-1 with project uid.
The board and board_issues streams are removed from list.
Task-board relationship is written on faros_issues stream using project uid.
Sprints and sprint history records are associated to the project board. If the same sprint is in more than one Jira Board, it will override it with the last one fetched.

TODO

Add more tests
Refactor to reduce duplication

Type of change

Bug fix
New feature
Breaking change

Related issues

Fix #1

Migration notes

Describe migration notes if any

Extra info

Add any additional information

…haviour is true (as it was working as today) When flag is set to false: - Board list is populated with project uids - Project-board relation created 1-1 with project uid. - The board_issues stream is removed from list. - Task-board relationship is written on faros_issues stream using project uid.

chalenge · 2025-02-04T13:58:55Z

Hmmm, did we say we wanna re-add this? We had previously explicitly. Also we that other dummy board

chalenge

How are sprint report boards being handled, are they all gonna be phantoms. Can we add some tests to validate these mappings

sources/jira-source/src/streams/faros_boards.ts

sources/jira-source/src/jira.ts

sources/jira-source/src/project-board-filter.ts

destinations/airbyte-faros-destination/src/converters/jira/faros_issues.ts

matiaslcoulougian · 2025-02-06T13:53:46Z

How are sprint report boards being handled, are they all gonna be phantoms. Can we add some tests to validate these mappings

Didn't finish that implementation yet. Trying to think the best approach. The impl. on feeds leaves all boards as phantoms, but then they get linked to the project dummy board on the reports. Maybe we could link them directly to the project board on the graph...
And we would write only one board sprint per project, hoping they don't differ that much.

…ecated. Refactor logic and other fixes.

matiaslcoulougian · 2025-02-06T15:16:52Z

How are sprint report boards being handled, are they all gonna be phantoms. Can we add some tests to validate these mappings

Didn't finish that implementation yet. Trying to think the best approach. The impl. on feeds leaves all boards as phantoms, but then they get linked to the project dummy board on the reports. Maybe we could link them directly to the project board on the graph... And we would write only one board sprint per project, hoping they don't differ that much.

The issue here is that we are using board slices for sprint and sprint reports, to query sprints by board id. But here the board slices will contain the project key... So I'm not sure if we should query sprints for all project's boards all together for this case and if that would be performant also.

# Conflicts: # destinations/airbyte-faros-destination/test/__snapshots__/index.test.ts.snap

chalenge

See comments. Can we validate feeds works with current destination

sources/jira-source/src/index.ts

sources/jira-source/src/jira.ts

destinations/airbyte-faros-destination/resources/source-specific-configs/jira.json

destinations/airbyte-faros-destination/src/converters/jira/common.ts

sources/jira-source/src/project-board-filter.ts

- Support Faros Graph inclusion, handling the case where board or excluded boards are provided by mistake.

…se project as boards mode.

- Add missing tms_TaskBoardProjectRelationship

- Re-add comments - Refactor to functions constructor filtering lists logic

chalenge

Looks good, see comments.

chalenge · 2025-02-25T15:29:13Z

sources/jira-source/src/project-board-filter.ts

+  protected verifyListsWithGraphSelection(
+    items: ReadonlyArray<string>,
+    excludedItems: ReadonlyArray<string>,
+    key: 'boards' | 'projects' = 'boards'


when do we ever fetch projects? We only fetch selected boards from the graph

We don't fetch projects. But we fetch boards that are actually projects and that we need to verify against config project list and use as project filter config, when use_projects_as_boards is enabled.

chalenge · 2025-02-25T15:36:23Z

sources/jira-source/src/project-filter.ts

+  ) {
+    super(config, logger, farosClient);
+
+    const {items: projects, excludedItems: excluded_projects} =


this feels a little more complicated and we can simplify this. I am thinking when

projects_as_boards is selected, ignore board list completely. filterConfig.projects = projects.

projects_as_boards is selected and use_faros_graph, ignore initial projects config too. then filterConfig.projects === boards come from the graph via selected boards
WYDT?

I'm not understanding how the simplification would be.
The check inside the constructor would be if use_faros_graph is enabled, ignore projects and excluded_projects config.
And if use_faros_graph is not enabled, we need to check that only projects or excluded_projects are defined and not both.
That is basically what is being done inside verifyListsWithGraphSelection function.

chalenge · 2025-02-25T15:43:31Z

sources/jira-source/src/project-board-filter.ts


      const jira = await Jira.instance(this.config, this.logger);
      if (!this.filterConfig.projects?.size) {
        const projects =
-          this.isWebhookSupplementMode() && this.hasFarosClient()
+          this.isWebhookSupplementMode && this.hasFarosClient()


we should prolly move this out too in next PR

Did this change because if we leave the RunMode enum import from common.ts as it was before, we end up with a circular dependency between imports: project-board-filter -> common -> project-filter -> project-board-filter.

chalenge · 2025-02-25T15:44:47Z

sources/jira-source/src/project-filter.ts

+
+      const jira = await Jira.instance(this.config, this.logger);
+
+      if (!this.filterConfig.projects?.size) {


looks awry similar to the other method. Okay for now but should look to generalize

destinations/airbyte-faros-destination/test/converters/__snapshots__/jira.test.ts.snap

sonarqubecloud · 2025-02-25T18:27:26Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

matiaslcoulougian requested review from chalenge, dbruno21 and ypc-faros February 4, 2025 13:34

matiaslcoulougian requested review from cjwooo and willmarks as code owners February 4, 2025 13:34

chalenge requested changes Feb 6, 2025

View reviewed changes

Rename flag to use_projects_as_boards. Add spec config marked as depr…

23d14d8

…ecated. Refactor logic and other fixes.

matiaslcoulougian added 2 commits February 7, 2025 11:37

Add use_boards_as_projects logic to sprint and sprint_report streams

355beb3

Rename function

c48a1e0

matiaslcoulougian requested a review from chalenge February 7, 2025 14:56

matiaslcoulougian added 6 commits February 7, 2025 13:00

Add tests

72ec3eb

Remove unnecessary mock

44c393f

Remove unnecessary mock

e4fd641

Remove wrong inclusion check

00ce4b0

Merge branch 'main' into mc/jira-use-board-ownership

746078b

Fix tests

79b218b

matiaslcoulougian force-pushed the mc/jira-use-board-ownership branch from a32b541 to 79b218b Compare February 18, 2025 14:41

matiaslcoulougian added 5 commits February 18, 2025 12:17

Rename destination flag to use_projects_as_boards

5e92ecf

Default use project as boards false

5dd7018

update snap

ed7ca09

Merge branch 'main' into mc/jira-use-board-ownership

7a40419

# Conflicts: # destinations/airbyte-faros-destination/test/__snapshots__/index.test.ts.snap

update snap

6ecee21

chalenge reviewed Feb 20, 2025

View reviewed changes

matiaslcoulougian added 3 commits February 20, 2025 11:54

- Fix/remove comments.

2771b02

- Support Faros Graph inclusion, handling the case where board or excluded boards are provided by mistake.

Support project inclusion and add separate class project-filter for u…

2efd2ca

…se project as boards mode.

- Add more tests

86a25e2

- Add missing tms_TaskBoardProjectRelationship

matiaslcoulougian added 4 commits February 24, 2025 13:35

Add destination tests.

e13bc24

Refactor to reduce duplication

9b71e12

Unused imports

171d77e

- Fix type

1571707

- Re-add comments - Refactor to functions constructor filtering lists logic

matiaslcoulougian requested a review from chalenge February 24, 2025 20:38

matiaslcoulougian added 2 commits February 24, 2025 17:38

Unused import

c027346

Fix isWebhookSupplementMode check

f4fe841

chalenge approved these changes Feb 25, 2025

View reviewed changes

Fix jira test config to leave it as it was

2cd1e4c

matiaslcoulougian merged commit 70cb986 into main Feb 26, 2025
8 checks passed

github-actions bot locked and limited conversation to collaborators Feb 26, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

FAI-14070/FAI-14071: Add use_board_ownership flag to Jira Source #1908

FAI-14070/FAI-14071: Add use_board_ownership flag to Jira Source #1908

matiaslcoulougian commented Feb 4, 2025 •

edited

Loading

chalenge commented Feb 4, 2025

chalenge left a comment

matiaslcoulougian commented Feb 6, 2025 •

edited

Loading

matiaslcoulougian commented Feb 6, 2025

chalenge left a comment

chalenge left a comment

chalenge Feb 25, 2025

matiaslcoulougian Feb 25, 2025

chalenge Feb 25, 2025

matiaslcoulougian Feb 25, 2025 •

edited

Loading

chalenge Feb 25, 2025

matiaslcoulougian Feb 25, 2025 •

edited

Loading

chalenge Feb 25, 2025

sonarqubecloud bot commented Feb 25, 2025


		const jira = await Jira.instance(this.config, this.logger);

		if (!this.filterConfig.projects?.size) {

FAI-14070/FAI-14071: Add use_board_ownership flag to Jira Source #1908

FAI-14070/FAI-14071: Add use_board_ownership flag to Jira Source #1908

Conversation

matiaslcoulougian commented Feb 4, 2025 • edited Loading

Description

Type of change

Related issues

Migration notes

Extra info

chalenge commented Feb 4, 2025

chalenge left a comment

Choose a reason for hiding this comment

matiaslcoulougian commented Feb 6, 2025 • edited Loading

matiaslcoulougian commented Feb 6, 2025

chalenge left a comment

Choose a reason for hiding this comment

chalenge left a comment

Choose a reason for hiding this comment

chalenge Feb 25, 2025

Choose a reason for hiding this comment

matiaslcoulougian Feb 25, 2025

Choose a reason for hiding this comment

chalenge Feb 25, 2025

Choose a reason for hiding this comment

matiaslcoulougian Feb 25, 2025 • edited Loading

Choose a reason for hiding this comment

chalenge Feb 25, 2025

Choose a reason for hiding this comment

matiaslcoulougian Feb 25, 2025 • edited Loading

Choose a reason for hiding this comment

chalenge Feb 25, 2025

Choose a reason for hiding this comment

sonarqubecloud bot commented Feb 25, 2025

Quality Gate passed

matiaslcoulougian commented Feb 4, 2025 •

edited

Loading

matiaslcoulougian commented Feb 6, 2025 •

edited

Loading

matiaslcoulougian Feb 25, 2025 •

edited

Loading

matiaslcoulougian Feb 25, 2025 •

edited

Loading