Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Payload optimization #19

Closed
0x4007 opened this issue Oct 27, 2024 · 8 comments · Fixed by #21
Closed

Payload optimization #19

0x4007 opened this issue Oct 27, 2024 · 8 comments · Fixed by #21

Comments

@0x4007
Copy link
Member

0x4007 commented Oct 27, 2024

This is an example of a diff q&a gone bad. Too much was sent to the model and it was rate limited. Below describes a dynamic strategy that should be simple to implement and should intelligently include only the most relevant changes.

ubiquity-os-marketplace/text-conversation-rewards#163 (comment)

Copy link

ubiquity-os-beta bot commented Oct 28, 2024

 [ 23.813 WXDAI ] 

@0x4007
Contributions Overview
ViewContributionCountReward
IssueSpecification19.18
ReviewComment1014.633
Conversation Incentives
CommentFormattingRelevanceReward
This is an example of a diff q&a gone bad. Too much was sent…
3.06
content:
  content:
    p:
      score: 0
      elementCount: 1
  result: 0
regex:
  wordCount: 56
  wordValue: 0.1
  result: 3.06
19.18
Perhaps my spec isnt clear but youre not following it. - Remove…
5.97
content:
  content:
    p:
      score: 0
      elementCount: 1
    ul:
      score: 1
      elementCount: 1
    li:
      score: 0.5
      elementCount: 4
  result: 3
regex:
  wordCount: 54
  wordValue: 0.1
  result: 2.97
0.95.673
Should definitely be dynamic from git ignore and such. Otherwise…
1.11
content:
  content:
    p:
      score: 0
      elementCount: 1
  result: 0
regex:
  wordCount: 17
  wordValue: 0.1
  result: 1.11
0.80.888
Wrong approach. Should not be any hard cut off from some arbitra…
1.06
content:
  content:
    p:
      score: 0
      elementCount: 1
  result: 0
regex:
  wordCount: 16
  wordValue: 0.1
  result: 1.06
0.70.742
Hey i just realized this is pointless. If its ignored it wont b…
2.2
content:
  content:
    p:
      score: 0
      elementCount: 1
  result: 0
regex:
  wordCount: 38
  wordValue: 0.1
  result: 2.2
0.91.98
Get rid of all this logic too
0.52
content:
  content:
    p:
      score: 0
      elementCount: 1
  result: 0
regex:
  wordCount: 7
  wordValue: 0.1
  result: 0.52
0.60.312
Max i guess config so that partners can budget their token use.
0.83
content:
  content:
    p:
      score: 0
      elementCount: 1
  result: 0
regex:
  wordCount: 12
  wordValue: 0.1
  result: 0.83
0.40.332
Default should be the max for the model we are using? 10k seems …
0.94
content:
  content:
    p:
      score: 0
      elementCount: 1
  result: 0
regex:
  wordCount: 14
  wordValue: 0.1
  result: 0.94
0.50.47
This could have been worded better. I meant that it would automa…
5.01
content:
  content:
    p:
      score: 0
      elementCount: 1
  result: 0
regex:
  wordCount: 100
  wordValue: 0.1
  result: 5.01
0.63.006
Alright so then it seems clear to me that we can measure the dif…
1.38
content:
  content:
    p:
      score: 0
      elementCount: 1
  result: 0
regex:
  wordCount: 22
  wordValue: 0.1
  result: 1.38
0.70.966
I think that's all you need to do. What is the problem?
0.88
content:
  content:
    p:
      score: 0
      elementCount: 1
  result: 0
regex:
  wordCount: 13
  wordValue: 0.1
  result: 0.88
0.30.264

 [ 3.05625 WXDAI ] 

@Keyrxng
Contributions Overview
ViewContributionCountReward
ReviewComment93.05625
Conversation Incentives
CommentFormattingRelevanceReward
These `pulls` rely on searching the repo for all PRs the…
7.51
content:
  content:
    p:
      score: 0
      elementCount: 3
  result: 0
regex:
  wordCount: 161
  wordValue: 0.1
  result: 7.51
0.91.69225
Is there a specific reason that you change this? The template us…
1.7
content:
  content:
    p:
      score: 0
      elementCount: 1
  result: 0
regex:
  wordCount: 28
  wordValue: 0.1
  result: 1.7
0.30.1325
should move this into `config` and pass this via the yml…
0.77
content:
  content:
    p:
      score: 0
      elementCount: 1
  result: 0
regex:
  wordCount: 11
  wordValue: 0.1
  result: 0.77
0.80.1515
Same here. Should also be renamed to `MAX_COMPLETION_TOKENS&…
0.52
content:
  content:
    p:
      score: 0
      elementCount: 1
  result: 0
regex:
  wordCount: 7
  wordValue: 0.1
  result: 0.52
0.50.065
This lib is only called twice and can be done just as simply wit…
2.59
content:
  content:
    p:
      score: 0
      elementCount: 1
  result: 0
regex:
  wordCount: 46
  wordValue: 0.1
  result: 2.59
0.40.2615
Might be best to wrap this in a try-catch because if e.g the rep…
1.44
content:
  content:
    p:
      score: 0
      elementCount: 1
  result: 0
regex:
  wordCount: 23
  wordValue: 0.1
  result: 1.44
0.70.252
Could be better named as it's actually fetching the linked issue…
1.11
content:
  content:
    p:
      score: 0
      elementCount: 1
  result: 0
regex:
  wordCount: 17
  wordValue: 0.1
  result: 1.11
0.60.169
Might be best to add handling for url matching too
0.71
content:
  content:
    p:
      score: 0
      elementCount: 1
  result: 0
regex:
  wordCount: 10
  wordValue: 0.1
  result: 0.71
0.80.1445
The plugin-template comes with `:4000` as standard so it…
3.66
content:
  content:
    p:
      score: 0
      elementCount: 2
  result: 0
regex:
  wordCount: 69
  wordValue: 0.1
  result: 3.66
0.20.188

 [ 0.156 WXDAI ] 

@gentlementlegen
Contributions Overview
ViewContributionCountReward
ReviewComment10.156
Conversation Incentives
CommentFormattingRelevanceReward
Does it combine this + gitignore?
0.39
content:
  content:
    p:
      score: 0
      elementCount: 1
  result: 0
regex:
  wordCount: 5
  wordValue: 0.1
  result: 0.39
0.40.156

@0x4007
Copy link
Member Author

0x4007 commented Oct 28, 2024

Assign isn't working on pull linked i thought we fixed this @gentlementlegen

Copy link

@sshivaditya2019 the deadline is at Mon, Oct 28, 5:11 AM UTC

@0x4007 0x4007 reopened this Oct 28, 2024
@0x4007 0x4007 closed this as completed Oct 28, 2024
Copy link

ubiquity-os-beta bot commented Oct 28, 2024

 [ 150 WXDAI ] 

@sshivaditya2019
Contributions Overview
ViewContributionCountReward
IssueTask1150
ReviewComment150
Conversation Incentives
CommentFormattingRelevanceReward
Resolves #19- Introduces a method to selectively retrieve diff…
2
content:
  content:
    p:
      score: 0
      elementCount: 3
    ul:
      score: 1
      elementCount: 1
    li:
      score: 0.5
      elementCount: 2
  result: 2
regex:
  wordCount: 22
  wordValue: 0
  result: 0
0.80
It uses the gitignore from the Repository if one is present.
0
content:
  content:
    p:
      score: 0
      elementCount: 1
  result: 0
regex:
  wordCount: 11
  wordValue: 0
  result: 0
0.50
Revised the approach:- Sorts the diffs in ascending order base…
2
content:
  content:
    p:
      score: 0
      elementCount: 4
    ul:
      score: 1
      elementCount: 1
    li:
      score: 0.5
      elementCount: 2
  result: 2
regex:
  wordCount: 38
  wordValue: 0
  result: 0
0.90
Switched to GraphQL to directly fetch pull requests that close i…
0
content:
  content:
    p:
      score: 0
      elementCount: 1
  result: 0
regex:
  wordCount: 37
  wordValue: 0
  result: 0
0.70
Is port 4000 required by some other service/kernel? I'm currentl…
0
content:
  content:
    p:
      score: 0
      elementCount: 1
  result: 0
regex:
  wordCount: 22
  wordValue: 0
  result: 0
0.40
Not applicable - replaced with GraphQL solution
0
content:
  content:
    p:
      score: 0
      elementCount: 1
  result: 0
regex:
  wordCount: 6
  wordValue: 0
  result: 0
0.60
It used to, I have removed all of that now. It just checks for f…
0
content:
  content:
    p:
      score: 0
      elementCount: 1
  result: 0
regex:
  wordCount: 19
  wordValue: 0
  result: 0
0.50
10k seemed like a reasonable limit. If someone is using older Op…
0
content:
  content:
    p:
      score: 0
      elementCount: 1
  result: 0
regex:
  wordCount: 26
  wordValue: 0
  result: 0
0.30
QA: [issue](https://github.com/sshivaditya2019/test-public/issue…
5
content:
  content:
    p:
      score: 0
      elementCount: 2
    a:
      score: 5
      elementCount: 1
  result: 5
regex:
  wordCount: 13
  wordValue: 0
  result: 0
0.10
QA: [Issue](https://github.com/sshivaditya2019/test-public/issue…
5
content:
  content:
    p:
      score: 0
      elementCount: 1
    a:
      score: 5
      elementCount: 1
  result: 5
regex:
  wordCount: 2
  wordValue: 0
  result: 0
0.10
I think that counting the number of changes isn't the best metri…
5
content:
  content:
    p:
      score: 0
      elementCount: 2
    a:
      score: 5
      elementCount: 1
  result: 5
regex:
  wordCount: 47
  wordValue: 0
  result: 0
0.70
It seems to use file size as a criterion to determine whether to…
0
content:
  content:
    p:
      score: 0
      elementCount: 2
  result: 0
regex:
  wordCount: 89
  wordValue: 0
  result: 0
0.60
@0x4007 I need some clarification. Currently, the system retriev…
0
content:
  content:
    p:
      score: 0
      elementCount: 2
  result: 0
regex:
  wordCount: 97
  wordValue: 0
  result: 0
0.80
Ok, I see now that I misunderstood these lines. I think this PR …
0
content:
  content:
    p:
      score: 0
      elementCount: 1
  result: 0
regex:
  wordCount: 25
  wordValue: 0
  result: 0
0.50
QA: [Link](https://github.com/sshivaditya2019/test-public/issues…
5
content:
  content:
    p:
      score: 0
      elementCount: 1
    a:
      score: 5
      elementCount: 1
  result: 5
regex:
  wordCount: 2
  wordValue: 0
  result: 0
0.10

 [ 25.327 WXDAI ] 

@0x4007
Contributions Overview
ViewContributionCountReward
IssueSpecification19.18
IssueComment10.264
ReviewComment1015.883
Conversation Incentives
CommentFormattingRelevanceReward
This is an example of a diff q&a gone bad. Too much was sent…
3.06
content:
  content:
    p:
      score: 0
      elementCount: 1
  result: 0
regex:
  wordCount: 56
  wordValue: 0.1
  result: 3.06
19.18
Assign isn't working on pull linked i thought we fixed this @gen…
0.88
content:
  content:
    p:
      score: 0
      elementCount: 1
  result: 0
regex:
  wordCount: 13
  wordValue: 0.1
  result: 0.88
0.30.264
Perhaps my spec isnt clear but youre not following it. - Remove…
5.97
content:
  content:
    p:
      score: 0
      elementCount: 1
    ul:
      score: 1
      elementCount: 1
    li:
      score: 0.5
      elementCount: 4
  result: 3
regex:
  wordCount: 54
  wordValue: 0.1
  result: 2.97
15.97
Should definitely be dynamic from git ignore and such. Otherwise…
1.11
content:
  content:
    p:
      score: 0
      elementCount: 1
  result: 0
regex:
  wordCount: 17
  wordValue: 0.1
  result: 1.11
0.80.888
Wrong approach. Should not be any hard cut off from some arbitra…
1.06
content:
  content:
    p:
      score: 0
      elementCount: 1
  result: 0
regex:
  wordCount: 16
  wordValue: 0.1
  result: 1.06
0.60.636
Hey i just realized this is pointless. If its ignored it wont b…
2.2
content:
  content:
    p:
      score: 0
      elementCount: 1
  result: 0
regex:
  wordCount: 38
  wordValue: 0.1
  result: 2.2
0.91.98
Get rid of all this logic too
0.52
content:
  content:
    p:
      score: 0
      elementCount: 1
  result: 0
regex:
  wordCount: 7
  wordValue: 0.1
  result: 0.52
0.70.364
Max i guess config so that partners can budget their token use.
0.83
content:
  content:
    p:
      score: 0
      elementCount: 1
  result: 0
regex:
  wordCount: 12
  wordValue: 0.1
  result: 0.83
0.30.249
Default should be the max for the model we are using? 10k seems …
0.94
content:
  content:
    p:
      score: 0
      elementCount: 1
  result: 0
regex:
  wordCount: 14
  wordValue: 0.1
  result: 0.94
0.50.47
This could have been worded better. I meant that it would automa…
5.01
content:
  content:
    p:
      score: 0
      elementCount: 1
  result: 0
regex:
  wordCount: 100
  wordValue: 0.1
  result: 5.01
0.84.008
Alright so then it seems clear to me that we can measure the dif…
1.38
content:
  content:
    p:
      score: 0
      elementCount: 1
  result: 0
regex:
  wordCount: 22
  wordValue: 0.1
  result: 1.38
0.70.966
I think that's all you need to do. What is the problem?
0.88
content:
  content:
    p:
      score: 0
      elementCount: 1
  result: 0
regex:
  wordCount: 13
  wordValue: 0.1
  result: 0.88
0.40.352

 [ 3.02275 WXDAI ] 

@Keyrxng
Contributions Overview
ViewContributionCountReward
ReviewComment93.02275
Conversation Incentives
CommentFormattingRelevanceReward
These `pulls` rely on searching the repo for all PRs the…
7.51
content:
  content:
    p:
      score: 0
      elementCount: 3
  result: 0
regex:
  wordCount: 161
  wordValue: 0.1
  result: 7.51
0.91.69225
Is there a specific reason that you change this? The template us…
1.7
content:
  content:
    p:
      score: 0
      elementCount: 1
  result: 0
regex:
  wordCount: 28
  wordValue: 0.1
  result: 1.7
0.30.1325
should move this into `config` and pass this via the yml…
0.77
content:
  content:
    p:
      score: 0
      elementCount: 1
  result: 0
regex:
  wordCount: 11
  wordValue: 0.1
  result: 0.77
0.70.13225
Same here. Should also be renamed to `MAX_COMPLETION_TOKENS&…
0.52
content:
  content:
    p:
      score: 0
      elementCount: 1
  result: 0
regex:
  wordCount: 7
  wordValue: 0.1
  result: 0.52
0.60.078
This lib is only called twice and can be done just as simply wit…
2.59
content:
  content:
    p:
      score: 0
      elementCount: 1
  result: 0
regex:
  wordCount: 46
  wordValue: 0.1
  result: 2.59
0.40.2615
Might be best to wrap this in a try-catch because if e.g the rep…
1.44
content:
  content:
    p:
      score: 0
      elementCount: 1
  result: 0
regex:
  wordCount: 23
  wordValue: 0.1
  result: 1.44
0.80.288
Could be better named as it's actually fetching the linked issue…
1.11
content:
  content:
    p:
      score: 0
      elementCount: 1
  result: 0
regex:
  wordCount: 17
  wordValue: 0.1
  result: 1.11
0.50.14125
Might be best to add handling for url matching too
0.71
content:
  content:
    p:
      score: 0
      elementCount: 1
  result: 0
regex:
  wordCount: 10
  wordValue: 0.1
  result: 0.71
0.60.109
The plugin-template comes with `:4000` as standard so it…
3.66
content:
  content:
    p:
      score: 0
      elementCount: 2
  result: 0
regex:
  wordCount: 69
  wordValue: 0.1
  result: 3.66
0.20.188

 [ 0.156 WXDAI ] 

@gentlementlegen
Contributions Overview
ViewContributionCountReward
ReviewComment10.156
Conversation Incentives
CommentFormattingRelevanceReward
Does it combine this + gitignore?
0.39
content:
  content:
    p:
      score: 0
      elementCount: 1
  result: 0
regex:
  wordCount: 5
  wordValue: 0.1
  result: 0.39
0.40.156

@gentlementlegen
Copy link
Member

gentlementlegen commented Oct 28, 2024

@0x4007 Seems to work: ubiquity-os-marketplace/text-conversation-rewards#77 (comment) ; there might be another reason for this.

@0x4007
Copy link
Member Author

0x4007 commented Oct 28, 2024

Could be the user configured privacy settings we discussed in another thread. We should throw a warning when we see the negative graphql response due to privacy settings.

@gentlementlegen
Copy link
Member

I will check the logs for details, hopefully more information is given there so we can determine the cause programmatically and take action.

@gentlementlegen
Copy link
Member

gentlementlegen commented Oct 28, 2024

Logs of the start stop
image

No POST was received meaning the event was never forwarded

Kernel logs
image

So it seems the fetch transport broke which is why it never got delivered. Seems tricky to reproduce, will have a look.

Error message:

Could not find a manifest for Worker https://ubiquity-os-telegram-bridge-development.ubiquity.workers.dev/manifest.json: Error: Cannot perform I/O on behalf of a different request. I/O objects (such as streams, request/response bodies, and others) created in the context of one request handler cannot be accessed from a different request's handler. This is a limitation of Cloudflare Workers which allows us to improve overall performance. (I/O type: SpanParent)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants