Archive Deals Insight #412

asutula · 2020-11-19T01:14:00Z

Adds DealInfo[] to the Mongo Archive object
Populates Archive.DealInfo[] every time we poll Powergate for job status
Exposes current and historical Archive data via a new/simplified API
Uses configurable "slow" and "fast" job polling intervals to periodically check job status with the interval depending on whether or not the archive contains deals that are sealing or not.

Signed-off-by: Aaron Sutula <[email protected]>

…ls-api Signed-off-by: Aaron Sutula <[email protected]>

Signed-off-by: Aaron Sutula <[email protected]>

…ls-api Signed-off-by: Aaron Sutula <[email protected]>

Signed-off-by: Aaron Sutula <[email protected]>

…ls-api Signed-off-by: Aaron Sutula <[email protected]>

Makefile

asutula · 2020-11-25T01:41:23Z

api/bucketsd/client/client.go

-		Key: key,
-	})
+// Archives returns information about current and historical archives.
+func (c *Client) Archives(ctx context.Context, key string) (*pb.ArchivesResponse, error) {


Combining those two old methods into one here. Archives returns an object that looks like:

{ Current: nil|Archive, History: Archive[] }

If the bucket has never been archived, both those values are the nil/empty values.
If the bucket has an archive currently processing, it will be listed in Current and is updates in near-real-time as new deal information is available. The previously completed archive is moved to the first slot in History.

I added this comment here in the client, but the change is reflected all the way through the stack.

asutula · 2020-11-25T01:43:47Z

buckets/archive/tracker.go

@@ -22,8 +23,7 @@ const (
 )

 var (
-	CheckInterval         = time.Second * 15
-	JobStatusPollInterval = time.Minute * 30


This is now configured and passed in here

asutula · 2020-11-25T01:45:07Z

buckets/archive/tracker.go

@@ -169,11 +177,10 @@ func (t *Tracker) trackArchiveProgress(
 	jid string,
 	bucketRoot cid.Cid,
 	powInfo *mdb.PowInfo,
-) (bool, string, error) {
+) (time.Duration, string, error) {


This now returns the duration that should be used for rescheduling. It will be different depending on if we have any deals in sealing state or not. 0 is interpreted as "don't reschedule".

asutula · 2020-11-25T01:45:34Z

buckets/archive/tracker.go

-
-	if !isJobStatusFinal(res.StorageJob.Status) {
-		return true, "no final status yet", nil
+		return t.jobPollIntervalSlow, fmt.Sprintf("getting current job %s for bucket %s: %s", jid, buckKey, err), nil


For recoverable errors, we reschedule with the "slow" interval.

asutula · 2020-11-25T01:46:00Z

buckets/archive/tracker.go

+		return t.jobPollIntervalSlow, fmt.Sprintf("updating archive status: %s", err), nil
+	}
+
+	rescheduleDuration := t.jobPollIntervalFast


Use the fast interval by default.

asutula · 2020-11-25T01:46:19Z

buckets/archive/tracker.go

+	message := "non-final status"
+	if isJobStatusFinal(res.StorageJob.Status) {
+		message = "reached final status"
+		rescheduleDuration = 0


If the job is final, use duration 0 meaning "don't reschedule"

asutula · 2020-11-25T01:48:00Z

buckets/archive/tracker.go

+
+	if rescheduleDuration > 0 {
+		for _, dealInfo := range res.StorageJob.DealInfo {
+			if dealInfo.StateId == storagemarket.StorageDealSealing {


If we are going to reschedule, inspect the deal states and decide if we should use short or long duration.

I really hate the use of storagemarkets here, but it's necessary since we aren't modeling those states ourself at any point in the stack. Maybe we should so we can avoid this and have typed states for deals.

Agree. We should eventually have our own states in the API.
Even if we use storagemarket here, there's the chance that storagemarket version used in Hub might diverge from the one returned by Powergate, if both are running different versions that doesn't break the API. (In fact, that was already the case for a long time since we didn't break APIs in Powergate and didn't update the Powergate client in Hub).

To be fair, I see a really tiny chance that the market's code will be changing the numbers of statuses, so isn't a panicking situation. But I completely agree that the API should be self-contained... not only because is nicer, but because is safer.

Might worth creating an issue?

Yes, done: textileio/powergate#715

asutula · 2020-11-25T01:48:28Z

buckets/archive/tracker.go

 	if err := t.colls.BucketArchives.Replace(ctx, ba); err != nil {
 		return fmt.Errorf("updating bucket archives status: %s", err)
 	}
 	return nil
 }

-func (t *Tracker) saveDealsInArchive(
+func (t *Tracker) saveDealsInBucket(


Rename for clarity.

asutula · 2020-11-25T01:51:59Z

buckets/local/archive.go

+}
+
+// ArchiveWatch delivers messages about the archive status.
+func (b *Bucket) ArchiveWatch(ctx context.Context, watch bool) (<-chan ArchiveStatusMessage, error) {


I decided to simplify the old pattern of "get status optionally watch by providing a flag" because the types of data returned by status (now Archives) and ArchiveWatch are very different.

asutula · 2020-11-25T05:11:05Z

cmd/hubd/main.go

+			},
+			"archivesJobPollIntervalFast": {
+				Key:      "archives.job_poll_interval_fast",
+				DefValue: time.Minute * 15,


Poll interval flags and their default values.

asutula · 2020-11-25T05:13:55Z

mongodb/bucketarchives.go

+	AbortedMsg string     `bson:"aborted_msg"`
+	FailureMsg string     `bson:"failure_msg"`
+	CreatedAt  int64      `bson:"created_at"`
+	DealInfo   []DealInfo `bson:"deal_info"`


New model to hold deal information. This would just unmarshal to an empty array for pre-existing archives I think? Will look at writing a test for that. Also, maybe we want to consider writing a migration for those pre-existing archives. Not sure how complex that could be.

Yeah, I'd think the same: unmarshal to empty array for existing things.

Regarding the migration, could be. We could wait a bit to see if existing archives really start asking for this data to be populated, and if justified do it?

Sounds like a plan to me.

Signed-off-by: Aaron Sutula <[email protected]>

jsign

LGTM!

Makefile

api/bucketsd/pb/bucketsd.proto

jsign · 2020-11-25T12:24:37Z

buckets/archive/tracker.go

@@ -22,8 +23,7 @@ const (
 )

 var (
-	CheckInterval         = time.Second * 15
-	JobStatusPollInterval = time.Minute * 30


jsign · 2020-11-25T12:30:09Z

buckets/archive/tracker.go

+
+	if rescheduleDuration > 0 {
+		for _, dealInfo := range res.StorageJob.DealInfo {
+			if dealInfo.StateId == storagemarket.StorageDealSealing {


Agree. We should eventually have our own states in the API.
Even if we use storagemarket here, there's the chance that storagemarket version used in Hub might diverge from the one returned by Powergate, if both are running different versions that doesn't break the API. (In fact, that was already the case for a long time since we didn't break APIs in Powergate and didn't update the Powergate client in Hub).

To be fair, I see a really tiny chance that the market's code will be changing the numbers of statuses, so isn't a panicking situation. But I completely agree that the API should be self-contained... not only because is nicer, but because is safer.

Might worth creating an issue?

jsign · 2020-11-25T12:38:59Z

integrationtest/pg/pg_test.go

+	conf.ArchiveJobPollIntervalFast = time.Second * 5
+	conf.ArchiveJobPollIntervalSlow = time.Second * 10


You scaled-down respecting semantics :P

Haha of course!

jsign · 2020-11-25T12:45:32Z

mongodb/bucketarchives.go

+	AbortedMsg string     `bson:"aborted_msg"`
+	FailureMsg string     `bson:"failure_msg"`
+	CreatedAt  int64      `bson:"created_at"`
+	DealInfo   []DealInfo `bson:"deal_info"`


Yeah, I'd think the same: unmarshal to empty array for existing things.

Regarding the migration, could be. We could wait a bit to see if existing archives really start asking for this data to be populated, and if justified do it?

Signed-off-by: Aaron Sutula <[email protected]>

jimmylee · 2020-12-02T00:12:51Z

💯

asutula added 7 commits November 18, 2020 19:11

simplify archive api, add deal information

087b825

Signed-off-by: Aaron Sutula <[email protected]>

Merge remote-tracking branch 'origin/master' into asutula/archive-dea…

9528ae2

…ls-api Signed-off-by: Aaron Sutula <[email protected]>

remove processing archive

607952b

Signed-off-by: Aaron Sutula <[email protected]>

Merge remote-tracking branch 'origin/master' into asutula/archive-dea…

7dfc20e

…ls-api Signed-off-by: Aaron Sutula <[email protected]>

cleaning up, better cli

d9cec95

Signed-off-by: Aaron Sutula <[email protected]>

use configurable poll intervals based on deal state

17c61b2

Signed-off-by: Aaron Sutula <[email protected]>

Merge remote-tracking branch 'origin/master' into asutula/archive-dea…

e621571

…ls-api Signed-off-by: Aaron Sutula <[email protected]>

asutula commented Nov 25, 2020

View reviewed changes

Makefile Show resolved Hide resolved

asutula commented Nov 25, 2020

View reviewed changes

asutula requested review from jsign and sanderpick November 25, 2020 05:19

asutula self-assigned this Nov 25, 2020

asutula marked this pull request as ready for review November 25, 2020 05:20

asutula mentioned this pull request Nov 25, 2020

Bucket Archive API Update textileio/js-textile#718

Closed

test fix

23aaab6

Signed-off-by: Aaron Sutula <[email protected]>

jsign approved these changes Nov 25, 2020

View reviewed changes

fix archive watch client

16a1580

Signed-off-by: Aaron Sutula <[email protected]>

asutula merged commit 2e8cb77 into master Nov 30, 2020

asutula deleted the asutula/archive-deals-api branch November 30, 2020 22:48

This was referenced Jan 5, 2021

[https://slate.host] Archive: buckets.archive needs to return useful information for third-party groups. #385

Closed

Textile & Filecoin integration tasks filecoin-project/slate#410

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Archive Deals Insight #412

Archive Deals Insight #412

asutula commented Nov 19, 2020 •

edited

Loading

asutula Nov 25, 2020

asutula Nov 25, 2020

asutula Nov 25, 2020

jsign Nov 25, 2020

asutula Nov 25, 2020

asutula Nov 25, 2020

asutula Nov 25, 2020

asutula Nov 25, 2020

asutula Nov 25, 2020

jsign Nov 25, 2020

asutula Nov 25, 2020

asutula Nov 25, 2020

asutula Nov 25, 2020

asutula Nov 25, 2020 •

edited

Loading

asutula Nov 25, 2020

jsign Nov 25, 2020

asutula Nov 25, 2020

jsign left a comment

jsign Nov 25, 2020

jsign Nov 25, 2020

jsign Nov 25, 2020

asutula Nov 25, 2020

jsign Nov 25, 2020

jimmylee commented Dec 2, 2020

		conf.ArchiveJobPollIntervalFast = time.Second * 5
		conf.ArchiveJobPollIntervalSlow = time.Second * 10

Archive Deals Insight #412

Archive Deals Insight #412

Conversation

asutula commented Nov 19, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

asutula Nov 25, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jsign left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jimmylee commented Dec 2, 2020

asutula commented Nov 19, 2020 •

edited

Loading

asutula Nov 25, 2020 •

edited

Loading