Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update Create/CreateBeta method #634

Open
wants to merge 9 commits into
base: master
Choose a base branch
from
Open

Update Create/CreateBeta method #634

wants to merge 9 commits into from

Conversation

pzhao5
Copy link
Contributor

@pzhao5 pzhao5 commented Nov 2, 2018

Saving the following data into S3 so that lambda could process it.

  1. Data format as gensis_hash/data/file_index. File contains a BatchSize of data.
  2. Genesis Hash as gensis_hash/hash/file_index.
  3. Each data and genesis hash is 1:1 mapping. And there is not treasure within.
  4. Config data(single file) so that lambda knows how to process it.
  5. Treasure map data (single file).

When lambda attach to the data, it would first download config data and treasure map.
And then start to upload each data/genesis hash in pairs.
When arrive to a particular index that need to skip (for inserting the treasure), Lambda will then insert it with treasure rather than the data.

@pzhao5 pzhao5 requested a review from rfornea November 2, 2018 00:43
/*GetObjectKeyForData will return object key for particular data. startIndex is the smallest index in data. */
func GetObjectKeyForData(genesisHash string, startIndex int, totalCount int, isReserveIteration bool, batchSize int) string {
index := 0
if isReserveIteration {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit isReverseIteration

rfornea
rfornea previously approved these changes Nov 2, 2018
@rfornea
Copy link
Contributor

rfornea commented Nov 2, 2018

Went ahead and approved this but we should see if we can figure out what the deal is with Travis

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants