Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create a Dagster asset that fetches npm downloads #2436

Open
Jabolol opened this issue Oct 30, 2024 · 0 comments
Open

Create a Dagster asset that fetches npm downloads #2436

Jabolol opened this issue Oct 30, 2024 · 0 comments
Assignees

Comments

@Jabolol
Copy link
Contributor

Jabolol commented Oct 30, 2024

What is it?

The dlt connector is already implemented on #2435. Next step would be to get the list of all npm packages we want to index (maybe from ossd?), and wire it together as follows:

```python
# NPM was launched on January 12, 2010
NPM_EPOCH = "2010-01-12T00:00:00Z"
@dlt_factory(
key_prefix="npm",
partitions_def=WeeklyPartitionsDefinition(
start_date=NPM_EPOCH.split("T", maxsplit=1)[0],
end_offset=1,
),
)
def downloads(
context: AssetExecutionContext,
):
yield get_all_downloads(
context,
package_names=[
"@angular/core",
"react",
"vue",
# ...
],
)
```

@Jabolol Jabolol self-assigned this Oct 30, 2024
@github-project-automation github-project-automation bot moved this to Backlog in OSO Oct 30, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Backlog
Development

No branches or pull requests

1 participant