Skip to content

Commit

Permalink
FIX: ✨ BIG Update the "intro" section of the peer review guide ✨ (#146)
Browse files Browse the repository at this point in the history 
An overhaul of the peer review intro! yahoo
  • Loading branch information
@lwasser
lwasser authored Dec 21, 2022
1 parent 4a30126 commit e214523
Show file tree
Hide file tree
Showing 11 changed files with 273 additions and 142 deletions.
6 changes: 6 additions & 0 deletions _static/pyos.css
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,12 @@ div.header__block {
color: #222;
}

/* not working and not sure why */
.caption-text {

}


/*
.announcement div {
background-color: #ccc;
Expand Down
9 changes: 6 additions & 3 deletions about-peer-review/aims-and-scope.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -24,9 +24,12 @@ package.
support open science.

## Is Your Package in Scope For pyOpenSci Review?
pyOpenSci reviews packages within a set of categories define below.
If you are unsure whether your package fits into one of these categories, please
open a [pre-submission inquiry via a GitHub Issue](LINK) to get feedback from

pyOpenSci only reviews packages that fall within our specified domain and
technical scope listed below.

If you are unsure whether your package is in scope for review, please
open a [pre-submission inquiry using a GitHub Issue](https://github.com/pyOpenSci/software-review/issues/new?assignees=&labels=0%2Fpresubmission&template=presubmission-inquiry.md&title=) to get feedback from
one of our editors. We are happy to look at your package and help you understand
whether it is in scope or not.

Expand Down
8 changes: 8 additions & 0 deletions about-peer-review/code-of-conduct.md
View file
Original file line number Diff line number Diff line change
@@ -1,4 +1,12 @@
# Community Code of Conduct

We keep our Code of Conduct in our governance documentation. [Click here to
go there now.](https://www.pyopensci.org/governance/code-of-conduct)


## NOTE: we are in the process of moving this file to our governance documentation and making significant changes to our code of conduct.


- We are committed to providing a friendly, safe and welcoming environment for all, regardless of gender, sexual orientation, disability, ethnicity, religion, or similar personal characteristic.
- Please avoid using openly sexual nicknames or other nicknames that might detract from a friendly, safe and welcoming environment for all.
- Please be kind and courteous. There’s no need to be mean or rude.
Expand Down
49 changes: 49 additions & 0 deletions about-peer-review/how-peer-review-works.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
# How pyOpenSci's open software peer review works


## Who submits packages and who runs the reviews

pyOpenSci's [suite of packages](https://www.pyopensci.org/python-packages/) are
contributed by community members with a great diversity of skills and backgrounds. This diversity
of developer backgrounds enables us to vet and promote a broad ecosystem of
high quality tools
that supports scientists across domains with a suite of different data
types and structures.

### Who reviews pyOpenSci packages ?

Our peer review process is run by volunteer members of the Python scientific
community:

* Editors manage incoming package submissions. Editors also ensure
that package reviews move through the review process efficiently;
* Authors create, submit and improve their package;
* Reviewers, two per submission, examine the software code and user experience.

Our [governance documentation](https://www.pyopensci.org/governance) clarifies
the various roles that support running our peer review process.

## Software reviews are contained within [GitHub](https://www.github.com/pyOpenSci) issues

Our entire peer review process occurs on GitHub in the
[pyOpenSci software-review repository](https://www.github.com/pyopensci/software-review).

We use GitHub because:

* It is free to create an account
* Anyone can read the review discussion without an account making the process entirely open
* It facilitates collaboration and supports community around a package
* It facilitates open discussion between reviewers and package maintainers and the pyOpenSci volunteers
* Numerous packages store their code bases on GitHub

### We use GitHub issue templates and labels to organize the review steps

* We use GitHub issue templates as submission templates for new reviews and pre-submission review questions.
* We label issues to track every step of the package submission and review progress (e.g. [1/initial-editor-checks, 2/reviewers-needed, 6/pyopensci-approved](https://github.com/pyOpenSci/software-review/labels)

```{note}
Click [here](https://github.com/ropensci/software-review/issues/24) to read the review thread from an rOpenSci review of the `ropenaq` package. Note that the process is an ongoing conversation until the package is accepted. Two external reviews are important milestones in the review process.
```

For more detailed overview of the peer review process, [check out our process
timeline document here.](../software-peer-review-guide/intro.md)
180 changes: 50 additions & 130 deletions about-peer-review/intro.md
View file
Original file line number Diff line number Diff line change
@@ -1,124 +1,72 @@
# How Peer Review Works
# About the pyOpenSci peer review process

```{tableofcontents}
```

## Why do we need pyOpenSci peer review?
## What is software peer review?

The open peer review process lead by pyOpenSci address several issues in the scientific Python
ecosystem:
Software peer review refers to a peer-review process that focuses on open source software code, documentation and infrastructure. In pyOpenSci this review includes:

1. [Multiple packages that have overlapping functionality. See this example on `PyPI`](https://pypi.org/search/?q=twitter)
2. Packages that have varying levels of maintenance yet are used by the community to support open science workflows.
3. Packages that are well-maintained and used but then maintenance comes to a halt when the maintainer needs to step down (burn-out is common and understandable).
4. Packages using varying types of packaging and documentation approaches making it more difficult to contribute.
5. Packages that are not documented enough to support:
* Contributions from others
* Directions on how to get started using the package functionality (quick start vignettes or tutorials)
* Packages that are missing OSI licensing and citation information
* Code quality,
* Documentation quality,
* Package usability,
* Test coverage that supports both maintenance of code function. Test coverage also makes it easier for contributors to see how their contributions impacts other parts of the code,
* Evaluation of infrastructure such as continuous integration to run rest suites and check code linters, that supports automated checks on pull requests. This infrastructure supports software quality and reliability.

## What types of packages does pyOpenSci review?
pyOpenSci reviews higher level software packages that support scientific workflows.

### How is pyOpenSci different from JOSS and other review processes?
pyOpenSci is different from other review processes
out there because:
:::{figure-md} fig-target

1. We specifically review to community accepted python packaging standards
2. We consider accepted packages as vetted and a part of our ecosystem. We recommend those packages to others. If the maintainer needs to step down, we will ensure a new maintainer takes over OR sunset and remove the package from our ecosystem.
<img src="../images/python-stack-jupyter-earth.png" alt="Image showing the tiers of software in the python ecosystem starting with Python itself and as you move out packages become more domain specific. In this image packages like xarray and numpy are considered core to scientific python. Packages and distributions like astropy, simpeg and metpy are considered to be domain specific." width="700px">

The JOSS review process is about publication. There, you will receive a DOI that is cross-ref
enabled. However JOSS will not followup with the maintainer to ensure that the package is maintained over time.
Diagram showing the tiers of software in the python ecosystem starting with Python itself and as you move out packages become more domain specific. In this image packages like xarray and numpy are considered core to scientific python. Packages and distributions like astropy, simpeg and metpy are considered to be domain specific.. pyOpenSci's review
process focuses on domain specific packages rather than core packages as
these packages tend to have more variability in long term maintenance and
package infrastructure and quality compared to established core packages. **Source: ["Jupyter meets earth" project](https://jupytearth.org/jupyter-resources/introduction/ecosystem.html)**
:::


## Why submit your package to pyOpenSci for review?
Currently, the packages that pyOpenSci reviews also need to fall into the
technical and applied scope of our organization. This scope may expand over time
as the organization grows.

There is a lot to gain from submitting your package to pyOpenSci.
But of course, before you submit, please be sure to read the policies
... to ensure pyOpenSci is the right fit for you.
## Why does the scientific community need software peer review?

First, and foremost, we hope you submit your package for review **because you value the feedback**. We aim to provide useful feedback to package authors and for our review process to be open, non-adversarial, and focused on improving software quality.
The pyOpenSci open peer review process addresses several issues in the
scientific Python ecosystem:

- Once your package is accepted, your package will receive **support from pyOpenSci members**. You'll retain ownership and control of your package. However, if you need help with ongoing maintenance issues, we will try to find people who can help.

- pyOpenSci will **promote your package** through our:
- [webpage](https://www.pyopensci.org/python-packages/),
- [blog](https://www.pyopensci.org/blog/), and
- [social media account](https://twitter.com/pyopensci).
1. [Multiple packages that have overlapping functionality. See this example of the many packages that interface with Twitter on `PyPI`](https://pypi.org/search/?q=twitter)
1. Packages that have varying levels of maintenance yet are used by the community to support open science workflows.
1. Packages that are well-maintained and used but then maintenance comes to a halt when the maintainer needs to step down (burn-out is common and understandable).
1. Packages using varying types of packaging and documentation approaches making it more difficult to contribute.
1. Packages that are not documented enough to support:
* Contributions from others
* Directions on how to get started using the package functionality (quick start vignettes or tutorials)
1. Packages that are missing OSI licensing and citation information.

- pyOpenSci packages that are in scope for the [Journal of Open-Source Software](https://joss.theoj.org/) and add the necessary accompanying short `paper.md` file, can, at the discretion of JOSS editors, benefit from a fast-tracked review process. [Read more about our partnership with JOSS here](#pyopensci-and-joss).
### Why is pyOpenSci focused on the Python programming language?

## Why do we need peer review for Python scientific software?
These challenges with open source software exist in other ecosystems as well. [rOpenSci](https://www.ropensci.org), an inspirational, sister organization of pyOpenSci, is an
example of an organization with similar values that operates in the scientific R programming language
space. However many of the issues
faced within the scientific Python community are broader in scale given the
numerous and diverse applications that the Python programming language is used for.

pyOpenSci's [suite of packages](https://pyopensci.org/python-packages/) are fully
contributed by community members with a great diversity of skills. This diversity
of developer backgrounds results in a range of quality associated with the suite
of tools available to process scientific data.
```{note}
[This blog post](https://www.numfocus.org/blog/how-ropensci-uses-code-review-to-promote-reproducible-science/) written by editors from our partner organization, rOpenSci, is a good introduction to pyOpenSci software peer review
```

### Peer review helps maintain consistent open source software quality
### Peer review of open source software helps maintain consistent quality

Peer review of python tools that support science is critical to enforcing
quality and usability standards. All pyOpenSci packages contributed by the
community undergo a transparent, constructive, non adversarial and open peer
review process. The goal of that process is to enforce commonly accepted standards.
These standards include technical structure of the package, usability of the
package, documenting package functionality in a way that is accessible
to all levels of users and proper licensing and citation information.

### A truly community-founded review process

Our peer review process is run by volunteer members of the Python scientific
community:

* Editors manage the incoming package review submissions and ensure
reviews move forward progress of submissions;
* authors create, submit and improve their package;
* Reviewers, two per submission, examine the software code and user experience.

```{note}
[This blog post](https://www.numfocus.org/blog/how-ropensci-uses-code-review-to-promote-reproducible-science/) written by editors from our partner organization, rOpenSci, is a good introduction to pyOpenSci software peer review
```
### How do I know a python package is pyOpenSci vetted?

You can identify pyOpenSci packages that have been peer-reviewed by the green
"peer-reviewed" badge at the top their README, [![pyOpenSci](https://tinyurl.com/y22nb8up)]() linking to the specific issue
where the tool was reviewed. [See this example from devicely, one of our packages](https://github.com/hpi-dhc/devicely).

### How do reviews work?

We use GitHub for our entire review process. We like GitHub because:
to all levels of users and proper licensing and citation information.

* It's free to create an account
* It facilitates collaboration and supports community around a package
* It facilitates open discussion via issues
* It supports version control
* Numerous packages store their code bases on GitHub

We make the most of [GitHub](https://github.com/)infrastructure in
our review process.

Each package review is contained within an issue in the [pyOpenSci/software-review GitHub repository](https://github.com/pyopensci/software-review/).


```{note}
For instance, click [here](https://github.com/ropensci/software-review/issues/24) to read the review thread from an rOpenSci review of the `ropenaq` package. Note that the process is an ongoing conversation until the package is accepted. Two external reviews are important milestones in the review process.
```

### GitHub tools including issue submission templates and labels help us streamline peer review
We use GitHub features including:

* issue templates (as submission templates), and
* labelling issues to track progress of submissions (from editor checks to approval).
* Project boards to track help wanted items

All of this functionality supports a streamlined and open peer review
process.

## Why review a package for pyOpenSci?

We hope you choose to review **to give back to the rOpenSci and scientific communities.** Our mission to expand the ability to efficiently use scientific data and promote a culture of open reproducible is only possible through the volunteer efforts of community members like you.

- Peer review is a two-way conversation. By reviewing packages, you'll have the chance to **continue to learn development practices from authors and other reviewers**.
- The open nature of our review process allows you to **network and meet colleagues and collaborators** through the review process. Our community is friendly and filled with supportive members expert in Python package development, a diversity of science domains and computer science.

### Volunteer to review for us
- To volunteer to be one of our reviewers, fill out [this short form](https://forms.gle/9dv99gri6XKNU8177). In the form you will provide your contact information and areas of expertise. We are always looking for more reviewers with both general package-writing experience and domain expertise in the fields where packages are used. Some of our reviewers focus on package usability and documentation. This allows you to review even if you don't have strong technical background. We will pair you with a second reviewer that can focus more on the technical aspect of the review.

## Why are reviews open?

Expand All @@ -144,42 +92,14 @@ comments in public and without the cover of anonymity.
> believe that having direct and public communication between authors and
> reviewers improves quality and fairness of reviews.
### How do I know that a Python package has been reviewed by pyOpenSci?

### If you submit to pyOpenSci You Can Also Be Accepted by JOSS
We have a strong partnership with JOSS (Journal of Open Source Software); JOSS
accepts our review as their own. If your package is within JOSS' scope you can
then submit it to JOSS, linking to the accepted pyOpenSci review. Note that
JOSS won't accept packages that are don't have research applications such as
API wrappers. JOSS will then accept our review (you will not need a second
review with JOSS!). JOSS will review your paper and you will get a JOSS badge
to add next to your pyOpenSci badge of review. And a cross-ref enabled DOI.


## pyOpenSci and JOSS

> You don't have to chose between pyOpenSci and JOSS; You can submit your package to both.
pyOpenSci and [the Journal of Open Source Software (JOSS)](https://joss.theoj.org/)
are complementary, partner organizations; and you don't have to chose one or the
other! After a package to pyOpenSci has been reviewed and accepted by pyOpenSci
you can chose to also register it with JOSS. JOSS has [more limited scope](https://joss.readthedocs.io/en/latest/review_criteria.html) of the
for packages that it will review. For instance while pyOpenSci will review and
accept API wrappers, JOSS won't.

If your package is accepted by pyOpenSci and in scope for JOSS, JOSS will fast
track your package through their process given it was already reviewed by us.
Once accepted by JOSS, you now have both a pyOpenSci acceptance and one by JOSS.
Joss will then give you a cross-ref supported DOI for citation.
You can identify pyOpenSci packages that have been peer-reviewed by the green
"peer-reviewed" badge at the top of their `README.md` file, [![pyOpenSci](https://tinyurl.com/y22nb8up)](). This badge is added by the package author after the package
has successfully completed review and ideally links to the specific GitHub issue
where the tool was reviewed. [See this example from devicely, one of our accepted pyOpenSci ecosystem packages](https://github.com/hpi-dhc/devicely).

### Why Two Review Processes JOSS and pyOpenSci?

the pyOpenSci review process is different from that of JOSS in a few ways:
* pyOpenSci is specific to the Python community and thus will enforce community specific python specific standards.
* pyOpenSci places heavy emphasis on documentation and usability in our reviews and associated standardization of both.
* pyOpenSci builds community around and visibility for it's tools.
* pyOpenSci supports long term tool maintenance.

********

JOSS reviews are [more limited in scope](https://joss.readthedocs.io/en/latest/review_criteria.html) compared to pyOpenSci and the
[submission criteria](https://joss.readthedocs.io/en/latest/review_criteria.html)
are, in places, less stringent than those of pyOpenSci.
Loading

0 comments on commit e214523

Please sign in to comment.