Prevent users from uploading and give warnings during upload if certain basic metadata and Wikitext criteria aren't met

[trnstlntk]

I'm checking some recent uploads with OpenRefine to Wikimedia Commons and some users keep these uploads extremely minimal.

This upload for instance doesn't include any structured data and Wikitext except for a license. This is not sufficient according to Wikimedia Commons guidelines; the file should at least have a set of minimal structured data statements and/or a minimal infobox template in Wikitext.

This is clearly explained in the current how-to - but not every user is aware of, or reads, these guidelines.

To prevent users from doing such (too minimal) uploads in the future, OpenRefine should provide warnings and probably even prevent uploads if minimal conditions aren't met.

Checks can be done for

• Files not having at least the minimally required structured data statements
• Files without a == {{int:filedesc}} == header and/or without one of the following bits of infobox template wikitext: {{Information}} – default template meant especially for photographs created by users.
  {{Artwork}} – for paintings, artworks and artifacts held by museums and other GLAM institutions
  {{Photograph}} – for photographs held by museums and other GLAM institutions
  {{Art photo}} – for photographs of artworks with more fields for photographer metadata
  {{Book}} – for books
  {{Map}} – for maps
  {{Musical work}} – for music files

[wetneb]

We could have this in the main repository, because those checks should be enforced regardless of whether the commons extension is installed, no?

[trnstlntk]

I was indeed wondering where to post this issue, as it's very Wikimedia Commons-specific.

…

On Sun, Sep 3, 2023 at 5:09 PM Antonin Delpeuch ***@***.***> wrote: We could have this in the main repository, because those checks should be enforced regardless of whether the commons extension is installed, no? — Reply to this email directly, view it on GitHub <#95 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACLMQLJQC44XC4NY35NJMUDXYSMRJANCNFSM6AAAAAA4JJEWIQ> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[wetneb]

Yes I guess those additional checks could be added by the Commons extension instead of the Wikibase one, it's debatable. I'm not sure which proportion of people doing Commons uploads actually have the Commons extension installed, it would be interesting to have some stats about that.

[trnstlntk]

I plan to ask that in a user survey soon!I also retrieved download counts from GitHub, but I don’t know very well how to evaluate these numbers. Stats in one tab here: https://docs.google.com/spreadsheets/d/1XNdRSOgVZeEfg8qqaNIDVgtgxWdpWncbW7ijA-wBx54/editOn 3 Sep 2023, at 21:16, Antonin Delpeuch ***@***.***> wrote: Yes I guess those additional checks could be added by the Commons extension instead of the Wikibase one, it's debatable. I'm not sure which proportion of people doing Commons uploads actually have the Commons extension installed, it would be interesting to have some stats about that. —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you authored the thread.Message ID: ***@***.***>

[sunilnatraj]

@wetneb Can you assign this to me.

[wetneb]

@sunilnatraj thanks for your initiative! The screenshot you have looks good. I expect there will be quite some design work on this issue (what severity for the warnings, how to specify this list of required properties, which text to use…), which would be worth doing in tandem with @Vesihiisi and @sebastian-berlin-wmse, so expect more rounds of back and forth exchanges on this one.

[sunilnatraj]

@Vesihiisi @sebastian-berlin-wmse Do share your inputs on this.

[sunilnatraj]

@wetneb @sebastian-berlin-wmse @Vesihiisi

The constraints for new media can be defined in the manifest file see snippet below. If the constraints are defined then the validations are carried out for new media entity.

"mediawiki": {
    "name": "Wikidata",
    "root": "https://www.wikidata.org/wiki/",
    "main_page": "https://www.wikidata.org/wiki/Wikidata:Main_Page",
    "api": "https://www.wikidata.org/w/api.php",
    "constraints": {
        "required_properties": "P7482, P571, P170, P6216, P275",
        "wikitext_requires_anyone_infobox_template" : "Information, Artwork, Photography, Art photo, Book, Map, Musical work"
    }
}

[wetneb]

Yes, it feels fitting to define the required property ids in the manifest.
Instead of encoding lists as strings, I would use JSON's native ability to represent arrays: ["P7482", "P571", "P170", "P6216", "P275"] instead of "P7482, P571, P170, P6216, P275".
There is then the question of whether that's expressive enough:

• should all those properties be required with the same severity, or would it be useful to specify the severity level independently? For instance with [{"pid": "P7482", "severity": "critical"}, {"pid": "P571", "severity": "warn"}]
• is this format expressive enough to represent the requirements on Commons side? For instance, are there any situations where one of two properties should be provided, but not necessarily both?

Concerning the wikitext, I wonder if we can find a reliable and current source of information about the requirements. I am not sure to what extent we can really validate this field, because some of the required parts, such as == {{int:filedesc}} ==, could potentially be added via a template, no? If we do add validation on this part, I would use a rather low severity level, because it's likely that whatever heuristic we use will not work perfectly.

[sunilnatraj]

@wetneb I reviewed the links in the Issue, as per the definition there are mandatory properties only for a media, which is why i suggested making a list of required/mandatory properties. If we need to also validate optional properties then your suggestion makes sense. The other case you mentioned requires P112 OR P114, if this has to be supported then is this definition available from the wikimedia system?

Wikitext validation - I referred this link as per this there are Infobox templates and the proposal is to verify if Any of the infobox templates is present and not empty.

[sunilnatraj]

@wetneb a more expansive definition model

"mediawiki": { "name": "Wikidata", "root": "https://www.wikidata.org/wiki/", "main_page": "https://www.wikidata.org/wiki/Wikidata:Main_Page", "api": "https://www.wikidata.org/w/api.php", "constraints": { "validations": { "requiredProperties": { "pids": [ "P170", "P452", "P453" ], "severity": "critical" }, "conditionalProperties": [ { "pids": [ "P121", "P144" ], "severity": "critical" } ], "optionalProperties": { "pids": [ "P542", "P543", "P544" ], "severity": "warn" } }, "wikitext_requires_anyone_infobox_template": [ "Information", "Artwork", "Photography", "Art photo", "Book", "Map", "Musical work" ] } }

[sunilnatraj]

@wetneb @sebastian-berlin-wmse @Vesihiisi Any inputs on the proposed solution

[sunilnatraj]

@wetneb @sebastian-berlin-wmse @Vesihiisi Following up -> Any inputs on the proposed solution

[wetneb]

Sorry that it's taking so long to get feedback on this. The problem in this case is that it's quite some work to do the associated research to make sure the solution is fit for purpose, and no one seems to have the time to work on this at the moment. But maybe you can?

You could go to Wikimedia Commons and ask the community for feedback on this. I would try to phrase it in terms that are understandable without deep knowledge of OpenRefine, such as:

As you may know, OpenRefine lets users upload media files to Commons in batch. Because some of the uploads done in this way add too little metadata to the uploaded files, we are considering introducing more pre-upload checks to prevent that. We need your help to determine which metadata fields should be required for any file uploaded via OpenRefine. Are these guidelines still up to date and accurate?
Based on this information, we would require the users to provide:

• A caption
• An "inception (P571)" statement
• A "source of file (P7482)" statement
• A "creator (P170)" statement
• A "copyright status (P6216)" statement

We would not require "copyright license (P275)" as this statement is not required for works in the public domain, and we don't anticipate being able to be able to express this conditional dependency.
We also looked into adding constraints on the wikitext associated to the media files but this is likely too complicated to implement reliably, as some required parts could be added via different sorts of templates, which OpenRefine isn't able to expand before upload.

What do you think of this plan? Can you think of any case where it would be fine to upload a file without one of the 5 fields mentioned above? Do you think OpenRefine should only warn the user about those missing fields, or even prevent the upload entirely if those fields are not provided?

You could try to write a message like this one (feel free to copy all/parts of it) and send it to the Wikimedia Commons community. Good places for it could be:

• https://commons.wikimedia.org/wiki/Commons:Village_pump
• the Telegram channel "OpenRefine - Wikimedia": https://t.me/+Qc23Jlay6f4wOGQ0
• the talk page of the page mentioned above: https://commons.wikimedia.org/wiki/Commons_talk:Structured_data/Modeling

It probably makes sense to post it in multiple places to maximize the chances of getting informed feedback.

[sunilnatraj]

@wetneb Thank you for the suggestion. I have posted on wikimedia.

• https://commons.wikimedia.org/wiki/Commons:Village_pump#OpenRefine_-_Commons_upload_validations
• https://commons.wikimedia.org/wiki/Commons_talk:Structured_data#OpenRefine_-_Commons_upload_validations