The first property is to note which social bias is implicated in the comment. Here, social bias means hasty guess or prejudice that `a person/group with a certain social identity will display a certain characteristic or act in a biased way'. The three labels of the question are as follows.
- Is there a gender-related bias, either explicit or implicit, in the text?
- If the text includes bias for gender role, sexual orientation, sexual identity, and any thoughts on gender-related acts (e.g., "Wife must be obedient to her husband's words" or "Homosexual person will be prone to disease.")
- Are there any other kinds of bias in the text?
- Other kinds of factors that are considered not gender-related but social bias, including race, background, nationality, ethnic group, political stance, skin color, religion, handicaps, age, appearance, richness, occupations, the absence of military service experience (Frequently observable in Korea, where the military service is mandatory for males.), etc.
- A comment that does not incorporate the bias
The second property is how aggressive the comment is. Since the level of "aggressiveness" depends on the linguistic intuition of annotators, we set the following categorization to draw a borderline as precise as possible.
- Is strong hate or insulting towards the article's target or related figures, writers of the article or comments, etc. displayed in a comment?
- In the case of insulting, it encompasses an expression that can severely harm the social status of the recipient.
- In the case of hate, it is defined as an expression that display aggressive stances towards individuals/groups with certain characteristics (gender role, sexual orientation, sexual identity, any thoughts on gender-related acts, race, background, nationality, ethnic group, political stance, skin color, religion, handicaps, age, appearance, richness, occupations, the absence of military service experience, etc.).
- Additionally, it can include sexual harassment, notification of offensive rumors or facts, and coined terms for bad purposes or in bad use, etc.
- Just an existence of bad words in the document does not always fall into this category.
- Although a comment is not as much hateful or insulting as the above, does it make the target or the reader feel offended?
- It may contain rude or aggressive contents, such as bad words, though not to the extent of hate or insult.
- It can emit sarcasm through rhetorical question or irony.
- It may encompass an unethical expression (e.g., jokes or irrelevant questions regarding the figures who passed away).
- A comment conveying unidentified rumors can belong to this category.
- A comment that does not incorporate any hatred or insulting