special token "[semantic]"

[GinnyXiao]

For training your newly released checkpoint supporting semantic level segmentation, you added a special token "[semantic]" before the input prompt in the training data. I was wondering what difference this token makes? You have not modified your model architecture, which gives 1-1 mapping between your prediction and your prompt. The training strategy for referring seg datasets and semantics seg datasets remain the same, except for this token. Why would adding a "[semantic]" token improve your model ability? Would training without it yield similar performance?

Thank you very much in advance!

[CoderZhangYx]

When I apply semantic-segmentation datasets like ade20k, I add '[semantic]' because the category name refers to all the instances within the image in those datasets. This is in contrast to original referring datasets where the prompt only refers to a single instance. I apply the special token to balance the conflict, making the model behave better during joint training. By employing semantic segmentation datasets, the model is able to segment multiple objects, and also stuff categories if prompted with '[semantic]'