Properly detect emoticons

[diasks2]

#4
it 'preserves emoticons' do
  text = "lol :-D"
  pt = PragmaticTokenizer::Tokenizer.new(text, downcase: false)
  expect(pt.tokenize).to eq(
    ["lol", ":-D"]
  )
end

[maia]

I've just came across retext-emoji and wonder if it might be smart to convert emoticons to emoji:

When encode, converts short-codes into their unicode equivalent (e.g., :heart: and <3 to ❤️)

While I think it might be too much ton attempt to convert all possible emoticons, one could pragmatically do so for the 10-20 most common emoticons very early in the processing, and only later handle remaining punctuation.

[diasks2]

Interesting idea. It might be something to consider doing internally so we don't confuse emoticons with other punctuation...however, in the output that is returned to a user I am kind of a purist and think that it should match the original input (i.e. <3 in the input text would not return ❤️‍ as a token)