Character frequency

[iandoug]

Let me start a new thread, others are getting long.

Character frequency from analysis of cleaned and paragraphed Leipzig corpus attached. I think the licence allows to post the compiled corpus, will upload to Zenodo once I have done the bigrams and trigrams, probably only tomorrow.

Version 2:
leipzig-spanish-char-freq.txt

Cheers, Ian

[Lobo-Feroz]

Hey Ian, this is really awesome!

If I have time tomorrow I'll try importing it into libreoffice calc to do some analysis.

Just by seeing the text I can see some hypothesis validated:

• The apostrophe ' is nowhere near as frequent as the quotes ". It's an order of magnitude less frequent. I don't think it deserves a key at the same level.
  • For instance, just the tilded á is more frequent than " and much, much more than '. I think the tilde or star key should have an important position in the keyboard.
• The capital Ü has a very low frequency indeed, so we don't need to make this a very easy character.

[iandoug]

Bigrams. All char pairs, I deleted those with zero.

Percentage is percentage of total count.
Make sure first column is formatted as text, or it will interpret some character pairs as formulas.

Trigrams tomorrow.

bigrams.ods

Treat as preliminary, since corpus appears to have dropped diacriticed vowels.

[iandoug]

Have replaced char frequency with version 2 in first post above.

There was a trailing space at the end of paragraphs, which increased the 'space' count, and created the 'space enter' bigram instead of the 'period enter' bigram.

Revised bigram file will be uploaded when done ... it takes a while.

[Lobo-Feroz]

Is space usually shown in frequencies for languages? I don't recall having seen it in wikipedia et al.

Maybe for public presentations and the like we'd rather use the relative frequency for characters withouth the space. But this is easy to do with the spreadsheet.

[iandoug]

Most "frequency lists" floating around are word-based and usually intended for cryptography.

Space frequency becomes important for people doing creative layouts that move space off the thumb.

I will also use it a few steps down the line when I create 'chained bigrams', which is artificial text that has the correct character and bigram frequency.

Space frequency varies by language because of average word length... see this Xhosa:
Njengoko iimfanelo zesidima soluntu semvelo kunye neemfanelo zoluntu-jikelele

vs same English
Whereas recognition of the inherent dignity and of the equal and inalienable

vs same Spanish
Considerando que la libertad, la justicia y la paz en el mundo tienen por base

[binarybottle]

@iandoug & @Lobo-Feroz -- Please alert me when you've got letter frequency and letter bigram frequency lists ready for me to run in place of the crypto lists I've been relying on. I need them in a spreadsheet or in the following format:

letters24 = ['E','A','O','S','N','I','R','L','D','C','T','U','P',
'M','B','G','V','Q','H','F','Y','J','Z','X','K','W']
instances24 = [10912000, 10301872, 7398419, 6128524, 5838540, 5694616, 5450913,
4808679, 4237020, 3648080, 3643515, 3292410, 2266539, 2152702,
1042859, 964043, 811441, 686200, 670756, 652415, 651259, 374173,
344877, 153120, 64167, 26492]
bigrams = ['DE','ES','EN',...]
bigram_instances = [2088635,1882962,1850294,...]

[binarybottle]

@iandoug --

For my engram-es layout optimization code to make sense of single-letter and letter bigram frequencies, I need just the frequencies of case insensitive letters or letter bigrams -- no spaces or non-letter characters. So I summed the cleaned-up Leipzig corpus counts for each letter across counts for that letter with whatever diacritical marks:

e 294897235
a 271738665
o 201996963
s 162205879
n 161440601
i 151438547
r 147218050
l 124544026
d 113430944
c 99562807
t 99294129
u 89773745
m 60501673
p 60057794
b 27527305
g 25968922
v 22134740
q 18664754
y 18537469
f 17869592
h 17384139
j 9976007
z 8509998
x 3992590
k 1766727
w 1021484

It will take me forever to do this for your bigrams, however.
Could you please recompute bigram frequencies for case-insensitive letters?

[iandoug]

Attached, I uppercased each occurrence. This kept diacritic letters separate .. I don't know if you need them that way. If you want diacritic letter merged with plain letter I can rerun.
spanish-bigrams-unicase-letters.ods

Cheers, Ian

[binarybottle]

Thank you, @iandoug, but I need diacritic letters merged with plain letters. Could you please rerun? And if I am to include Gutenberg sources, could you provide similar output?

[binarybottle]

Would it be straightforward for you to provide onegram frequencies for diacritic-merged plain letters for both sources (Leipzig & Gutenberg) as well?

[iandoug]

Char frequencies.
There is a difference in the order between Leipzig and Gutenberg around N R I L

Spreadsheet sorted by average.
spanish-letters-space-unicase-leipzig-gutenberg.ods

[binarybottle]

Thank you!!!

[Lobo-Feroz]

Signs frequencies.

Based on @iandoug 's spanish-character-frequency-v1.ods, I have grouped the punctuation signs per key with their shifted versions.

I was getting errors when sorting by "Average", so I made a 2nd sheet in the ods for a VLOOKUP table. Probably my calc-fu is not up to par, I'm sure it could have been done in the first sheet.

The ods:

spanish-character-frequency-v1 - signs - 20200820.ods

The result:

• The accute accent is the most used sign key (no surprises here)
• The hyphen-underscore beats the interrogation and exclamation signs (surprise here, I expected the other way around)
• Apostrophe has a very very low frequency compared to the others (no surprises here either)

@binarybottle, I hope these can help us sort the layout of the punctuation signs.

[binarybottle]

Thank you, @Lobo-Feroz -- this is very helpful!

[binarybottle]

@iandoug -- I don't know about the relative degree of strain/discomfort in the central columns between upper keys and shift+center keys, and suspect that the effects of lateral strain and the simultaneous pressing of two keys varies across individuals. I like your idea of comma+period equalling the frequency of the accent, but wish to avoid the period being the shifted version of the comma, because I try to avoid key combinations for frequent characters. It might serve the memory well to have the accent on the upper right center column key, since the marks are made above the letter in all cases except for the cC, and most of the letters are vowels, on the left side.

[iandoug]

Think you mean @Lobo-Feroz ... :-)

[iandoug]

I need to stay out of this debate because I am starting to realise the benefits of punctuation being where Colemak puts it (ie basically Qwerty), but that is at odds with Engram model. Also Spanish usage seems different to English.

[Lobo-Feroz]

@iandoug -- I don't know about the relative degree of strain/discomfort in the central columns between upper keys and shift+center keys, and suspect that the effects of lateral strain and the simultaneous pressing of two keys varies across individuals. I like your idea of comma+period equalling the frequency of the accent, but wish to avoid the period being the shifted version of the comma, because I try to avoid key combinations for frequent characters. It might serve the memory well to have the accent on the upper right center column key, since the marks are made above the letter in all cases except for the cC, and most of the letters are vowels, on the left side.

Having the diacritics key in the upper row right side (in the Y of qwerty) was also an earlier idea of mine, when you released the first version of engram-es three weeks ago with the frequencies from the cryptography website. See here: #9 (reply in thread)

So that would be an OK for me. But then we need to move the rest of the keys around. ; and : would go, I suppose in the shifted versions of , and . But we need to relocate at the very least one of ( - _ ) and ( " ' ), since there's no more room on the upper row. Or move the ¿¡ and ?! from the bottom row, though that would break symmetry.