Replies: 6 comments 10 replies
-
|
The words in the second list seem to be two concatenated words, missing the space in between: internacionalactualización --> "internacional actualización" (international update) Some are typos: internacionalzación --> "internacionalización" BTW I thought that maybe you could add the grave and circumflex accents to your list of characters: àèìòù ÀÈÌÒÙ âêîôû ÂÊÎÔÛ While they are not used in spanish, they could appear somewhere like for instance the literal name of a frech person's name, location or movie, like with english. |
Beta Was this translation helpful? Give feedback.
-
|
Can't think of an easy/fast way to figure out if it is a concatenated word. It is doable, but not fast. I will have to ignore them. Will make some program changes to try to find more plurals, and then see what is fixable and what must be ignored. My list of characters is going to be the "baseline" that any keyboard layout MUST provide if it claims to be "Spanish". On the flip side, all input texts may only contain those characters, so that texts and keyboards are in agreement. Other characters are optional. |
Beta Was this translation helpful? Give feedback.
-
How many are we talking about? I think I can manually clean a txt file of that rather quickly (it's basically hitting space).
Ah, ok, with the goal in mind, it's clearer. In that case, I think you should drop these two symbols: § ¶ Technically, they are spanish punctuation symbols. But they are not on qwerty-es, and noone, ever, uses them in any kind of text. The frequencies are negligible as well: Also the yen and pound are not in qwerty-es. I think they're useful. but they don't follow your rule of being in a spanish keyboard. But I think in an optimized kb it'd be nice to have them: AlGr + L for £, and AltGr + Y for ¥. Also, I wonder why |
Beta Was this translation helpful? Give feedback.
-
Okay, that also reduces bigram list a lot.
Those are not in the Leipzig corpus ( I scanned for some of them to convince myself.). could be because of their sources ... If I get Gutenberg to work some should show up. I had similar issues with the English corpus that I made (no Leipzig). Current "Spanish" character list is |
Beta Was this translation helpful? Give feedback.
-
|
You can use MOAR backticks :) Here I opened with 2 backticks, closed with two backticks. The escaped backtick shall not be next to the code backticks, so I added a space at the end. |
Beta Was this translation helpful? Give feedback.
-
|
Are these actual words? Here's the top 100 of the others... I'm guessing they are all okay. As you can see the counts are getting low. éste 52797 Thanks, Ian |
Beta Was this translation helpful? Give feedback.
-
|
The majority of those are correct words. Perhaps a little on the rare side, meaning they're not very common, and maybe that's why they aren't in the dictionary. But they are fine. Some have typos, mostly with the accented vowels. Let me try to catch them all: comprension --> comprensión (understanding) A special case is "ícono". The typical form is "icono" (unaccented) but I just checked and accented is also correct. It's more likely a typo, but it happens to be valid. I think those would be all, if I haven't missed any. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks. I guess I will use regex to replace them in the corpus. |
Beta Was this translation helpful? Give feedback.
-
|
As Arno say, we're getting to the point of diminishing returns. I'm down to the "7 counts" in the unknown words, and the Capitalised ones are starting to look a bit suspicious, so will stop with that process. If we can maybe do another 200 regular words, then I will drop the rest of the unknown (and probably rare) words and make final 'cleaner' corpus. Metrics: 100 unknown words: abstenciones 1451 Thanks, Ian |
Beta Was this translation helpful? Give feedback.
-
|
Hi there, As the last batch, the majority of these words are valid, though a little uncommon, with a few typos in between: situaci --> situación The rest are OK. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks, have updated and rerunning. |
Beta Was this translation helpful? Give feedback.
-
|
@iandoug & @Lobo-Feroz -- thank you so much for your persistence and diligence! |
Beta Was this translation helpful? Give feedback.
-
|
Last batch, thanks. I looked at the Gutenburg code, it pulls all books .. for English, I discarded any that had non-US-ANSI chars. That won't work here. Instead I got a book list which shows language, will parse that then use wget/curl to pull the Spanish ones. interactúan 980 |
Beta Was this translation helpful? Give feedback.
-
|
Same same. Mostly uncommon words, some typos: travs --> través This one is special: oacute oacute is not spanish, in fact I think this is html markup to display an ó, the complete form being maybe in the rest of the words you have more like this: aacute eacute iacute uacute ntilde and others here's a list: https://www.ou.edu/research/electron/internet/special.shtml |
Beta Was this translation helpful? Give feedback.
-
|
Thanks, you are correct about it being HTML. |
Beta Was this translation helpful? Give feedback.
-
|
Had some other stuff I needed to sort out over our long weekend, hope to carry on with this today .. one more work work thing to sort out first. |
Beta Was this translation helpful? Give feedback.
-
Declutter threads.
From "known words" list:
internacional 157568
internacionalice 14
internacionalicemos 4
internacionalicen 6
internacionalidad 107
internacionalismo 291
internacionalista 610
internacionaliza 26
internacionalizaba 2
internacionalización 1898
internacionalizado 63
internacionalizamos 1
internacionalizan 11
internacionalizando 22
internacionalizar 232
internacionalizara 1
internacionalizaran 1
internacionalizaremos 1
internacionalizaron 11
internacionalizará 2
internacionalizaría 1
internacionalizó 42
internacionalmente 3514
From "unknown words" list: Are these (generally) supposed to be one word? Am trying to differentiate between "normal usage" and badly collected/cleaned text.
internacionala 4
internacionalactualización 2
internacionalalex 1
internacionalallan 1
internacionalamente 1
internacionalamerica 1
internacionalamericanaaustraliana 1
internacionalamérica 1
internacionalanimadora 1
internacionalargentina 1
internacionalargentino 1
internacionalarévalo 1
internacionalasdi 2
internacionalasistencia 1
internacionalasí 1
internacionalatentados 1
internacionalatlético 1
internacionalaunque 2
internacionalavisos 1
internacionalavtocomplex 1
internacionalbajo 1
internacionalbarcelona 1
internacionalbueno 1
internacionalcampeonato 1
internacionalcapítulo 1
internacionalcbi 1
internacionalch 1
internacionalchile 2
internacionalchilenos 4
internacionalcolombia 1
internacionalcomo 1
internacionalcon 3
internacionalconsecuencias 1
internacionalconsultor 1
internacionalcontactos 1
internacionalcoritiba 1
internacionalcruz 1
internacionalcti 1
internacionalcuando 1
internacionalcumbre 2
internacionalcuál 1
internacionalcáffaro 1
internacionald 1
internacionaldaniele 1
internacionalde 6
internacionaldeberia 1
internacionaldebido 2
internacionaldeclaran 1
internacionaldefinen 1
internacionaldenunciará 1
internacionaldesde 1
internacionaldetalla 1
internacionaldiario 1
internacionaldivisión 1
internacionaldocentes 1
internacionaldt 1
internacionaldónde 1
internacionale 10
internacionaleden 1
internacionaleei 1
internacionalees 1
internacionalel 17
internacionaleliminatorias 5
internacionalels 1
internacionalemente 4
internacionalemilio 1
internacionalemnte 1
internacionalemtne 1
internacionalen 6
internacionalenfrentar 1
internacionalentrenador 1
internacionalequipo 1
internacionalerp 1
internacionales 57489
internacionalesa 1
internacionalesactualmente 1
internacionalesaguas 1
internacionalesapuntan 1
internacionalesarzobispo 1
internacionalesasi 1
internacionalesatentados 1
internacionalesbank 1
internacionalesbogotá 1
internacionalesbolivia 1
internacionalesbush 1
internacionalescanciller 1
internacionalescarlos 1
internacionalescarrá 1
internacionalescasi 1
internacionalescelebran 1
internacionaleschina 1
internacionaleschávez 1
internacionalesclasificados 1
internacionalescolombianos 1
internacionalescomienza 1
internacionalescondecora 1
internacionalesconfirman 1
internacionalesconflicto 1
internacionalescontinúa 1
internacionalescrucial 1
internacionalescue 1
internacionalesde 2
internacionalesdebates 1
internacionalesdiosdado 1
internacionalesdirección 1
internacionalesdos 1
internacionalesdía 1
internacionaleseconómicos 1
internacionalesecuador 1
internacionaleseditar 1
internacionalesedward 1
internacionaleseeuu 1
internacionalesejército 1
internacionalesel 5
internacionaleseligen 1
internacionalesemilio 1
internacionalesempresa 1
internacionalesen 1
internacionalesentre 1
internacionaleses 1
internacionalesexigirán 1
internacionalesfavor 1
internacionalesfidel 1
internacionalesfin 1
internacionalesforo 1
internacionalesfrancia 1
internacionalesfuerzas 1
internacionalesgobierno 1
internacionaleshija 1
internacionaleshomenajean 1
internacionaleshttpwww 1
internacionalesintensifican 1
internacionalesinter 1
internacionalesinternas 1
internacionalesintervención 1
internacionalesinvita 1
internacionalesjornada 1
internacionalesla 5
internacionaleslas 2
internacionalesldquopero 1
internacionaleslondres 1
internacionalesmaria 1
internacionalesmartes 2
internacionalesmerkel 1
internacionalesmoscu 1
internacionalesnadie 1
internacionalesnorcorea 1
internacionalesobama 1
internacionalesola 1
internacionalesonu 1
internacionalespanama 1
internacionalespara 1
internacionalesparlamento 1
internacionalespor 1
internacionalesportal 1
internacionalespresidente 3
internacionalespresionan 1
internacionalesprimer 2
internacionalesprocurando 1
internacionalesque 1
internacionalesquinta 1
internacionalesrealizan 1
internacionalesreanudaran 1
internacionalesreelección 1
internacionalesretiran 1
internacionalesrodriguezunju 1
internacionaless 1
internacionalessanto 3
internacionalessin 1
internacionalessolo 1
internacionalestambién 1
internacionalesteban 1
internacionalesto 1
internacionalestrella 1
internacionalestrump 1
internacionalestá 1
internacionalesuniversidad 1
internacionalesunsam 1
internacionalesvenezuela 1
internacionalesverifica 1
internacionalesvie 1
internacionalesy 1
internacionales¨ 1
internacionalexige 1
internacionalexigen 1
internacionalfigo 1
internacionalfmi 4
internacionalfue 1
internacionalfundación 1
internacionalgareca 1
internacionalgoles 1
internacionalgran 1
internacionalgremio 1
internacionalguardiola 2
internacionalha 1
internacionalhay 1
internacionalhelicobacter 1
internacionalheynckes 1
internacionalhrw 1
internacionalhttpst 1
internacionalhttpwww 1
internacionali 1
internacionaliación 1
internacionalicardi 1
internacionalidades 89
internacionalifi 1
internacionalimagen 1
internacionalindia 1
internacionalinflación 1
internacionalinicia 1
internacionalinstituto 1
internacionalinteligencia 1
internacionalinternacional 1
internacionalinterpol 1
internacionalinvitado 1
internacionalis 3
internacionalisaac 1
internacionalismohttpwww 1
internacionalistaantiglobalización 1
internacionalistala 1
internacionalistaliga 1
internacionalistasolidaridad 3
internacionalius 1
internacionalivan 2
internacionalizaci 15
internacionalizacin 19
internacionalizacioacuten 1
internacionalizacion 2
internacionalizaciones 1
internacionalizació 1
internacionalizacióncolocandose 1
internacionalizaciónlan 1
internacionalizada 28
internacionalizadas 38
internacionalizador 1
internacionalizadora 1
internacionalizarla 3
internacionalizarlas 1
internacionalizarlo 6
internacionalizarme 3
internacionalizarnos 3
internacionalizarse 64
internacionalizarsey 1
internacionalize 1
internacionalizándose 3
internacionaljefe 1
internacionalk 1
internacionalla 20
internacionallas 3
internacionalle 2
internacionalledecky 1
internacionallejos 1
internacionalleyenda 2
internacionalllegando 1
internacionallo 3
internacionallocal 1
internacionallorenzo 1
internacionallos 1
internacionalluego 1
internacionalmaradona 1
internacionalmarketing 1
internacionalmaría 1
internacionalmayores 1
internacionalmenteacoge 1
internacionalmentede 1
internacionalmentedesde 1
internacionalmenteen 1
internacionalmenteinditex 1
internacionalmenteno 1
internacionalmento 1
internacionalmete 1
internacionalmisión 1
internacionalmourinho 3
internacionalmujer 1
internacionalmundial 1
internacionalmuralista 1
internacionalnacional 1
internacionalneymar 1
internacionalno 1
internacionalnos 1
internacionalobstaculizar 1
internacionaloea 1
internacionalopositores 1
internacionalordóñez 1
internacionalpara 2
internacionalparaliza 1
internacionalpaula 1
internacionalpelé 1
internacionalpermanecidas 1
internacionalpero 1
internacionalpochettino 1
internacionalpopulismo 1
internacionalpor 1
internacionalpreguntas 38
internacionalproblema 1
internacionalpronóstico 1
internacionalpágina 1
internacionalque 4
internacionalqueda 1
internacionalqué 1
internacionalramón 1
internacionalreal 1
internacionalrecibiría 1
internacionalreforzarán 1
internacionalregión 1
internacionalren 1
internacionalreporteros 1
internacionalreunión 1
internacionalribéry 1
internacionalriverboca 1
internacionalrueda 1
internacionalrumbo 13
internacionalsan 1
internacionalsanta 6
internacionalsantiago 1
internacionalse 2
internacionalsean 1
internacionalsección 1
internacionalselecciones 4
internacionalselección 3
internacionalsevilla 1
internacionalsi 1
internacionalsimeone 1
internacionalsin 1
internacionalsinceridad 1
internacionalsino 1
internacionalsorprendido 1
internacionalsúper 1
internacionalteléfono 1
internacionaltoutain 1
internacionaltreinta 1
internacionaltu 1
internacionalun 3
internacionalusted 1
internacionalvarios 2
internacionalvehículos 1
internacionalvenezuela 2
internacionalvideochatinicio 219
internacionalviernes 1
internacionalvulneración 1
internacionalyo 1
internacionalzación 1
internacionalísimo 1
Thanks, Ian
Beta Was this translation helpful? Give feedback.
All reactions