Gutenberb corpus

[iandoug]

Hi

Is -- a standard way to indicate dialogue?
Google Translate switches some to single dash, and others to matching " ".

---

--Como es usted soltero--replico el ayuda de cámara--, no tiene que
pensar en nadie. Pero yo tengo familia. Por eso pienso entregarle el
dinero a ese buen señor que va a la Bolsa, y algo me producirá.

--Es una buena idea, señor Fernando--dijo el marmitón--. Cuando vaya
usted, llévele mis cuarenta francos.

• as you are single - I replied camera help--, you do not have to
  Think of anyone. But I have a family. That's why I plan to give him the
  Money to that good man who goes to the bag, and something will produce me.

"It's a good idea, Mr. Fernando," said the Marmiton. When I go
You, take my forty francs.

I will leave as is but if this is common then the dash frequency is going to shoot up.
Github is bolding things ... markdown confused.

[Lobo-Feroz]

Well, it's not canonical, that's for sure.

The proper way to indicate dialogue is the dash: —

Here's an english article mentioning that: https://www.thoughtco.com/introduction-to-spanish-punctuation-3080305

"Another difference you're likely to see often is the use of a dash—such as the ones separating this clause from the rest of the sentence—to indicate the beginning of dialogue. The dash is also used to end dialogue within a paragraph or to indicate a change in speaker, although none is needed at the end of dialogue if the end comes at the end of a paragraph. In other words, the dash can substitute for quotation marks under some circumstances."

I'm suspecting someone wrote there a couple of hyphens to simulate a dash, but the proper symbol is the dash.

BTW maybe we could include the dash in the engram-es keyboard. Maybe the AltGr version of the - _ key. It could be useful for writers.

Edit: maybe you can use \ to escape github markup?

[iandoug]

Since long dash is not in ASCII, nor on US keyboards, the usual method in plain text files is to use two hyphen / minus signs, usually also called dash.

In a previous message I pointed out that the suggested character set had both n-dash ( == hyphen / minus) and m-dash (long dash). I took the long dash out of "baseline" character set since it is not on qwerty-es.

Busy grabbing books off Gutenberg. There are 723.

[Lobo-Feroz]

Since long dash is not in ASCII, nor on US keyboards, the usual method in plain text files is to use two hyphen / minus signs, usually also called dash.

Ah, ok! But we must be aware that this is a hack, and not proper spanish language.

I took the long dash out of "baseline" character set since it is not on qwerty-es.

Oh, but for this one, I think it would be useful to have it. It's a proper punctuation symbol in spanish:
#7

Guillemets also aren't there in qwerty-es, and are the canonical way to do first level quotes.

[iandoug]

Okay will add it. Gutenberg has mostly plain-text files, some are labelled as utf-8.

The other issue (from experience scrubbing the English corpus) is that typographers/whoever typed the text are endlessly creative in using characters in Unicode that look like m-dashes but are not.

[Lobo-Feroz]

LOL. I suppose that's a direct consequence of not having a proper dash in the keyboard. I've also seen the guillemets written as double diples << and >> .

To be accurate here, I'm assuming the spanish dash symbol, AFAIK, it's an Em Dash U+2014 . There's a shorter En Dash, which is different from the hyphen, which I don't know if it's even used in spanish.

It seems the unicode-table.com website has also the spanish names for the symbols, in unicode-table.com/es/XXXX URL:

U+2010 is called here the Guión (hyphen)
U+2014 is called "Guión largo" (long hyphen), so I'm assuming this is the actual spanish dash "raya"

Wikipedia seems to confirm the "raya" is U+2014: https://es.wikipedia.org/wiki/Raya_(puntuaci%C3%B3n)#Usos_en_la_inform%C3%A1tica

But of course, Unicode has more similar symbols. I suppose they all have their uses, but I don't know which of these, apart from the hyphen and em dash, should be included. I suppose, none:

https://unicode-table.com/es/blocks/general-punctuation/

[iandoug]

The Gutenberg books have «, “ and —.
I only take ~20k extracts so it won't affect the character frequency too much. Arno is only interested in the letters anyway.

[iandoug]

Okay, Done fiddling with the books.
Checked 702 books, of which 517 were suitable ... these produced extracts of about 10kB. So I doubled the extract size,
Checked 702 books, of which 473 were suitable ... so we have 473 20kB extracts and 44 10kB extracts. All only using the defined character set. The rejected ones had unsavoury characters.

Analysis attached. I catted them all into one file, which is 11,275,853.
It's a bit small to just add to the Leipzig collection, which is GB. So am open to suggestions as to how to merge the frequencies.

Suppose convert both sets to percentages first, then average the percentages ...

There are some differences in the character frequency, given the different nature of the inputs.

Now need to rewrite end re-clean the Leipzig file.

spanish-books-freq.txt

[Lobo-Feroz]

Awesome work, @iandoug !

I have both monogram freqs open side by side. From e to d the order is the same, and even after that the differences are minimal until you reach the lower freqs and uncommon symbols. I don't know really how much weight they should carry compared to the Leipzig frequencies, but since the main characters seem to have very similar frequencies in both, I suppose any kind of averaging won't distort too much the result.

There seems to be a blank character near the bottom, between € and ¬ (or at least, it shows blank in my computer). Do you know why is that?

Also, could I have a sample of those unsavoury characters? Just to see if I can clean those up.

[iandoug]

Missing one is
ord 32 hex 20 == space. It got stuck in char array at beginning, but all occurrences were stored as ⍽.
The program went to the next book as soon as it found an unwanted char. Typically things like à or è etc ... I assume from names or French words etc.

Here's a few. For the English corpus I tried cleaning them but Unicode is vast (fix one, a few lines later there is a different one) and it got boring, so instead I switched to just selecting clean text. Though Gutenberg has a lot more English than Spanish.

Bad char è
Bad char ê
Bad char à
Bad char î
Bad char ö
Bad char °
Bad char è
Bad char ö
Bad char æ

[iandoug]

FWIW.... :-)

function cleanup()
{
global $line, $red, $off;

$line = preg_replace('/ΕυτΕλησ/u','word',$line);

$line = preg_replace("/</u",'',$line);
$line = preg_replace("/>/u",'',$line);
$line = preg_replace("/dlrs/u",'dollars',$line);
$line = preg_replace("/Dlrs/u",'Dollars',$line);
$line = preg_replace("/mln/u",'million',$line);
$line = preg_replace("/ cts /u",' cents ',$line);
$line = preg_replace("/ ct /u",' cent ',$line);

$line = preg_replace("/ Shr /u",' Share ',$line);
$line = preg_replace("/ shr /u",' share ',$line);
$line = preg_replace("/ stg /u",' sterling ',$line);
$line = preg_replace("/ Stg /u",' Sterling ',$line);

$line = preg_replace("/˗/u",'-',$line);
$line = preg_replace("/—/u",'-',$line);
$line = preg_replace("/–/u",'--',$line);
$line = preg_replace("/´/u","'",$line);
$line = preg_replace("/–/u",'--',$line);
$line = preg_replace("/―/u",'-',$line);

# fix en-dash U+2013 to LETTER MINUS
$line = preg_replace("/à/u",'a',$line); #
$line = preg_replace("/á/u",'a',$line); #
$line = preg_replace("/ä/u",'a',$line); #
$line = preg_replace("/â/u",'a',$line); #
$line = preg_replace("/ā/u",'a',$line); #
$line = preg_replace("/ă/u",'a',$line); #
$line = preg_replace("/å/u",'a',$line); #
$line = preg_replace("/ã/u",'a',$line); #
$line = preg_replace("/ą/u",'a',$line); #

$line = preg_replace("/À/u",'A',$line); #
$line = preg_replace("/Â/u",'A',$line); #
$line = preg_replace("/Å/u",'A',$line); #
$line = preg_replace("/Á/u",'A',$line); #
$line = preg_replace("/Ä/u",'A',$line); #
$line = preg_replace("/Ā/u",'A',$line); #

$line = preg_replace("/æ/u",'ae',$line); #
$line = preg_replace("/æ/u",'ae',$line); #
$line = preg_replace("/Æ/u",'Ae',$line); #
#$line = preg_replace("/Æ/u","Ae",$line); #

$line = preg_replace("/œ/u",'oe',$line); #
$line = preg_replace("/Œ/u",'Oe',$line); #

$line = preg_replace("/ç/u",'c',$line); #
$line = preg_replace("/Ç/u",'C',$line); #
$line = preg_replace("/ĉ/u",'c',$line); #

$line = preg_replace("/é/u",'e',$line); #
$line = preg_replace("/è/u",'e',$line); #
$line = preg_replace("/ê/u",'e',$line); #
$line = preg_replace("/ë/u",'e',$line); #
$line = preg_replace("/ē/u",'e',$line); #
$line = preg_replace("/ě/u",'e',$line); #
$line = preg_replace("/Ê/u",'E',$line); #
$line = preg_replace("/Ě/u",'E',$line); #
$line = preg_replace("/É/u",'E',$line); #
$line = preg_replace("/È/u",'E',$line); #
$line = preg_replace("/Ë/u",'E',$line); #
$line = preg_replace("/ę/u",'e',$line); #

$line = preg_replace("/î/u",'i',$line); #
$line = preg_replace("/ï/u",'i',$line); #
$line = preg_replace("/í/u",'i',$line); #
$line = preg_replace("/ì/u",'i',$line); #
$line = preg_replace("/ī/u",'i',$line); #
$line = preg_replace("/ĭ/u",'i',$line); #
$line = preg_replace("/Í/u",'I',$line); #
$line = preg_replace("/Î/u",'I',$line); #
$line = preg_replace("/Ï/u",'I',$line); #

$line = preg_replace("/č/u",'c',$line); #
$line = preg_replace("/ć/u",'c',$line); #
$line = preg_replace("/Č/u",'C',$line); #
$line = preg_replace("/ñ/u",'n',$line); #
$line = preg_replace("/ň/u",'n',$line); #
$line = preg_replace("/ń/u",'n',$line); #
$line = preg_replace("/Ñ/u",'N',$line); #

$line = preg_replace("/ß/u",'s',$line); #
$line = preg_replace("/ž/u",'z',$line); #
$line = preg_replace("/ź/u",'z',$line); #

$line = preg_replace("/š/u",'s',$line); #
$line = preg_replace("/ř/u",'r',$line); #
$line = preg_replace("/Ž/u",'Z',$line); #
$line = preg_replace("/ş/u",'s',$line); #
$line = preg_replace("/Š/u",'S',$line); #
$line = preg_replace("/ŝ/u",'s',$line); #
$line = preg_replace("/ł/u",'l',$line); #
$line = preg_replace("/ś/u",'s',$line); #
$line = preg_replace("/Ł/u",'L',$line); #
$line = preg_replace("/ż/u",'z',$line); #
$line = preg_replace("/Ś/u",'S',$line); #
$line = preg_replace("/ŕ/u",'r',$line); #
$line = preg_replace("/ħ/u",'h',$line); #
$line = preg_replace("/ņ/u",'n',$line); #
$line = preg_replace("/ĺ/u",'l',$line); #
$line = preg_replace("/Ş/u",'S',$line); #
$line = preg_replace("/Ĺ/u",'L',$line); #
$line = preg_replace("/ď/u","d'",$line); #

$line = preg_replace("/λ/u",'lambda',$line); #
$line = preg_replace("/Ř/u",'R',$line); #
$line = preg_replace("/ť/u",'t',$line); #
$line = preg_replace("/ţ/u",'t',$line); #

$line = preg_replace("/ô/u",'o',$line); #
$line = preg_replace("/ö/u",'o',$line); #
$line = preg_replace("/ò/u",'o',$line); #
$line = preg_replace("/ō/u",'o',$line); #
$line = preg_replace("/ŏ/u",'o',$line); #
$line = preg_replace("/ó/u",'o',$line); #
$line = preg_replace("/ô/u",'o',$line); #
$line = preg_replace("/ø/u",'o',$line); #
$line = preg_replace("/õ/u",'o',$line); #
$line = preg_replace("/Ó/u",'O',$line); #
$line = preg_replace("/Ö/u",'O',$line); #
$line = preg_replace("/Ø/u",'O',$line); #
$line = preg_replace("/Ô/u",'O',$line); #
$line = preg_replace("/Õ/u",'O',$line); #

$line = preg_replace("/ü/u",'u',$line); #
$line = preg_replace("/ú/u",'u',$line); #
$line = preg_replace("/û/u",'u',$line); #
$line = preg_replace("/ù/u",'u',$line); #
$line = preg_replace("/ū/u",'u',$line); #
$line = preg_replace("/ŭ/u",'u',$line); #
$line = preg_replace("/Ü/u",'U',$line); #
$line = preg_replace("/Ú/u",'U',$line); #
$line = preg_replace("/ů/u",'u',$line); #
$line = preg_replace("/¨/u",'u',$line); #
$line = preg_replace("/Û/u",'U',$line); #

$line = preg_replace("/ /u",' ',$line); #
$line = preg_replace("/§/u",'',$line); #
$line = preg_replace("/“/u",'"',$line); #
$line = preg_replace("/”/u",'"',$line); #
$line = preg_replace("/’/u","'",$line); #
$line = preg_replace("/‘/u","'",$line); #

$line = preg_replace("/ÿ/u","y",$line); #
$line = preg_replace("/ý/u","y",$line); #
$line = preg_replace("/Ÿ/u","Y",$line); #
$line = preg_replace("/ŷ/u","y",$line); #

$line = preg_replace("/ŵ/u","w",$line); #

$line = preg_replace("/×/u","x",$line); #
$line = preg_replace("/÷/u","/",$line); #
$line = preg_replace("/€/u","EUR",$line); #
$line = preg_replace("/£/u","GBP",$line); #
$line = preg_replace("/¥/u","JPY",$line); #
$line = preg_replace("/¶/u","",$line); #
$line = preg_replace("/©/u","Copyright ",$line); #
$line = preg_replace("/№/u","#",$line); #
$line = preg_replace("/•/u","",$line); #
$line = preg_replace("/∘/u","Open Circle",$line); #

$line = preg_replace("/…/u","... ",$line); #
$line = preg_replace("/·/u","-- ",$line); #
$line = preg_replace("/⁂/u","* ",$line); #
$line = preg_replace("/❦/u","*",$line); #
$line = preg_replace("/┌/u","",$line); #
$line = preg_replace("/─/u","",$line); #
$line = preg_replace("/┐/u","",$line); #
$line = preg_replace("/┬/u","",$line); #
$line = preg_replace("/│/u","",$line); #
$line = preg_replace("/├/u","",$line); #
$line = preg_replace("/┼/u","",$line); #
$line = preg_replace("/┤/u","",$line); #
$line = preg_replace("/└/u","",$line); #
$line = preg_replace("/┘/u","",$line); #
$line = preg_replace("/┴/u","",$line); #
$line = preg_replace("/¼/","¼",$line); #
$line = preg_replace("/½/","½",$line); #
$line = preg_replace("/⁶⁄₁₀/","⅗",$line); #
$line = preg_replace("/¾/","¾",$line); #
$line = preg_replace("/⅜/","⅜",$line); #
$line = preg_replace("/⅔/","⅔",$line); #
$line = preg_replace("/⅙/","⅙",$line); #
$line = preg_replace("/⅛/","⅛",$line); #
$line = preg_replace("/⅓/","⅓",$line); #
$line = preg_replace("/⅘/","⅘",$line); #
$line = preg_replace("/⅕/","⅕",$line); #
$line = preg_replace("/⅞/","⅞",$line); #
$line = preg_replace("/⅗/","⅗",$line); #
$line = preg_replace("/⅖/","⅖",$line); #

$line = preg_replace("/δ/"," delta ;",$line); #
$line = preg_replace("/Δ/","Delta';",$line); #
$line = preg_replace("/✝/","Died ;",$line); #
$line = preg_replace("/°/u"," degrees",$line); #
$line = preg_replace("/¿/u","",$line); #
$line = preg_replace("/±/u","±",$line); #
$line = preg_replace("/⅚/u","⅚",$line); #
$line = preg_replace("/♥/u","heart",$line); #

$line = preg_replace("/¹/u","&sup1;",$line);

$line = preg_replace("/²/u","^2",$line);
$line = preg_replace('/″/u','"',$line);
$line = preg_replace("/′/u","'",$line); #
$line = preg_replace("/‐/u","-",$line); #
$line = preg_replace("/✓/u","tickmark ",$line); #
$line = preg_replace("/∼/u","~",$line); #

$line = preg_replace('/→/u','->',$line);
$line = preg_replace('/™/u','(TM)',$line);
$line = preg_replace('/⅝/u','2/3',$line);
$line = preg_replace('/µm/u','micrometre',$line);
$line = preg_replace('/μm/u','micrometre',$line);
$line = preg_replace('/φ/u','phi',$line);

$line = preg_replace('/µV/u',' microvolt',$line);
$line = preg_replace('/kΩ/u','kilo-Ohm',$line);
$line = preg_replace('/10Ω/u','10 Ohm',$line);
$line = preg_replace('/MΩ/u','1 mega-Ohm',$line);
$line = preg_replace('/µs/u',' microseconds',$line);
$line = preg_replace('/µW/u',' microwatts',$line);
$line = preg_replace('/µT/u',' microtesla',$line);
$line = preg_replace('/Ω/u','ohm',$line);
$line = preg_replace('/Ω/u','ohm',$line);

$line = preg_replace('/μ/u',' micro',$line);
$line = preg_replace('/µ/u',' micro',$line);

$line = preg_replace('/Þ/u','Th',$line);
$line = preg_replace('/đ/u','th',$line);
$line = preg_replace('/ð/u','th',$line);
$line = preg_replace('/þ/u','th',$line);

$line = preg_replace('/¡/u','!',$line);
$line = preg_replace('/¢/u',' cents',$line);

$line = preg_replace('/θ/u','theta',$line);
$line = preg_replace('/Β/u','B',$line);
$line = preg_replace('/Ε/u','E',$line);

$line = preg_replace('/π/u',' pi',$line);
$line = preg_replace('/√/u',' square root ',$line);
$line = preg_replace('/³/u',' cubed ',$line);
$line = preg_replace('/α/u',' alpha ',$line);
$line = preg_replace('/β/u',' beta ',$line);
$line = preg_replace('/γ/u','gamma',$line);
$line = preg_replace('/ω/u','omega',$line);
$line = preg_replace('/Ŷ/u','Y',$line);
$line = preg_replace('/®/u',' (Registered Trademark) ',$line);
$line = preg_replace('/←/u',' left arrow ',$line);
$line = preg_replace('/↓/u',' down arrow ',$line);
$line = preg_replace('/▾/u',' down triangle ',$line);
$line = preg_replace('/○/u',' white circle ',$line);

$line = preg_replace('/Σ/u','Sigma',$line);
$line = preg_replace('/∞/u','infinity',$line);
$line = preg_replace('/ε/u','epsilon',$line);
$line = preg_replace('/σ/u','sigma',$line);
$line = preg_replace('/ψ/u','psi',$line);
$line = preg_replace('/Ψ/u','Psi',$line);
$line = preg_replace('/η/u','eta',$line);
$line = preg_replace('/Φ/u','Phi',$line);
$line = preg_replace('/ρ/u','rho',$line);
$line = preg_replace('/ν/u','nu',$line);
$line = preg_replace('/Μ/u','Mu',$line);
$line = preg_replace('/χ/u','chi',$line);
$line = preg_replace('/ζ/u','zeta',$line);
$line = preg_replace('/ι/u','iotr',$line);
$line = preg_replace('/κ/u','kappa',$line);
$line = preg_replace('/ξ/u','xi',$line);
$line = preg_replace('/ο/u','omicron',$line);
$line = preg_replace('/τ/u','tau',$line);
$line = preg_replace('/υ/u','upsilon',$line);
$line = preg_replace('/Υ/u','Upsilon',$line);
$line = preg_replace('/Π/u','Pi',$line);
$line = preg_replace('/Γ/u','Gamma',$line);

$line = preg_replace('/Ο/u','Omicron',$line);
$line = preg_replace('/Θ/u','Theta',$line);
$line = preg_replace('/Ζ/u','Zeta',$line);

$line = preg_replace('/≃/u','approximately equal to',$line);

$line = preg_replace('/♭/u','Flat ',$line);
$line = preg_replace('/♯/u','Sharp ',$line);
$line = preg_replace('/♮/u','Natural ',$line);

$line = preg_replace('/≥/u',' greater than or equal to ',$line);
$line = preg_replace('/≤/u',' less than or equal to ',$line);
$line = preg_replace('/∗/u','*',$line);

[Lobo-Feroz]

The program went to the next book as soon as it found an unwanted char. Typically things like à or è etc ... I assume from names or French words etc.

Yeah, all these characters seem french, german, etc. I agree that if the files have that many "bad chars" it might be best to discard them.

Is there any way I can help you at this point?

[iandoug]

Not yet, have started the cleaning process on the Leipzig files, discarding lines with unknown words. Last runs took about 100 minutes. Will see what is left after.

[binarybottle]

Fantastic work! I would normalize by volume, then give relative frequencies for me to chug on.