Aquitanian Staff Finding Job

[DeannaLC]

I am working on the first job toward Aquitanian Pitch Finding with @fujinaga and @martha-thomae to find a center line in Salamanca Missal. Since the line isn't visible enough to be found by the computer, I plan to work off the observation that the midpoint of the text lies in the center between two lyric lines.

I am first planning to use Pixel to create 2-3 layers to try out.

1. One layer will contain the just the neumes
2. The other layer(s) will have the lyrics and the red lines
   • This can either be 1 layer with both the lyrics and the red line, or one layer with the lyrics and one with the red line

Another observation about finding the center line was that the bottom of the larger text on the other side of the page from the neumes lines up with the center line, so if needed or if this works better, it could be classified as another layer.

I am planning to reference the Staff Distance job for reference, but would also be very happy to hear any other suggestions for jobs I can use for a starting point.

[martha-thomae]

I am planning to reference the Staff Distance job for reference, but would also be very happy to hear any other suggestions for jobs I can use for a starting point.

Maybe you could look at the Miyao Staff Finding job. @fujinaga, @annamorphism, @JoyfulGen, @kyrieb-ekat would this be the right job for Deanna to grab inspiration for what she is implementing? She wants to extrapolate a full straight line from segments of the red lines in the lyrics (see example image).

Folio 020v, Salamanca missal

[martha-thomae]

@elsinhadl said:

• It is safe to assume that the one staff line is in the midpoint o two text lines
• However, although the red lines at the bottom of the text appear in many manuscripts, they don’t always appear throughout the whole text (e.g., they don't appear when the text is tightly close together). So we will need to eventually use the text as reference rather than the red lines.

We need to find the baseline of the text to use that as reference and then find the midpoint staff line with that.

[martha-thomae]

Next steps:

• Include the job as it is in Rodan (make your own branch) to see if it works as expected. So far, Deanna has detected the red lines and expanded them and detected the line between these as the one staff line. The output of the job should be similar to the output of Miyao Staff Finding job, as this is the job preceding the Heuristic Pitch Finding

  

• Then move to use the baseline of the text rather than the red lines

[DeannaLC]

I'm working through a couple approaches in order to find the music reference line and will evaluate which one works better. I'm using cv2 for image processing on one layer I get with Pixel that has both lyrics and red lines. Since each folio has two columns of music and text, I also need to work on a good method of column separation.

Approach 1 - Strikethrough line

This uses HoughLines in order to detect lines, then draws a line through the lyrics and text. Many lines are produced so I filter out the lines to keep the straightest one through each line of text. I sort the lines by their y values then take pairs of them and draw a line in between.

Approach 2 - Bounding Box/Underlining

I use cv2's bounding boxes that I take the bottom line of so it underlines the text rather than doing a strikethrough. There's a slight issue with letters such as y which go below the text and change the box. I'm working on figuring out how to expand the lines, then use that to get the midpoint.

(I put random colors based on whether sets of lines are grouped so the same image has different colors between the pictures)
Bounding Boxes around text:

Underline by taking bounding box points:

[annamorphism]

I like Approach 2 (since ultimately we want to get the line the letters are sitting on, not their midpoint.) I can see that letters like y (or p, or q) with significant descenders would be a problem, though...

[fujinaga]

I'm going to suggest something drastic. Looking at this image (@martha-thomae and @elsinhadl: can you find about ten different sample images from six to eight different Acquitanian manuscripts and upload them here?), it looks to me like it will be very hard to find the correct positions of the staff line heuristically. I would suggest, at least for now, manually draw the staff lines for each page (using Pixel?). Then use that image to create the output similar to Miyao's output. An advantage of manually drawing the staff line is you will have ground-truth data to check any automatic Aquatanian staff-finding algorithms.

[martha-thomae]

Hi @fujinaga! Sure, we will look for a few examples images and upload them here.

@DeannaLC, for now, let's focus on what Ichiro says here:

I would suggest, at least for now, manually draw the staff lines for each page (using Pixel?). Then use that image to create the output similar to Miyao's output. An advantage of manually drawing the staff line is you will have ground-truth data to check any automatic Aquatanian staff-finding algorithms.

After obtaining an output similar to Miayo Staff Finding, please work on the new "Heuristic Pitch Finding" for Aquitanian notation (issue #1240). Please save the layers in our repo ECHOES/OMR_Portuguese_Sources, as we can later use them for ground truth to train a machine learning algorihtm to detect this by itself.

In the meantime, we will check the results of your two approaches. I will consult with @elsinhadl and @fujinaga. So please, @DeannaLC, upload them here! Thank you!

[martha-thomae]

@DeannaLC for the Miyao thing. First, try to look with a square notation manuscript (maybe Salzinnes?) what Miayo Staff Finding does, and then you can adapt it for our purposes of detecting the path of our one music reference line for Aquitanian notation.

[fujinaga]

@DeannaLC if you need help about how to use Rodan or looking for sample files, contact @JoyfulGen or @kyrieb-ekat .

[elsinhadl]

Hello!
I hope these images will suffice. Please do not hesitate to let me know if there is anything else I can do. Thank you, Elsa

[DeannaLC]

@fujinaga @martha-thomae @elsinhadl
Here's the outputs from two different folios of the two approaches from the heuristic staff finding.

Folio 043v Seq 001

Strikethrough Line Approach:

Bounding Box Underline Approach:

Folio 029v Seq 001

Strikethrough Line Approach:

Bounding Box Underline Approach:

[elsinhadl]

Hello,
Both approaches present some errors due to the fact that the page is not flat but slightly convex. Although neither of them is ideal, I have a slight preference for the Bounding Box Underline Approach.

[martha-thomae]

If neither of them is good enough @elsinhadl, then we should forget about a heuristic approach and provide the music reference line manually with pixel for now (like Ich suggested). We can later use that to train a machine learning algorithm to get the reference music line in the future. And in the mean time, we can use the Pixel data (the manually written "music reference lines") to move along on designing a "Heuristic pitch detection for Aquitanian neumes," especially because the other examples of Aquitanian manuscripts provided before don't seem to have the issue of predicting the music reference line. Thoughts, @fujinaga and @elsinhadl ?

[fujinaga]

These images are great!! Thank you @elsinhadl ! These images have now convinced me that, in general, finding the staff lines heuristically will be very difficult. On the other hand, I think, when the staff line is coloured (e.g., red), I think we can train Paco's method to automatically recognize them eventually.

[elsinhadl]

Fine. Even if it this has no effect on the OMR, I just wanted to add that, among all the surviving manuscripts with Aquitanian neumes, those without the coloured line are in the minority. It was only in the early days of Aquitanian neumes that music scribes used the dry-point lines, ruled for text reference, to position the neumes around them. Later, they employed red, brown or yellow ink for the music line.

[martha-thomae]

After evaluating the results and discussing together, this was the conclusion at which we arrived:

@martha-thomae asked:

Would it be harder to manually correct all the "pitches" (vertical location of the notes) in Neon at the end or to manually draw the music reference lines in Pixel?

@elsinhadl answered:

It will be harder to correct them at the end. It is better to manually draw the music reference lines in Pixel and have less to correct at the end.

So we are following Ichiro's suggestion and moving on to the "heuristic pitch finding."

[martha-thomae]

Deanna is working now on getting the music reference line with pixel, as we agreed, and modifying Miyao Staff Finding to work with one staff line (see this new issue #1254). If she has any questions about Miyao Staff Finding and how to adapt it to work with one staff line, she will be using that issue (#1254). For now, I will close this issue.