Create API endpoints for SEO scores

[leonidasmi]

Context

Summary

This PR can be summarized in the following changelog entry:

• Creates API endpoints for SEO scores

Relevant technical choices:

• The API endpoint returns a 400 error code when:
  • The content type in the request is not indexable (aka, excluded/disabled from indexable creation), like non-public post types
  • The filtering taxonomy, if included, is not a public taxonomy that has a REST URL or it's not associated with the given content type. It should also be an active filtering pair (because of the wpseo_{$content_type}_filtering_taxonomy filter or because it's a hardoced filtering pair, described in this PR)
  • The filtering term, if included, is not a part of the given taxonomy
• If the API request includes a taxonomy that has a false query_var, there will be no View URLs returned, because there can't be a URL that filters the CPT page on that taxonomy. In those cases, we will have no View buttons in the UI

Test instructions

Test instructions for the acceptance test before the PR gets merged

This PR can be acceptance tested by following these steps:

Note: We're going to use POSTMAN to test the API endpoint, so use this post to learn how to authenticate.

For posts and categories:

• Create a bunch of posts, some with key phrases and some with not.
  • Make sure you spread those posts in different categories, so we can check the category filtering later on
• You can check in the indexable table, each post's primary_focus_keyword_score, to have a view on what to expect
• Do a GET request to /wp-json/yoast/v1/seo_scores?contentType=post and confirm that you get a response similar to the below, with the amounts representing what you have in the database:

{
    "scores": [
        {
            "name": "good",
            "amount": 2,
            "links": {
                "view": "http://example.com/wp-admin/edit.php?post_status=publish&post_type=post&seo_filter=good"
            }
        },
        {
            "name": "ok",
            "amount": 1,
            "links": {
                "view": "http://example.com/wp-admin/edit.php?post_status=publish&post_type=post&seo_filter=ok""
            }
        },
        {
            "name": "bad",
            "amount": 2,
            "links": {
                "view": "http://example.com/wp-admin/edit.php?post_status=publish&post_type=post&seo_filter=bad"
            }
        },
        {
            "name": "notAnalyzed",
            "amount": 1,
            "links": {
                "view": "http://example.com/wp-admin/edit.php?post_status=publish&post_type=post&seo_filter=na"
            }
        }
    ]
}

• The above represent a state where you have 2 posts with primary_focus_keyword_score over 70, 1 post with primary_focus_keyword_score between 41 and 70, 2 posts with under 40 and 1 post without keyphrase.
• Go to each of the links and confirm that you get to the right page and that you see the same amount of posts as the ones you got from the response.
• Now, turn one of your posts with a primary_focus_keyword_score over 70 into a draft and repeat the request -> you should get one less amount in the good object.
• Now, turn one of your posts with with a primary_focus_keyword_score over 70 into a noindexed post and repeat the request -> you should get one less amount in the good object and when visiting the link you get the same amount of posts in the filtered posts page.
• Now do a GET request with taxonomy filtering enabled to /wp-json/yoast/v1/seo_scores?contentType=post&taxonomy=category&term=<TERM_ID> (replace TERM_ID with the correct term ID)
• Now confirm that the response changed accordingly. Also check the links get you to the right filtered state in the posts page.
• Do the same tests with WooCommerce enabled and querying for /wp-json/yoast/v1/seo_scores?contentType=product&taxonomy=product_cat&term=<TERM_ID> - pay extra focus on the links to confirm that they get you to the right filtered state in the products page
• Do the same with a CPT and quering for /wp-json/yoast/v1/seo_scores?contentType=<CPT_NAME>&taxonomy=<TAX_NAME>&term=<TERM_ID> - pay extra focus on the links to confirm that they get you to the right filtered state in the CPT page
  • Follow this PR's instructions for Added filter via hook, for a taxonomy with custom REST API URL in order to be able to create a CPT and connect it to a filtering taxonomy.

For when SEO scores are at the end of the ranges:

• We need to have posts that have SEO scores equal to 40, 41, 70, 71, to make sure they are categorized as bad, ok, ok and good respectively
• Have 4 posts and go to their rows in the indexable table and change their primary_focus_keyword to to 40, 41, 70, 71.
• Also go to their rows in the postmeta table and change their _yoast_wpseo_linkdex column to the same
• Do the tGET request to /wp-json/yoast/v1/seo_scores?contentType=post and when you click on each score's link, confirm that you get the same amount of posts with the amount attribute of each post.

For when the post type requested is excluded from indexable creation:

• Add the following filter, to exclude page from indexable creation:

add_filter( 'wpseo_indexable_excluded_post_types', 'exclude_page' );
function exclude_page( ) {
	return ['page',];
}

• Do a GET request to /wp-json/yoast/v1/seo_scores?contentType=page
• Confirm that you get a 400 response code with the Invalid content type error.

For when the term requested is not part of the taxonomy:

• Do a GET request to /wp-json/yoast/v1/seo_scores?contentType=post&taxonomy=category&term=<TERM_ID> (replace TERM_ID with an ID that doesn't exist in terms)
• Confirm that you get a 400 response code with the Invalid term error.

For a custom taxonomy that doesn't have links because it has a false query_var:

• Have the Yoast Test Helper active and enable the custom post types
• Create a custom taxonomy that has a false query_var attribute and assign it to books:

function register_genre_taxonomy() {
    $args = array(
        'label'        => 'Alternative genres',
        'public'       => true,
        'hierarchical' => true, // true if it's like categories, false for tags
        'query_var'    => false,
        'show_in_rest' => true, // Enable REST API
        'rest_base'    => 'alternative_genres', // Custom REST base
        'rest_namespace' => 'my_custom_namespace/v1', // Custom REST namespace
        'rewrite'      => array('slug' => 'alternative-genre'), // Custom slug
    );

    register_taxonomy('alternative-genre', array('book'), $args);
}
add_action('init', 'register_genre_taxonomy');

• Create a term for that taxonomy and then create a book and assign it to that term.
• Do a GET request to /wp-json/yoast/v1/seo_scores?contentType=book&taxonomy=alternative-genre&term=<TERM_ID> (replace TERM_ID with the correct term ID)
• Confirm that you get a 400 response code with the Invalid taxonomy error
• Add the following filter, which enables us to have the alternative-genre taxonomy as filter for books and repeat the GET request:

add_filter( 'wpseo_book_filtering_taxonomy', 'book_filtering_taxonomy' );
function book_filtering_taxonomy ( ) {
	return 'alternative-genre';
}

• Confirm that you get the following response (depending on the SEO score of the book you created, the right score will be incremented to 1):

{
    "scores": [
        {
            "name": "good",
            "amount": 0,
            "links": []
        },
        {
            "name": "ok",
            "amount": 0,
            "links": []
        },
        {
            "name": "bad",
            "amount": 0,
            "links": []
        },
        {
            "name": "notAnalyzed",
            "amount": 1,
            "links": []
        }
    ]
}

• Note the empty arrays in the links attributes - In those cases, we will have no View buttons in the UI.

Relevant test scenarios

• Changes should be tested with the browser console open
• Changes should be tested on different posts/pages/taxonomies/custom post types/custom taxonomies
• Changes should be tested on different editors (Default Block/Gutenberg/Classic/Elementor/other)
• Changes should be tested on different browsers
• Changes should be tested on multisite

Test instructions for QA when the code is in the RC

• QA should use the same steps as above.

Impact check

This PR affects the following parts of the plugin, which may require extra testing:

UI changes

• This PR changes the UI in the plugin. I have added the 'UI change' label to this PR.

Other environments

• This PR also affects Shopify. I have added a changelog entry starting with [shopify-seo], added test instructions for Shopify and attached the Shopify label to this PR.

Documentation

• I have written documentation for this change. For example, comments in the Relevant technical choices, comments in the code, documentation on Confluence / shared Google Drive / Yoast developer portal, or other.

Quality assurance

• I have tested this code to the best of my abilities.
• During testing, I had activated all plugins that Yoast SEO provides integrations for.
• I have added unit tests to verify the code works as intended.
• If any part of the code is behind a feature flag, my test instructions also cover cases where the feature flag is switched off.
• I have written this PR in accordance with my team's definition of done.
• I have checked that the base branch is correctly set.

Innovation

• No innovation project is applicable for this PR.
• This PR falls under an innovation project. I have attached the innovation label.
• I have added my hours to the WBSO document.

Fixes https://github.com/Yoast/reserved-tasks/issues/321

[coveralls]

Pull Request Test Coverage Report for Build 84c653c5441f79891f1b0a96f89b4fc3eb384333

Warning: This coverage report may be inaccurate.

This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

• For more information on this, see Tracking coverage changes with pull request builds.
• To avoid this issue with future PRs, see these Recommended CI Configurations.
• For a quick fix, rebase this PR at GitHub. Your next report should be accurate.

Details

• 0 of 199 (0.0%) changed or added relevant lines in 13 files are covered.
• No unchanged relevant lines lost coverage.
• Overall first build on 321-create-api-endpoints-for-the-seoreadability-score at 51.094%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
src/dashboard/domain/taxonomies/taxonomy.php	0	2	0.0%
src/dashboard/domain/content-types/content-type.php	0	3	0.0%
src/dashboard/application/content-types/content-types-repository.php	0	7	0.0%
src/dashboard/domain/seo-scores/abstract-seo-score.php	0	8	0.0%
src/dashboard/infrastructure/content-types/content-types-collector.php	0	9	0.0%
src/dashboard/domain/seo-scores/bad-seo-score.php	0	10	0.0%
src/dashboard/domain/seo-scores/good-seo-score.php	0	10	0.0%
src/dashboard/domain/seo-scores/no-seo-score.php	0	10	0.0%
src/dashboard/domain/seo-scores/ok-seo-score.php	0	10	0.0%
src/dashboard/application/seo-scores/seo-scores-repository.php	0	11	0.0%

Totals
Change from base Build c8821a7deff0184df569923576a0b2542cf39394:	51.1%
Covered Lines:	16416
Relevant Lines:	32129

---

💛 - Coveralls

[schlessera]

@leonidasmi For the query with taxonomy filtering, can you try if this version is producing the same output and whether it is more efficient:

SELECT 
    SUM(I.primary_focus_keyword_score < 41) AS needs_improvement,
    SUM(I.primary_focus_keyword_score BETWEEN 41 AND 69) AS ok,
    SUM(I.primary_focus_keyword_score >= 70) AS good,
    SUM(I.primary_focus_keyword_score IS NULL) AS not_analyzed
FROM %i AS I
INNER JOIN %i AS tr ON I.object_id = tr.object_id
INNER JOIN %i AS tt ON tr.term_taxonomy_id = tt.term_taxonomy_id
WHERE (I.post_status = 'publish' OR I.post_status IS NULL)
    AND I.object_type = 'post'
    AND I.object_sub_type IN (%s)
    AND tt.term_id = %d
    AND tt.taxonomy = %s

I haven't tested it yet for lack of environment right now, so double-check if everything is as expected.

[schlessera]

Hmm, you might need to be more explicit to make sure the expressions are compatible across all supported DB versions.
Here's another take:

SELECT 
    SUM(CASE WHEN I.primary_focus_keyword_score < 41 THEN 1 ELSE 0 END) AS needs_improvement,
    SUM(CASE WHEN I.primary_focus_keyword_score >= 41 
              AND I.primary_focus_keyword_score < 70 THEN 1 ELSE 0 END) AS ok,
    SUM(CASE WHEN I.primary_focus_keyword_score >= 70 THEN 1 ELSE 0 END) AS good,
    SUM(CASE WHEN I.primary_focus_keyword_score IS NULL THEN 1 ELSE 0 END) AS not_analyzed
FROM %i AS I
    INNER JOIN %i AS tr ON I.object_id = tr.object_id
    INNER JOIN %i AS tt ON tr.term_taxonomy_id = tt.term_taxonomy_id
WHERE (I.post_status = 'publish' OR I.post_status IS NULL)
    AND I.object_type = 'post'
    AND I.object_sub_type IN (%s)
    AND tt.term_id = %d 
    AND tt.taxonomy = %s

[leonidasmi]

For the query with taxonomy filtering, can you try if this version is producing the same output and whether it is more efficient

@schlessera I compared the one you shared with the one I already have and while they produce the same results, they also have similar, if not identical, performance.

Specifically, their EXPLAINs are exactly the same on my main testing case:

---

I then tried to tinker with my wp_term_taxonomy table, to add rows that have different term_taxonomy_id and term_id columns, to check if that produced different EXPLAINs between the two queries, and realized that we probably don't have to have a second JOIN/nested SELECT, because term_taxonomy_id can never be different from term_id, ever since WP 4.3.

Which means that we can simplify the query likeso:

SELECT
	COUNT(CASE WHEN I.primary_focus_keyword_score >= 1 AND I.primary_focus_keyword_score <= 40 THEN 1 END) AS `bad`,
	COUNT(CASE WHEN I.primary_focus_keyword_score >= 71 AND I.primary_focus_keyword_score <= 100 THEN 1 END) AS `good`,
	COUNT(CASE WHEN primary_focus_keyword_score IS NULL THEN 1 END) AS `notAnalyzed`,
	COUNT(CASE WHEN I.primary_focus_keyword_score >= 41 AND I.primary_focus_keyword_score <= 70 THEN 1 END) AS `ok`
FROM `wp_yoast_indexable` AS I
WHERE ( I.post_status = 'publish' OR I.post_status IS NULL )
	AND I.object_type IN ('post')
	AND I.object_sub_type IN ('post')
	AND I.object_id IN (
		SELECT object_id
		FROM `wp_term_relationships`
		WHERE term_taxonomy_id = 3
	)