CMS 2024 queries

[nrllh]

Makes progress on #3608

[mgifford]

This should be using httparchive.all.pages instead of  httparchive.technologies.2024_06_01_*  

This will give access to secondary resources and also be more future-proof.

[nrllh]

This should be using httparchive.all.pages instead of httparchive.technologies.2024_06_01_*

This will give access to secondary resources and also be more future-proof.

Thanks! I'm currently adjusting the queries, the results remain very stable and there are almost no changes.

[mgifford]

Is it worth adding ? 

WHERE is_root_page

[mgifford]

In testing this I'm getting "There is no data to display." as an error.

I don't think it was updated correctly. Maybe the dateformat is wrong?

[mgifford]

I got better results with this:

#standardSQL
# CMS popularity per geo
WITH geo_summary AS (
SELECT
`chrome-ux-report`.experimental.GET_COUNTRY(country_code) AS geo,
IF(device = 'desktop', 'desktop', 'mobile') AS client,
origin,
COUNT(DISTINCT origin) OVER (PARTITION BY country_code, IF(device = 'desktop', 'desktop', 'mobile')) AS total
FROM
`chrome-ux-report.materialized.country_summary`
WHERE
yyyymm = 202406
)


SELECT
client,
geo,
cms,
COUNT(0) AS pages,
ANY_VALUE(total) AS total,
COUNT(DISTINCT url) / ANY_VALUE(total) AS pct
FROM (
-- Step 1: Extract distinct URLs from geo_summary (grouped by country and device).
SELECT DISTINCT
geo,
client,
CONCAT(origin, '/') AS url,
total
FROM
geo_summary
) JOIN (
-- Step 2: Join with the CMS data from httparchive.all.pages for the top CMS per country.
SELECT DISTINCT
client,
technologies.technology AS cms,
page AS url
FROM
`httparchive.all.pages`,
UNNEST(technologies) AS technologies,
UNNEST(technologies.categories) AS categories
WHERE
categories = 'CMS' AND
technologies.technology != '' AND
date = '2024-06-01' AND
is_root_page
) USING (client, url)
GROUP BY
client,
geo,
cms
HAVING
pages > 1000 -- Include only CMSes with more than 1000 pages in a country.
ORDER BY
pages DESC

[kevinfarrugia]

I do not get this error. Can you confirm it is still an error please?

[tunetheweb]

Overall LGTM but with one suggestion.

Let me know when good to merge

[kevinfarrugia]

From my end this is ready to be merged. @sirjonathan can you confirm?

[tunetheweb]

LGTM.. Let me know if good to merge.