Listings grabber for UK Freeview

[honir]

I've written a new grabber for Freeview (UK) broadcasts, using the freeview.co.uk tv guide.

Please note the conditions of use of the Freeview website specify personal use only, and specifically exclude you sharing the data with anyone else.

As with all grabbers, you should download data only for the channels you want to watch. Anything else just wastes your time and hits the source website with unnecessary usage. You select the channels to download during the configuration -- you can easily repeat the configuration at any time to add or remove channels.

[misar1]

I have been testing the grabber and it seems to work OK when "lightly loaded" but fails as the load (channels * days) increases. I used consistent .conf files, always starting with the same initial set of channels and adding more to the end to increase the number. Without any proper progress reporting by the script it is unclear how to narrow down the issue. I cannot find a cache location so could there be a limit on the total number of programmes that can be retained in memory before writing data to the .xml?

1 day by 20 channels: ended after 20 min, OK
1 day by 40 channels: ended after 47 min, OK
1 day by 60 channels: ended after 31 min with errors:
Timezone is +0100
getting list of channels: ##################################################
getting listings: ###unknown attribute: SL
##############Use of uninitialized value $p_ser in substitution (s///) at \XMLTV_1\par-6d696b6573\cache-e1b1563a021779c5084d873b11553dd9e308b482\inc/script/tv_grab_uk_freeview line 656.
#####Not a HASH reference at \XMLTV_1\par-6d696b6573\cache-e1b1563a021779c5084d873b11553dd9e308b482\inc/script/tv_grab_uk_freeview line 631.
.xml was empty

2 day by 9 channels: ended after 18 min, OK
2 day by 15 channels: ended after 30 min, OK
2 day by 20 channels: ended after 43 min, with error:
Code point \u0019 is not a valid character in XML at \XMLTV_1\par-6d696b6573\cache-e1b1563a021779c5084d873b11553dd9e308b482\inc\lib/XMLTV.pm line 2197.
.xml listed 20 channels but with programmes for only 9 channels

2 day by 38 Channels: ended after 81 min with an error message:
(didn't save message but it mentioned "hash")
.xml had correct list of channels but no programmes.

Added the .conf for the maximum set.
tv_grab_uk_freeview_wintv.zip

[honir]

Thanks for the feedback.

The two errors you have found are not to do any "loading" or limits in the program. What you report appear to be data weirdness in the Freeview site's data stream.

I can fix this for you but I would need to know the specific channel+day on which it occurs so I can inspect the raw data coming from Freeview.

It is easy to "narrow down the issue":

1 day by 20 channels: ended after 20 min, OK
1 day by 40 channels: ended after 47 min, OK
1 day by 60 channels: ended after 31 min with errors:

So do a run with channels 41-50. If the error occurs then do a run with 41-45. Etc. until the error goes away. Then you know the error is in the batch you just excluded. You can identify the error channel in only a few runs.

Since the data are cached (for a short period) it is very fast to do these repeat runs to home in on the offending channel.

If you can identify the channel+day then open a new issue report, and I will take a look for you.

[misar1]

Thanks for the quick reply. I don't mind trying your suggestion but have another question first.

If the issue is a rogue channel, why did the first 20 channels of the .config I attached worked OK with 1 day but the same set of 20 channels with 2 days ended with the error I posted previously? That suggests the error could be due to a specific programme entry (which changes daily) rather than a particular channel. Today's (1 day) listings will already have different programmes from yesterday making error tracking a moving target.