Skip to content
This repository has been archived by the owner on Mar 2, 2021. It is now read-only.

CSVify everything #36

Open
wants to merge 3 commits into
base: master
Choose a base branch
from
Open

CSVify everything #36

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
5ac8c52
CSVify everything
hoshsadiq Nov 8, 2020
1fb6c4d
formatting
hoshsadiq Nov 8, 2020
d51cbc8
update modules
hoshsadiq Nov 9, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
187 changes: 53 additions & 134 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
# m3ufilter

This is a utility that will allow you to cleanup your M3U/M3U8/M3U+ files. This can change feed titles, names, tvg attributes, add/remove additional entries and much more.
This is a utility that will allow you to cleanup your M3U/M3U8/M3U+ files based on a CSV.

> Warning: right now, due to it's rapid development, I cannot guarantee breaking changes.
> Warning: right now, due to its rapid development, I cannot guarantee breaking changes.

## How to run
Simply create a config (see example below), and the you can run the binary:
Expand All @@ -22,61 +22,53 @@ Usage of m3ufilter:
```

### Example config
#### Main config
```yaml
core:
server_listen: localhost:8080
update_schedule: "*/24 * * * *"
output: m3u
group_order:
- Entertainment
- Family/Kids
- News
- Drama
providers:
- uri: file:///path/to/m3u/playlist.m3u
filters:
- match(Group, "UK.*") && !match(Name, "^24/7")
- match(Id, "3e.ie")
setters:
- name: replace(Name, "[\\s\\:\\|]+", " ")
- name: replace(Name, "^VIP ", "")
- name: replace(Name, "USA", "")
attributes:
tvg-id: tvg_id(Name) + ".us"
filters:
- Name == "USA CNN"
- Name == "CNN"
- Name == "CNN HD"
csv: file:///path/to/m3u/playlist.csv
epg_providers:
- uri: file:///path/to/epg.xml
channel_id_renames:
replacement: find # key = what to replace it with, value = what to find
bbc.uk: "BBC One"
- from: find
to: replacement
- from: BBC One
to: bbc.uk
```

#### CSV config
```csv
search-name,chno,id,name,group,shift,logo
BBC One UK,1,bbc1.uk,BBC One,Entertainment,,
CNN UK,160,cnn.uk,CNN,News,,
```

#### The meaning of the config options are as follows:

##### Core config
- `core.server_listen` (`string`)

If set, this will run as a server, rather a single run. If you want single runs, you can omit this option. See the arguments to specify the output.
If set, this will run as a server, rather a single run. If you want single runs, you can omit this option.
Default: disabled

- `core.update_schedule` (`string`)

How often it should retrieve the latest channels. This is expressed in [cron syntax](https://github.com/mileusna/crontab#crontab-syntax-). It is highly recommended that you do not set this to a low interval. Set this to at least once every day. You can use the tool [crontab.guru](https://crontab.guru/) to figure out what interval you want.
Default: `* */24 * * *` (that is, once every 24 hours)
Default: `0 */24 * * *` (that is, once every 24 hours)

- `core.auto_reload_config` (`true|false`)

Whether or not to reload the config before every run. Please note, this will not affect `core.server_listen` and `core.update_schedule`
Default: `true`

- `core.output` (`m3u|csv`)

What to output. This can be either `csv` or `m3u`. CSV is useful for debugging and ensuring you've gone through all the channels, outside that, you generally want this to be `m3u`.
Default: `m3u`

- `core.group_order` (`list` of `string`) (experimental)

The order to put the categories in.
Expand All @@ -91,10 +83,11 @@ epg_providers:

The URL of where to retrieve the M3U list. This can start with `file://` to retrieve a list from a local file system (this must be an absolute path).

- `providers.ignore_parse_errors` (`true|false`)
- `providers.csv` (`string`)

If true, this will ignore errors when trying to parse an individual channel. E.g. `tvg-id="Channel "1"` would not be possible to parse, and will be ignored without any errors.
Default: `false`
The URL of where to retrieve the CSV data list. This can start with `file://` to retrieve a list from a local file system (this must be an absolute path). See below for more information on the format.

If this is set, only entries in this file will be in the final CSV.

- `providers.check_streams.enabled` (`true|false`)

Expand All @@ -117,35 +110,6 @@ epg_providers:

Default: `remove`

- `providers.filters` (`list` of `string`)

A list of filters to apply to channels. This must return true or false (no strings or anything else). If it returns true, it will include the channel in the final list.
You can use the functions and variables below to specify your logic.
Default: `true` (meaning all channels are included)

- `providers.setters`

A list of things to set on channels based on the filter.

- `providers.setters.name` (`string`)

Set the name for this individual channel. This MUST return a string.

- `providers.setters.attributes`

What to set any attribute too. This is go for setting logos where none exist, and/or enforcing `tvg-id` in case a channel does not have one but should. All attributes are listed below. This again, must return a string, and has the functions and variables below available.

Example
```yaml
providers:
setters:
tvg-id: mychannel.us
```

- `providers.setters.filters`

A list of filters to limit this providers setter to. The same logic applies as the above filters method, and thus again, must return true/false. If true, it will run the setters.

##### EPG providers

- `epg_providers`
Expand All @@ -160,93 +124,48 @@ epg_providers:

This is a key value pair of channel IDs to rename within the XMLTV. This is useful in case the EPG data's channel IDs don't match the channel IDs in the M3U files. This will change the ID.

##### For filters, name and setters, the following functions are available:

- `strlen(text string) int`

Will return the length of the string

- `match(subject string, regexp string) bool`
#### CSV configuration
In order to manage the m3u, you will need to create a CSV with the relevant values you want to set everything to. The CSV must have the following columns:

Will return `true` if the `subject` matches the regular expression
- `search-name`
- `chno`
- `tvg-id`
- `tvg-name`
- `group-title`
- `tvg-shift`
- `tvg-logo`

- `replace(subject string, find_regexp string, replace string) string`
You can set these up as the first row, in which case, the order of columns does not matter (and you can also remove individual columns to ignore them), however, if you don't have that header row, it must be in the above order.

Will look for the regular express `find_regexp` and replace with the value of `replace` and return that.
What will happen is each M3U stream name (the bit after the `,` on the `#EXTINF` line) will be matched up against the `search-name` column. If no match is found, the stream will not be included in the final playlist. If it is, the stream is included in the final playlist, in addition to the values in the other columns getting set.

- `tvg_id(text string)`
For example, if you have a CSV as follows:

Will try its best to turn text into a valid tvg-id attribute value. This does not include the usual country extension. The idea is that you pass the channel name into this, and it will spit out something that can be used as tvg-id.

For example:
```
tvg_id("CCN HD") > cnn
```

The login behind this will be improved, but right now, all it does is simply remove SD/HD/FHD from the title and any character that isn't `a-zA-Z0-9`.

- `title(subject string) string`

Will turn the text in `subject` into a title, by capitalising all words, and also ensures all letters in SD/HD/FHD are capitalised.

- `upper_words(subject string, word string...) string`

Will turn the text `word` in `subject` into uppercase. Argument `word` can be repeated as multiple times.

- `starts_with(subject string, prefix string...) bool`

Will return true if the text in `subject` starts with the text in `prefix`.

- `endss_with(subject string, suffix string...) bool`

Will return true if the text in `subject` ends with the text in `suffix`.

##### Additionally, the following variables are available:

|variable|content|m3u tag mapping|
|--------|-------|-------|
|`ChNo`|The channel number|`tvg-chno`|
|`Id`|The ID to sync up with XMLTV|`tvg-id`|
|`Name`|This is the channel name|`tvg-name`|
|`Uri`|The URL for the stream|The URL (not a tag)|
|`Duration`|The duration of the stream, this is usually -1 due to Live TV being being.. well.. live.|The duration (not a tag)|
|`Logo`|The logo (can be either a url or a base64 data string)|`tvg-logo`|
|`Language`|The language of the stream|`tvg-language`|
|`Group`|The group category|`group-title`|

##### Generic expression syntax

Within the filters, you may use [these syntax operations](https://github.com/maja42/goval#operators) to filter channels. Many of those are also available within the name and attribute section. As long as they return the expected data type.

#### gotchas
Due to the underlying library used for the logic parsing, setting a value to a generic string is not straight forward and must be double quoted, first with single quote, then double quote.
```csv
search-name,chno,id,name,group,shift,logo
BBC One UK,1,BBC1.uk,BBC One,Entertainment,,
```

For example, if you want to set the for a channel to "My Channel", you have to do is as follows:
```yaml
setters:
- name: '"My Channel"' # this works
filters:
- Name == "some criteria"
- name: "My Channel" # this is invalid
filters:
- Name == "some criteria"
- name: 'My Channel' # this is invalid
filters:
- Name == "some criteria"
- name: My Channel # this is invalid
filters:
- Name == "some criteria"
With the following M3U:
```m3u
#EXTM3U
#EXTINF:-1 tvg-id="" tvg-name="BBC" tvg-logo="https://picon-13398.kxcdn.com/bbcone.jpg" group-title="UK",BBC One UK
http://somewhere.com/111
#EXTINF:-1 tvg-id="Channel4.uk" tvg-name="Channel 4 HD UK" tvg-logo="https://picon-13398.kxcdn.com/channel4hd.jpg" group-title="UK",Channel 4 HD UK
http://somewhere.com/222
```

In theory all of the above should be valid, but until a solution has been thought of, the workaround is to simply prefix it with an equals, e.g.:
```yaml
setters:
- name: = My Channel
filters:
- Name == "some criteria"
The final playlist will be:
```m3u
#EXTM3U
#EXTINF:-1 tvg-id="BBC1" tvg-name="BBC One" tvg-logo="https://picon-13398.kxcdn.com/bbcone.jpg" group-title="Entertainment",BBC One
http://somewhere.com/111
```

Note that prefixing it with an equals marks the whole expression as literal string, excluding the equals. If you want a string with an equals in front of the text, you'll need to use two equals.
Notice how the `search-name` column matches the `BBC One UK` in the source, and in the final, each of the values match the relevant columns in the CSV.
Notice, also, that Channel 4 is not in the final list, as it wasn't in the CSV.

You can generate an initial CSV file using the `-csv` argument. Please note, that this will output additional columns that are otherwise ignored. You can remove them, or leave them.

### Server endpoints
The following server endpoints are available for use:
Expand Down
20 changes: 14 additions & 6 deletions cmd/m3u-filter/main.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ import (
"github.com/hoshsadiq/m3ufilter/config"
"github.com/hoshsadiq/m3ufilter/logger"
"github.com/hoshsadiq/m3ufilter/m3u"
"github.com/hoshsadiq/m3ufilter/net"
"github.com/hoshsadiq/m3ufilter/server"
"github.com/hoshsadiq/m3ufilter/writer"
"github.com/mitchellh/go-homedir"
Expand All @@ -22,6 +23,7 @@ func main() {
configFile := flag.String("config", "~/.m3u.conf", "Config file location")
playlistOutput := flag.String("playlist", "", "Where to output the playlist data. Ignored when using server options in the config. Defaults to stdout")
logOutput := flag.String("log", "", "Where to output logs. Defaults to stderr")
generatingCsv := flag.Bool("csv", false, "Generate CSV instead of processing the M3U")
versionFlag := flag.Bool("version", false, "show version and exit")
flag.Parse()

Expand All @@ -35,10 +37,10 @@ func main() {
}

logger.Setup(appPath)
run(path, fd(*playlistOutput, false), fd(*logOutput, true))
run(path, *generatingCsv, fd(*playlistOutput, false), fd(*logOutput, true))
}

func run(configFilename string, stdout *os.File, stderr *os.File) {
func run(configFilename string, generatingCsv bool, stdout *os.File, stderr *os.File) {
log := logger.Get()
log.SetOutput(stderr)

Expand All @@ -47,12 +49,18 @@ func run(configFilename string, stdout *os.File, stderr *os.File) {
os.Exit(1)
}

m3u.InitClient(conf)
if conf.Core.ServerListen != "" {
net.InitClient(conf)
if generatingCsv {
playlist, _, _ := m3u.ProcessConfig(conf, generatingCsv)
writer.WriteCsv(stdout, playlist)
return
}

if !generatingCsv && conf.Core.ServerListen != "" {
server.Serve(conf)
} else {
playlist, _ /*todo epg*/, _ := m3u.ProcessConfig(conf)
writer.WriteOutput(conf.Core.Output, stdout, playlist)
playlist, _ /*todo epg*/, _ := m3u.ProcessConfig(conf, generatingCsv)
writer.WriteM3U(stdout, playlist)
}
}

Expand Down
31 changes: 4 additions & 27 deletions config.yaml.example
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,36 +1,13 @@
core:
output: m3u
auto_reload_config: true
canonicalise:
enable: true
group_order:
- Entertainment
- Family/Kids
providers:
- uri: file://playlist.m3u
ignore_parse_errors: true
filters:
- |
Group in [
"Documentaries",
"Entertainment",
"Kids",
"Movies",
"Music",
"News",
]

setters:
- name: replace(Name, " +", " ")
- Group: Family/Kids
filters:
- match(Group, "Kids")
- name: replace(Name, "^Sometoon Network", "SN")
filters:
- match(Name, "^Sometoon Network")
- id: =bbc.us
shift: =+3
filters:
- Name == "BBC America" && match(Group, "^US |")
csv: file://playlist.csv
epg_providers:
- uri: file://myepg.xml
channel_id_renames:
- from: BBC One
to: bbc.uk
Loading