New xb.py: Possible schema to standardize xklb db's + main operations? [need to align w/ Calibre-Web's metadata.db & app.db!] #253

deldesir · 2024-09-12T15:11:35Z

This pull request introduces a new module, xb.py, which proposes a standardized schema for the xklb-metadata.db database. The objective of this initial experiment is to:

Establish a structured schema for xklb-metadata.db to standardize data handling and storage.
Implement key database operations within xb.py to facilitate interaction with xklb-metadata.db.
Ensure compatibility and alignment with Calibre-Web's existing databases (metadata.db and app.db).

This update lays the groundwork for future enhancements, aiming to improve data integrity and streamline a better integration of xklb-metadata.db within the Calibre-Web ecosystem. Context for this change can be found in issue #246.

Tested on Ubuntu 24.04

Function to get a specific book (which maps to a media entry) for a given caption

holta · 2024-09-13T06:30:44Z

While I understand that the name xb.py is trying to emulate upstream Calibre-Web's naming of db.py for metadata.db db operations, and ub.py for config/app.db db operations — conversely we might instead want a far more clear name like xklb-db-ops.py (or some such) — so that everything's more crystal clear months/years down the road?

deldesir · 2024-09-19T12:03:21Z

"Official" xklb-metadata.db schema, as derived from an IIAB Calibre-Web example, after running lb tubeadd and lb dl: (in other words, after having downloaded a video... e.g. in fact @deldesir downloaded a small playlist of 5 videos!)

root@box:~# sqlite3 /library/calibre-web/xklb-metadata.db
SQLite version 3.45.1 2024-01-30 16:01:20
Enter ".help" for usage hints.
sqlite> .fullschema
CREATE TABLE IF NOT EXISTS "playlists" (
   [id] INTEGER PRIMARY KEY,
   [time_modified] INTEGER,
   [time_deleted] INTEGER,
   [time_created] INTEGER,
   [hours_update_delay] INTEGER,
   [path] TEXT,
   [extractor_key] TEXT,
   [profile] TEXT,
   [extractor_config] TEXT,
   [extractor_playlist_id] TEXT,
   [title] TEXT,
   [uploader] TEXT
);
CREATE TABLE IF NOT EXISTS 'playlists_fts_data'(id INTEGER PRIMARY KEY, block BLOB);
CREATE TABLE IF NOT EXISTS 'playlists_fts_idx'(segid, term, pgno, PRIMARY KEY(segid, term)) WITHOUT ROWID;
CREATE TABLE IF NOT EXISTS 'playlists_fts_docsize'(id INTEGER PRIMARY KEY, sz BLOB);
CREATE TABLE IF NOT EXISTS 'playlists_fts_config'(k PRIMARY KEY, v) WITHOUT ROWID;
CREATE TABLE IF NOT EXISTS "media" (
   [id] INTEGER PRIMARY KEY,
   [time_deleted] INTEGER,
   [playlists_id] INTEGER,
   [size] INTEGER,
   [duration] INTEGER,
   [time_created] INTEGER,
   [time_modified] INTEGER,
   [time_downloaded] INTEGER,
   [fps] INTEGER,
   [view_count] INTEGER,
   [path] TEXT,
   [webpath] TEXT,
   [extractor_id] TEXT,
   [title] TEXT,
   [uploader] TEXT
, [time_uploaded] INTEGER, [width] INTEGER, [height] INTEGER, [live_status] TEXT, [type] TEXT, [video_codecs] TEXT, [audio_codecs] TEXT, [subtitle_codecs] TEXT, [other_codecs] TEXT, [video_count] INTEGER, [audio_count] INTEGER, [chapter_count] INTEGER, [other_count] INTEGER, [language] TEXT, [subtitle_count] INTEGER, [download_attempts] INTEGER, [error] TEXT);
CREATE TABLE IF NOT EXISTS 'media_fts_data'(id INTEGER PRIMARY KEY, block BLOB);
CREATE TABLE IF NOT EXISTS 'media_fts_idx'(segid, term, pgno, PRIMARY KEY(segid, term)) WITHOUT ROWID;
CREATE TABLE IF NOT EXISTS 'media_fts_docsize'(id INTEGER PRIMARY KEY, sz BLOB);
CREATE TABLE IF NOT EXISTS 'media_fts_config'(k PRIMARY KEY, v) WITHOUT ROWID;
CREATE INDEX [idx_playlists_id]
    ON [playlists] ([id]);
CREATE INDEX [idx_playlists_time_modified]
    ON [playlists] ([time_modified]);
CREATE INDEX [idx_playlists_time_deleted]
    ON [playlists] ([time_deleted]);
CREATE INDEX [idx_playlists_time_created]
    ON [playlists] ([time_created]);
CREATE INDEX [idx_playlists_hours_update_delay]
    ON [playlists] ([hours_update_delay]);
CREATE INDEX [idx_playlists_uploader]
    ON [playlists] ([uploader]);
CREATE INDEX [idx_playlists_extractor_config]
    ON [playlists] ([extractor_config]);
CREATE INDEX [idx_playlists_profile]
    ON [playlists] ([profile]);
CREATE UNIQUE INDEX [idx_playlists_path]
    ON [playlists] ([path]);
CREATE INDEX [idx_playlists_extractor_key]
    ON [playlists] ([extractor_key]);
CREATE INDEX [idx_media_id]
    ON [media] ([id]);
CREATE INDEX [idx_media_time_deleted]
    ON [media] ([time_deleted]);
CREATE INDEX [idx_media_playlists_id]
    ON [media] ([playlists_id]);
CREATE INDEX [idx_media_size]
    ON [media] ([size]);
CREATE INDEX [idx_media_duration]
    ON [media] ([duration]);
CREATE INDEX [idx_media_time_created]
    ON [media] ([time_created]);
CREATE INDEX [idx_media_time_modified]
    ON [media] ([time_modified]);
CREATE INDEX [idx_media_time_downloaded]
    ON [media] ([time_downloaded]);
CREATE INDEX [idx_media_fps]
    ON [media] ([fps]);
CREATE INDEX [idx_media_view_count]
    ON [media] ([view_count]);
CREATE INDEX [idx_media_uploader]
    ON [media] ([uploader]);
CREATE UNIQUE INDEX [idx_media_path]
    ON [media] ([path]);
CREATE VIRTUAL TABLE [playlists_fts] USING FTS5 (
    [path], [title],
    tokenize='trigram',
    content=[playlists]
);
CREATE TRIGGER [playlists_ai] AFTER INSERT ON [playlists] BEGIN
  INSERT INTO [playlists_fts] (rowid, [path], [title]) VALUES (new.rowid, new.[path], new.[title]);
END;
CREATE TRIGGER [playlists_ad] AFTER DELETE ON [playlists] BEGIN
  INSERT INTO [playlists_fts] ([playlists_fts], rowid, [path], [title]) VALUES('delete', old.rowid, old.[path], old.[title]);
END;
CREATE TRIGGER [playlists_au] AFTER UPDATE ON [playlists] BEGIN
  INSERT INTO [playlists_fts] ([playlists_fts], rowid, [path], [title]) VALUES('delete', old.rowid, old.[path], old.[title]);
  INSERT INTO [playlists_fts] (rowid, [path], [title]) VALUES (new.rowid, new.[path], new.[title]);
END;
CREATE VIRTUAL TABLE [media_fts] USING FTS5 (
    [path], [webpath], [title],
    tokenize='trigram',
    content=[media]
);
CREATE TRIGGER [media_ai] AFTER INSERT ON [media] BEGIN
  INSERT INTO [media_fts] (rowid, [path], [webpath], [title]) VALUES (new.rowid, new.[path], new.[webpath], new.[title]);
END;
CREATE TRIGGER [media_ad] AFTER DELETE ON [media] BEGIN
  INSERT INTO [media_fts] ([media_fts], rowid, [path], [webpath], [title]) VALUES('delete', old.rowid, old.[path], old.[webpath], old.[title]);
END;
CREATE TRIGGER [media_au] AFTER UPDATE ON [media] BEGIN
  INSERT INTO [media_fts] ([media_fts], rowid, [path], [webpath], [title]) VALUES('delete', old.rowid, old.[path], old.[webpath], old.[title]);
  INSERT INTO [media_fts] (rowid, [path], [webpath], [title]) VALUES (new.rowid, new.[path], new.[webpath], new.[title]);
END;
CREATE TABLE [captions] (
   [media_id] INTEGER,
   [time] INTEGER,
   [text] TEXT
);
ANALYZE sqlite_schema;
INSERT INTO sqlite_stat1 VALUES('media_fts_docsize',NULL,'5');
INSERT INTO sqlite_stat1 VALUES('media_fts_idx','media_fts_idx','1 1 1');
INSERT INTO sqlite_stat1 VALUES('media_fts_config','media_fts_config','1 1');
INSERT INTO sqlite_stat1 VALUES('playlists','idx_playlists_extractor_key','1 1');
INSERT INTO sqlite_stat1 VALUES('playlists','idx_playlists_path','1 1');
INSERT INTO sqlite_stat1 VALUES('playlists','idx_playlists_profile','1 1');
INSERT INTO sqlite_stat1 VALUES('playlists','idx_playlists_extractor_config','1 1');
INSERT INTO sqlite_stat1 VALUES('playlists','idx_playlists_uploader','1 1');
INSERT INTO sqlite_stat1 VALUES('playlists','idx_playlists_hours_update_delay','1 1');
INSERT INTO sqlite_stat1 VALUES('playlists','idx_playlists_time_created','1 1');
INSERT INTO sqlite_stat1 VALUES('playlists','idx_playlists_time_deleted','1 1');
INSERT INTO sqlite_stat1 VALUES('playlists','idx_playlists_time_modified','1 1');
INSERT INTO sqlite_stat1 VALUES('playlists','idx_playlists_id','1 1');
INSERT INTO sqlite_stat1 VALUES('playlists_fts_data',NULL,'3');
INSERT INTO sqlite_stat1 VALUES('playlists_fts_idx','playlists_fts_idx','1 1 1');
INSERT INTO sqlite_stat1 VALUES('playlists_fts_config','playlists_fts_config','1 1');
INSERT INTO sqlite_stat1 VALUES('media','idx_media_path','5 1');
INSERT INTO sqlite_stat1 VALUES('media','idx_media_uploader','5 3');
INSERT INTO sqlite_stat1 VALUES('media','idx_media_view_count','5 2');
INSERT INTO sqlite_stat1 VALUES('media','idx_media_fps','5 3');
INSERT INTO sqlite_stat1 VALUES('media','idx_media_time_downloaded','5 3');
INSERT INTO sqlite_stat1 VALUES('media','idx_media_time_modified','5 3');
INSERT INTO sqlite_stat1 VALUES('media','idx_media_time_created','5 3');
INSERT INTO sqlite_stat1 VALUES('media','idx_media_duration','5 2');
INSERT INTO sqlite_stat1 VALUES('media','idx_media_size','5 3');
INSERT INTO sqlite_stat1 VALUES('media','idx_media_playlists_id','5 3');
INSERT INTO sqlite_stat1 VALUES('media','idx_media_time_deleted','5 5');
INSERT INTO sqlite_stat1 VALUES('media','idx_media_id','5 1');
INSERT INTO sqlite_stat1 VALUES('playlists_fts_docsize',NULL,'1');
INSERT INTO sqlite_stat1 VALUES('media_fts_data',NULL,'3');
ANALYZE sqlite_schema;

deldesir · 2024-09-19T12:06:37Z

Schema currently proposed by this evolving PR, that would be generated with SQLAlchemy:

root@box:~# sqlite3 /library/calibre-web/xklb.db
SQLite version 3.45.1 2024-01-30 16:01:20
Enter ".help" for usage hints.
sqlite> .fullschema
CREATE TABLE media (
        id INTEGER NOT NULL,
        playlists_id INTEGER,
        size INTEGER,
        duration INTEGER,
        time_created INTEGER,
        time_modified INTEGER,
        time_deleted INTEGER,
        time_downloaded INTEGER,
        fps INTEGER,
        view_count INTEGER,
        path VARCHAR,
        webpath VARCHAR,
        extractor_id VARCHAR,
        title VARCHAR,
        uploader VARCHAR,
        live_status VARCHAR,
        error VARCHAR,
        time_uploaded INTEGER,
        width INTEGER,
        height INTEGER,
        type VARCHAR,
        video_codecs VARCHAR,
        audio_codecs VARCHAR,
        subtitle_codecs VARCHAR,
        video_count INTEGER,
        audio_count INTEGER,
        language VARCHAR,
        subtitle_count INTEGER,
        download_attempts INTEGER,
        PRIMARY KEY (id)
);
CREATE TABLE playlists (
        id INTEGER NOT NULL,
        time_modified INTEGER,
        time_deleted INTEGER,
        time_created INTEGER,
        hours_update_delay INTEGER,
        path VARCHAR,
        extractor_key VARCHAR,
        profile VARCHAR,
        extractor_config VARCHAR,
        extractor_playlist_id VARCHAR,
        title VARCHAR,
        uploader VARCHAR,
        PRIMARY KEY (id)
);
CREATE TABLE captions (
        id INTEGER NOT NULL,
        media_id INTEGER,
        time INTEGER,
        text TEXT,
        PRIMARY KEY (id),
        FOREIGN KEY(media_id) REFERENCES media (id)
);
CREATE TABLE book_media_map (
        book_id INTEGER NOT NULL,
        media_id INTEGER,
        PRIMARY KEY (book_id),
        FOREIGN KEY(media_id) REFERENCES media (id)
);
CREATE TABLE history (
        id INTEGER NOT NULL,
        media_id INTEGER,
        time_played INTEGER,
        done BOOLEAN,
        PRIMARY KEY (id),
        FOREIGN KEY(media_id) REFERENCES media (id)
);
/* No STAT tables available */

holta · 2024-09-19T13:16:58Z

"Official" xklb-metadata.db schema, as derived from an IIAB Calibre-Web example, after running lb tubeadd and lb dl: (in other words, after having downloaded a video!)
root@box:~# sqlite3 /library/calibre-web/xklb-metadata.db
SQLite version 3.45.1 2024-01-30 16:01:20
Enter ".help" for usage hints.
sqlite> .fullschema

Likewise a 333-line actual .dump, also thanks to @deldesir — click below to expand this collapsible section:

root@box:/usr/local/calibre-web-py3/cps# sqlite3 xklb-original.db
SQLite version 3.45.1 2024-01-30 16:01:20
Enter ".help" for usage hints.
sqlite> .dump
/* WARNING: Script requires that SQLITE_DBCONFIG_DEFENSIVE be disabled */
PRAGMA foreign_keys=OFF;
BEGIN TRANSACTION;
CREATE TABLE IF NOT EXISTS "playlists" (
   [id] INTEGER PRIMARY KEY,
   [time_modified] INTEGER,
   [time_deleted] INTEGER,
   [time_created] INTEGER,
   [hours_update_delay] INTEGER,
   [path] TEXT,
   [extractor_key] TEXT,
   [profile] TEXT,
   [extractor_config] TEXT,
   [extractor_playlist_id] TEXT,
   [title] TEXT,
   [uploader] TEXT
);
INSERT INTO playlists VALUES(1,1726745614,0,1726745612,70,'https://www.youtube.com/playlist?list=PLJicmE8fK0Ehc_d78_1WRKT2JXhzFAAB2','ydl_Youtube','video','{"force": true}','PLJicmE8fK0Ehc_d78_1WRKT2JXhzFAAB2','Student Voices from #TEDTalksEd','@TEDEd');
ANALYZE sqlite_schema;
INSERT INTO sqlite_stat1 VALUES('media_fts_docsize',NULL,'5');
INSERT INTO sqlite_stat1 VALUES('media_fts_idx','media_fts_idx','1 1 1');
INSERT INTO sqlite_stat1 VALUES('media_fts_config','media_fts_config','1 1');
INSERT INTO sqlite_stat1 VALUES('playlists','idx_playlists_extractor_key','1 1');
INSERT INTO sqlite_stat1 VALUES('playlists','idx_playlists_path','1 1');
INSERT INTO sqlite_stat1 VALUES('playlists','idx_playlists_profile','1 1');
INSERT INTO sqlite_stat1 VALUES('playlists','idx_playlists_extractor_config','1 1');
INSERT INTO sqlite_stat1 VALUES('playlists','idx_playlists_uploader','1 1');
INSERT INTO sqlite_stat1 VALUES('playlists','idx_playlists_hours_update_delay','1 1');
INSERT INTO sqlite_stat1 VALUES('playlists','idx_playlists_time_created','1 1');
INSERT INTO sqlite_stat1 VALUES('playlists','idx_playlists_time_deleted','1 1');
INSERT INTO sqlite_stat1 VALUES('playlists','idx_playlists_time_modified','1 1');
INSERT INTO sqlite_stat1 VALUES('playlists','idx_playlists_id','1 1');
INSERT INTO sqlite_stat1 VALUES('playlists_fts_data',NULL,'3');
INSERT INTO sqlite_stat1 VALUES('playlists_fts_idx','playlists_fts_idx','1 1 1');
INSERT INTO sqlite_stat1 VALUES('playlists_fts_config','playlists_fts_config','1 1');
INSERT INTO sqlite_stat1 VALUES('media','idx_media_path','5 1');
INSERT INTO sqlite_stat1 VALUES('media','idx_media_uploader','5 3');
INSERT INTO sqlite_stat1 VALUES('media','idx_media_view_count','5 2');
INSERT INTO sqlite_stat1 VALUES('media','idx_media_fps','5 3');
INSERT INTO sqlite_stat1 VALUES('media','idx_media_time_downloaded','5 3');
INSERT INTO sqlite_stat1 VALUES('media','idx_media_time_modified','5 3');
INSERT INTO sqlite_stat1 VALUES('media','idx_media_time_created','5 3');
INSERT INTO sqlite_stat1 VALUES('media','idx_media_duration','5 2');
INSERT INTO sqlite_stat1 VALUES('media','idx_media_size','5 3');
INSERT INTO sqlite_stat1 VALUES('media','idx_media_playlists_id','5 3');
INSERT INTO sqlite_stat1 VALUES('media','idx_media_time_deleted','5 5');
INSERT INTO sqlite_stat1 VALUES('media','idx_media_id','5 1');
INSERT INTO sqlite_stat1 VALUES('playlists_fts_docsize',NULL,'1');
INSERT INTO sqlite_stat1 VALUES('media_fts_data',NULL,'3');
CREATE TABLE IF NOT EXISTS 'playlists_fts_data'(id INTEGER PRIMARY KEY, block BLOB);
INSERT INTO playlists_fts_data VALUES(1,X'01461d');
INSERT INTO playlists_fts_data VALUES(10,X'000000000101010001010101');
INSERT INTO playlists_fts_data VALUES(137438953473,X'00000305043020237401060101150202667201060101100202766f01060101090103237465010601011601032e636f0102150202796f01020d01032f2f770102080202706c010219020277770102090103306568010232010331777201023b0103326a78010240010337385f0102380103385f310102390202666b01022f01033a2f2f01020701033d706c01022701033f6c6901022201035f317701023a0202643701023601036161620102460202623201024702026c6b010601011b0202796c01021c010362652e0102130103635f6401023502026573010601010d02026d6501022c02026f6d01021601036437380102370202656e01060101050202746101060101190103652e630102140202386601022e0202647401060101180202686301023302026e74010601010602027320010601010e010366616101024502026b300102300202726f0106010111010368635f0102340202747401020202027a660102430103696365010601010c03016d01022b0202737401041f0701036a696301022a0202786801024101036b306501023102027365010601011d0202743201023e01036c617901021b0202697301041e0702026a6901022902026b73010601011c01036d2023010601011402022f700102180202653801022d01036e7420010601010701036f6963010601010b02026d20010601011303012f0102170202757401020f0103706c6101021a03016a0102280202733a0102050103726b7401023d02026f6d01060101120103732066010601010f02023a2f01020602026564010601011e0202743d01022503013f0102200301750106010102010374207601060101080202326a01023f02023d7001022602023f6c0102210202616c010601011a02026564010601011702027073010204020274700102030202756201021103016401060101030103756265010212020264650106010104020274750102100103766f69010601010a0103772e7901020c0202726b01023c0202772e01020b03017701020a010378687a0102420103796c6901021d02026f7501020e01037a6661010244040a09090a080708070708080808080708080808070807090708080907070809090807090709090807090807070a06080807080907080807090a07070a0a09060708060708090a07090706080a0707070909070707080809070a08070706080807');
CREATE TABLE IF NOT EXISTS 'playlists_fts_idx'(segid, term, pgno, PRIMARY KEY(segid, term)) WITHOUT ROWID;
INSERT INTO playlists_fts_idx VALUES(1,X'',2);
CREATE TABLE IF NOT EXISTS 'playlists_fts_docsize'(id INTEGER PRIMARY KEY, sz BLOB);
INSERT INTO playlists_fts_docsize VALUES(1,X'461d');
CREATE TABLE IF NOT EXISTS 'playlists_fts_config'(k PRIMARY KEY, v) WITHOUT ROWID;
INSERT INTO playlists_fts_config VALUES('version',4);
CREATE TABLE IF NOT EXISTS "media" (
   [id] INTEGER PRIMARY KEY,
   [time_deleted] INTEGER,
   [playlists_id] INTEGER,
   [size] INTEGER,
   [duration] INTEGER,
   [time_created] INTEGER,
   [time_modified] INTEGER,
   [time_downloaded] INTEGER,
   [fps] INTEGER,
   [view_count] INTEGER,
   [path] TEXT,
   [webpath] TEXT,
   [extractor_id] TEXT,
   [title] TEXT,
   [uploader] TEXT
, [time_uploaded] INTEGER, [width] INTEGER, [height] INTEGER, [live_status] TEXT, [type] TEXT, [video_codecs] TEXT, [audio_codecs] TEXT, [subtitle_codecs] TEXT, [other_codecs] TEXT, [video_count] INTEGER, [audio_count] INTEGER, [chapter_count] INTEGER, [other_count] INTEGER, [language] TEXT, [subtitle_count] INTEGER, [download_attempts] INTEGER, [error] TEXT);
INSERT INTO media VALUES(1,0,NULL,0,0,1726746112,1726746112,0,0,NULL,'https://www.youtube.com/watch?v=wH2-CJapQzs','https://www.youtube.com/watch?v=wH2-CJapQzs','1',NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,1,'Unable to download video thumbnail 21: HTTPSConnectionPool(host=''i.ytimg.com'';port=443): Read timed out. (read timeout=20.0)');
INSERT INTO media VALUES(2,0,1,26999563,136,1726746018,1726746023,1726745997,29.9700299700299694,39693,'/usr/local/calibre-web-py3/cps/Youtube/TED-Ed/Meet Julia Delmedico_39.69k_[br1TOrdoli4].mp4','https://www.youtube.com/watch?v=br1TOrdoli4','br1TOrdoli4','Meet Julia Delmedico','UCsooa4yRKGN_zEE8iknghZA',1368025321,1920,1080,'not_live','video/mp4','h264','aac','mov_text','bin_data',1,1,8,1,'eng',1,1,NULL);
INSERT INTO media VALUES(3,0,1,33546869,153,1726746044,1726746049,1726745997,29.9700299700299694,31948,'/usr/local/calibre-web-py3/cps/Youtube/TED-Ed/Meet Melissa Perez_31.95k_[19HB_AdAnCY].mp4','https://www.youtube.com/watch?v=19HB_AdAnCY','19HB_AdAnCY','Meet Melissa Perez','UCsooa4yRKGN_zEE8iknghZA',1368025334,1920,1080,'not_live','video/mp4','h264','aac','mov_text',NULL,1,1,NULL,NULL,'eng',1,1,NULL);
INSERT INTO media VALUES(4,0,1,17449030,96,1726746064,1726746072,1726745997,29.9700299700299694,25690,'/usr/local/calibre-web-py3/cps/Youtube/TED-Ed/Meet Shayna Cody_25.69k_[o2HcKp1WN3c].mp4','https://www.youtube.com/watch?v=o2HcKp1WN3c','o2HcKp1WN3c','Meet Shayna Cody','UCsooa4yRKGN_zEE8iknghZA',1368025361,1920,1080,'not_live','video/mp4','h264','aac','mov_text',NULL,1,1,NULL,NULL,'eng',1,1,NULL);
INSERT INTO media VALUES(5,1726745997,NULL,0,0,1726746120,1726746120,0,0,NULL,'https://www.youtube.com/watch?v=mChts3z6h4o','https://www.youtube.com/watch?v=mChts3z6h4o','5',NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,1,'ERROR: [youtube] mChts3z6h4o: No video formats found!;ERROR: [youtube] mChts3z6h4o: No video formats found!;[youtube] Private video. Sign in if you''ve been granted access to this video;No video formats found!;[info] Video Thumbnail 37 does not exist;[info] Video Thumbnail 36 does not exist;[info] Video Thumbnail 35 does not exist;[info] Video Thumbnail 34 does not exist;[info] Video Thumbnail 33 does not exist;[info] Video Thumbnail 32 does not exist;[info] Video Thumbnail 31 does not exist;[info] Video Thumbnail 30 does not exist;[info] Video Thumbnail 29 does not exist;[info] Video Thumbnail 28 does not exist;[info] Video Thumbnail 27 does not exist;[info] Video Thumbnail 26 does not exist;[info] Video Thumbnail 25 does not exist;[info] Video Thumbnail 24 does not exist;[info] Video Thumbnail 23 does not exist;[info] Video Thumbnail 22 does not exist;[info] Video Thumbnail 21 does not exist;[info] Video Thumbnail 20 does not exist;[info] Video Thumbnail 19 does not exist;[info] Video Thumbnail 18 does not exist;[info] Video Thumbnail 17 does not exist;[info] Video Thumbnail 16 does not exist;[info] Video Thumbnail 15 does not exist;[info] Video Thumbnail 14 does not exist;[info] Video Thumbnail 13 does not exist;[info] Video Thumbnail 12 does not exist;[info] Video Thumbnail 11 does not exist;[info] Video Thumbnail 10 does not exist;[info] Video Thumbnail 9 does not exist;[info] Video Thumbnail 8 does not exist;[info] Video Thumbnail 7 does not exist;[info] Video Thumbnail 6 does not exist;[info] Video Thumbnail 5 does not exist;[info] Video Thumbnail 4 does not exist;[info] Video Thumbnail 3 does not exist;[info] Video Thumbnail 2 does not exist;[info] Video Thumbnail 1 does not exist;[info] Video Thumbnail 0 does not exist');
CREATE TABLE IF NOT EXISTS 'media_fts_data'(id INTEGER PRIMARY KEY, block BLOB);
INSERT INTO media_fts_data VALUES(1,X'058257814d30');
INSERT INTO media_fts_data VALUES(10,X'000000000106060006010101020101030101040101050101060101');
INSERT INTO media_fts_data VALUES(137438953473,X'000007d0043020636f040601020d02026465020601020c02026a75020601020602026d65030601020602027065030601020e02027368040601020601032d636a01082501012501032e636f0108150101150108150101150108150101150108150101150108150101150202796f01080d01010d01080d01010d01080d01010d01080d01010d01080d01010d01032f2f770108080101080108080101080108080101080108080101080108080101080202776101081901011901081901011901081901011901081901011901081901011903017701080901010901080901010901080901010901080901010901080901010901033139680308220101220202746f0208240101240202776e0408280101280103322d63010824010124020268630408230101230103337a360508270101270103366834050829010129010339686203082301012301033a2f2f01080701010701080701010701080701010701080701010701080701010701033d31390308210101210202627202082101012102026d6305082101012102026f320408210101210202776801082101012101033f763d01081f01011f01081f01011f01081f01011f01081f01011f01081f01011f01035f61640308260101260103612063040601020c030164020601020b030170030601020d0202646103082701012702026e63030829010129020270710108280101280202746301081b01011b01081b01011b01081b01011b01081b01011b01081b01011b0202796e04060102090103625f610308250101250202652e01081301011301081301011301081301011301081301011301081301011302027231020822010122010363683f01081d01011d01081d01011d01081d01011d01081d01011d01081d01011d03017405082301012302026a6101082601012602026b7004082501012502026f64040601020e03016d010816010116010816010116010816010116010816010116010816010116010364616e0308280101280202656c020601020d02026963020601021202026f6c0208280101280103652e630108140101140108140101140108140101140108140101140108140101140202646902060102110202657402060102030106010203010601020302026c69030601020803016d020601020e02027265030601021002027420020601020401060102040106010204010368322d0108230101230202346f05082a01012a02023f7601081e01011e01081e01011e01081e01011e01081e01011e01081e01011e0202617904060102080202625f0308240101240202636b040824010124020274730508240101240301740108020101020108020101020108020101020108020101020108020101020103696120020601020a0202636f020601021302027373030601020a01036a61700108270101270202756c020601020701036b703104082601012601036c693402082a01012a0301610206010209030173030601020902026d65020601020f01036d2f770108180101180108180101180108180101180108180101180108180101180202636805082201012202026564020601021003016502060102020106010202010601020203016c030601020701036e336304082a01012a02026120040601020b0202637903082a01012a01036f326804082201012202026479040601020f02026c6902082901012902026d2f010817010117010817010117010817010117010817010117010817010117020272640208260101260202757401080f01010f01080f01010f01080f01010f01080f01010f01080f01010f010370317704082701012702026572030601020f0202717a0108290101290202733a0108050101050108050101050108050101050108050101050108050101050103717a7301082a01012a01037231740208230101230202646f0208270101270202657a0306010211010373337a05082601012602023a2f01080601010601080601010601080601010601080601010601080601010602026120030601020c02026861040601020702027361030601020b010374206a020601020503016d030601020503017304060102050202636801081c01011c01081c01011c01081c01011c01081c01011c01081c01011c02026f7202082501012502027073010804010104010804010104010804010104010804010104010804010104020273330508250101250202747001080301010301080301010301080301010301080301010301080301010302027562010811010111010811010111010811010111010811010111010811010111010375626501081201011201081201011201081201011201081201011201081201011202026c690206010208020274750108100101100108100101100108100101100108100101100108100101100103763d3103082001012003016202082001012003016d05082001012003016f0408200101200301770108200101200103772e7901080c01010c01080c01010c01080c01010c01080c01010c01080c01010c0202617401081a01011a01081a01011a01081a01011a01081a01011a01081a01011a0202683201082201012202026e330408290101290202772e01080b01010b01080b01010b01080b01010b01080b01010b01080b01010b03017701080a01010a01080a01010a01080a01010a01080a01010a01080a01010a0103796e61040601020a02026f7501080e01010e01080e01010e01080e01010e01080e01010e01080e01010e01037a3668050828010128040a09090909090b23222322210b0a0a0b0a0b0b0b230b0a0a0a0a230b0a08080a0a0a22090b220a23090a0a09210b09090a230913090809130b0a22090a0a0a210a09090b090b0b080809230a0912080b090a0b090a220a220b090a220b0b0a090b220909090a0808220a220a22222309220b0909090923220a0a22210a22');
INSERT INTO media_fts_data VALUES(274877906945,X'000003d6043020646502093a01020c02026a7502093401020601032d656402022c020270790202180202776502021401032e36390202470202636f020701011502026d700202590202796f020701010d01032f2f7702070101080202636102020c03017002021c02026c6f02020602026d6502022f020274650202280202757302020202027761020701011903017702070101090202796f020220010331746f02094f0101240103332f6302021b0202392e0202450103345d2e020257010336396b0202480103392e3602024602026b5f02024901033a2f2f020701010701033d6272020701012101033f763d020701011f01035b627202024c01035d2e6d02025801035f333902024402025b6202024b010361206402093901020b02026c2f02020a03016902020e02027463020701011b0103622d700202170202652e020701011303012f0202260202723102094d010122030165020211010363616c020409060202683f020701011d02026f5f02024203016d02070101160202707302021d0103642d6502022b02022f6d02022e0202656c02093b01020d0202696302094001021202026f6c0209530101280103652d7702021302022e63020701011402022f740202270202622d0202160202642d02022a03012f02022d03016902093f0102110202657402093101020302026c6d02093c01020e020274200209320102040103683f76020701011e020274740207010102010369345d0202560202612002093801020a020262720202100202636f02094101021301036a756c02093501020701036b5f5b02024a01036c2f6302020b0202693402095501012a03016102093701020903016202020f02026d6502093d01020f02026f6302020701036d2f7702070101180202656402093e0102100301650209300102020202703402025a01036f5f330202430202636102020802026c6902095401012902026d2f0207010117020272640209510101260202757402092201010f010370732f02021e03013a0207010105020279330202190103722f6c0202050202317402094e0101230202646f0209520101270202652d0202120103732f7902021f02023a2f02070101060202722f020204010374206a02093301020502026368020701011c0202656402022902026f7202095001012502027073020701010402027470020701010302027562020924010111010375626502092501011202026c6902093601020802027372020203020274750209230101100103763d6202070101200103772e79020701010c02026174020701011a020265620202150202772e020701010b030177020701010a010379332f02021a02026f7502092101010e040b0a080707080907090a0706070707070908070b0807080808070a0a0a080808070b0706090809060a06090907080708070a0a0a080907070706090a0a0a0a09080a070a0b08080a09060a070a0a090708070a090a0a080807080a0a070809070b09070a09090a0b0a070a0a0a0907090808');
INSERT INTO media_fts_data VALUES(412316860417,X'000003c50430206d650309340102060202706503093c01020e01032d656403022c020270790302180202776503021401032e39350302450202636f030701011502026d700302570202796f030701010d01032f2f7703070101080202636103020c03017003021c02026c6f03020602026d6503022f020274650302280202757303020202027761030701011903017703070101090202796f0302200103312e390302440202396803094b0101220103332f6303021b0202312e0302430103356b5f030247010339356b0302460202686203094c01012301033a2f2f030701010701033d3139030701012101033f763d030701011f01035b313903024a01035d2e6d03025601035f333103024202025b310302490202616403094f010126010361207003093b01020d0202646103095001012702026c2f03020a03016903020e02026e6303095201012902027463030701011b0103622d7003021702025f6103094e0101250202652e030701011303012f03022602027265030211010363616c030409060202683f030701011d02026f6d03070101160202707303021d0202795d0302540103642d6503022b02022f6d03022e0202616e0309510101280103652d7703021302022e63030701011402022f740302270202622d0302160202642d03022a03012f03022d0202657403093101020302026c690309360102080202726503093e0102100202742003093201020402027a5f0302400103683f76030701011e0202625f03094d01012402027474030701010201036962720302100202737303093801020a01036b5f5b03024801036c2f6303020b0202696203020f03017303093701020902026f6303020701036d2f7703070101180202656503093001020203016c0309350102070202703403025801036e637903095301012a01036f636103020802026d2f03070101170202757403092201010f010370657203093d01020f0202732f03021e03013a0307010105020279330302190103722f6c0302050202652d03021203017a03093f0102110103732f7903021f02023a2f03070101060202612003093a01020c0202722f0302040202736103093901020b010374206d03093301020502026368030701011c0202656403022902027073030701010402027470030701010302027562030924010111010375626503092501011202027372030203020274750309230101100103763d3103070101200103772e79030701010c02026174030701011a020265620302150202772e030701010b030177030701010a010379332f03021a02025d2e03025502026f7503092101010e01037a5f33030241040b0a080707080907090a070607070707090807080a080708080a0a0a0a080808070a0b0a07060a09080a090607090909070708070a0809070707060a0a0a0a070a0a09080a08080709070a0a09070b08090a0b07080708070908090a070a0b090709090a0b070a0a0a0907090808070a');
INSERT INTO media_fts_data VALUES(549755813889,X'000003b4043020636f04093b01020d0202736804093401020601032d656404022c020270790402180202776504021401032e36390402430202636f040701011502026d700402550202796f040701010d01032f2f7704070101080202636104020c03017004021c02026c6f04020602026d6504022f020274650402280202757304020202027761040701011903017704070101090202796f040220010331776e04094f010128010332352e0402410202686304094a0101230103332f6304021b0202635d0402520103352e36040242010336396b0402440103396b5f04024501033a2f2f040701010701033d6f32040701012101033f763d040701011f01035b6f3204024801035d2e6d04025401035f323504024002025b6f040247010361206304093a01020c02026c2f04020a03016904020e02027463040701011b0202796e0409370102090103622d700402170202652e040701011303012f040226020272650402110103635d2e0402530202616c040409060202683f040701011d02026b7004094c01012502026f6404093c01020e03016d04070101160202707304021d0103642d6504022b02022f6d04022e0202795f04023e0103652d7704021302022e63040701011402022f740402270202622d0402160202642d04022a03012f04022d02026574040931010203020274200409320102040103683f76040701011e020261790409360102080202636b04094b010124020274740407010102010369627204021001036b5f5b0402460202703104094d01012601036c2f6304020b0202696204020f02026f6304020701036d2f770407010118020265650409300102020202703404025601036e336304095101012a0202612004093901020b01036f3268040949010122020263610402080202647904093d01020f02026d2f04070101170202757404092201010f010370317704094e0101270202732f04021e03013a0407010105020279330402190103722f6c0402050202652d0402120103732f7904021f02023a2f0407010106020268610409350102070202722f040204010374207304093301020502026368040701011c0202656404022902027073040701010402027470040701010302027562040924010111010375626504092501011202027372040203020274750409230101100103763d6f04070101200103772e79040701010c02026174040701011a0202656204021502026e330409500101290202772e040701010b030177040701010a010379332f04021a02025f3204023f02026e6104093801020a02026f7504092101010e040b0a080707080907090a0706070707070908070b080a08070808080a0a0a080808070b0706090a080906070808090a0a08070807070809070707060a0a0a0a0a0908080a0807070a0a070b0a0b070a090a0b070807080708090a070b090709090a0b070a0a0a09070a090808070a');
INSERT INTO media_fts_data VALUES(687194767361,X'000001b304302d636a01092501012501032e636f0109150101150202796f01090d01010d01032f2f77010908010108020277610109190101190301770109090101090103322d6301092401012401033a2f2f01090701010701033d776801092101012101033f763d01091f01011f01036170710109280101280202746301091b01011b010362652e010913010113010363683f01091d01011d02026a6101092601012602026f6d0109160101160103652e63010914010114010368322d01092301012302023f7601091e01011e0202747401090201010201036a617001092701012701036d2f7701091801011801036f6d2f0109170101170202757401090f01010f010370717a0109290101290202733a0109050101050103717a7301092a01012a0103733a2f010906010106010374636801091c01011c0202707301090401010402027470010903010103020275620109110101110103756265010912010112020274750109100101100103763d770109200101200103772e7901090c01010c0202617401091a01011a020268320109220101220202772e01090b01010b03017701090a01010a0103796f7501090e01010e040b0b0a0b0a090b0b0b0b0b0a0b0b0a0a0b0b0a0a0b0b0b0a0b0a0b0b0b0a0a0a0b0a0b0b0a0a0a09');
INSERT INTO media_fts_data VALUES(824633720833,X'000001b004302e636f0509150101150202796f05090d01010d01032f2f77050908010108020277610509190101190301770509090101090103337a36050927010127010336683405092901012901033a2f2f05090701010701033d6d6305092101012101033f763d05091f01011f010361746305091b01011b010362652e050913010113010363683f05091d01011d03017405092301012302026f6d0509160101160103652e63050914010114010368346f05092a01012a02023f7605091e01011e0202747305092401012403017405090201010201036d2f770509180101180202636805092201012201036f6d2f0509170101170202757405090f01010f010370733a050905010105010373337a05092601012602023a2f050906010106010374636805091c01011c020270730509040101040202733305092501012502027470050903010103020275620509110101110103756265050912010112020274750509100101100103763d6d0509200101200103772e7905090c01010c0202617405091a01011a0202772e05090b01010b03017705090a01010a0103796f7505090e01010e01037a3668050928010128040b0a0b0a090b0b0b0b0b0b0b0b090a0b0b0a0a090b0a0b0a0b0b0a0b0a0a0a0a0b0a0b0b0a0a090b');
CREATE TABLE IF NOT EXISTS 'media_fts_idx'(segid, term, pgno, PRIMARY KEY(segid, term)) WITHOUT ROWID;
INSERT INTO media_fts_idx VALUES(1,X'',2);
INSERT INTO media_fts_idx VALUES(2,X'',2);
INSERT INTO media_fts_idx VALUES(3,X'',2);
INSERT INTO media_fts_idx VALUES(4,X'',2);
INSERT INTO media_fts_idx VALUES(5,X'',2);
INSERT INTO media_fts_idx VALUES(6,X'',2);
CREATE TABLE IF NOT EXISTS 'media_fts_docsize'(id INTEGER PRIMARY KEY, sz BLOB);
INSERT INTO media_fts_docsize VALUES(1,X'292900');
INSERT INTO media_fts_docsize VALUES(2,X'592912');
INSERT INTO media_fts_docsize VALUES(3,X'572910');
INSERT INTO media_fts_docsize VALUES(4,X'55290e');
INSERT INTO media_fts_docsize VALUES(5,X'292900');
CREATE TABLE IF NOT EXISTS 'media_fts_config'(k PRIMARY KEY, v) WITHOUT ROWID;
INSERT INTO media_fts_config VALUES('version',4);
PRAGMA writable_schema=ON;
INSERT INTO sqlite_schema(type,name,tbl_name,rootpage,sql)VALUES('table','playlists_fts','playlists_fts',0,'CREATE VIRTUAL TABLE [playlists_fts] USING FTS5 (
    [path], [title],
    tokenize=''trigram'',
    content=[playlists]
)');
INSERT INTO sqlite_schema(type,name,tbl_name,rootpage,sql)VALUES('table','media_fts','media_fts',0,'CREATE VIRTUAL TABLE [media_fts] USING FTS5 (
    [path], [webpath], [title],
    tokenize=''trigram'',
    content=[media]
)');
CREATE TABLE [captions] (
   [media_id] INTEGER,
   [time] INTEGER,
   [text] TEXT
);
INSERT INTO captions VALUES(2,0,'language:en;This is one of five student perspectives featured in the PBS special "TED Talks Education" (#TEDTalksed.) Learn more here: https://www.ted.com/promos/TEDTalksEducation Special thanks to our partners in this endeavor: WNET;PBS and the Corporation for Public Broadcasting.;Education;TEDEducation;\TED;PBS\;\Julia;Delmedico\');
INSERT INTO captions VALUES(2,0,'Intro');
INSERT INTO captions VALUES(2,15,'What do you think about public schools');
INSERT INTO captions VALUES(2,30,'What do you think about students');
INSERT INTO captions VALUES(2,42,'What do you hear in the hallways');
INSERT INTO captions VALUES(2,57,'What classes do you take');
INSERT INTO captions VALUES(2,85,'What is your favorite subject');
INSERT INTO captions VALUES(2,105,'What is the best way to learn');
INSERT INTO captions VALUES(2,118,'Outro');
INSERT INTO captions VALUES(2,0,'Transcriber: Andrea McDonough Reviewer: Jessica Ruby');
INSERT INTO captions VALUES(2,6,'I''m Julia, and I''m 17.');
INSERT INTO captions VALUES(2,8,'My parents are from Argentina,');
INSERT INTO captions VALUES(2,10,'and I''ve grown up in Manhattan my whole life');
INSERT INTO captions VALUES(2,13,'and go to school in the Bronx.');
INSERT INTO captions VALUES(2,15,'I think a lot of high schools');
INSERT INTO captions VALUES(2,17,'are trying to get their students');
INSERT INTO captions VALUES(2,19,'into the best colleges.');
INSERT INTO captions VALUES(2,21,'And because the way students are being measured');
INSERT INTO captions VALUES(2,23,'is through numbers that are so harsh,');
INSERT INTO captions VALUES(2,25,'I think public schools are killing creativity.');
INSERT INTO captions VALUES(2,28,'They''re quantifying students.');
INSERT INTO captions VALUES(2,31,'Students are only given an incentive');
INSERT INTO captions VALUES(2,33,'to learn for a good grade,');
INSERT INTO captions VALUES(2,35,'but nothing more.');
INSERT INTO captions VALUES(2,37,'I probably just skim the surface');
INSERT INTO captions VALUES(2,38,'just to pass exams a lot of the time.');
INSERT INTO captions VALUES(2,42,'''Cause what you hear mostly in the hallways');
INSERT INTO captions VALUES(2,44,'are what projects are due,');
INSERT INTO captions VALUES(2,45,'what grades you got,');
INSERT INTO captions VALUES(2,46,'how you''re going to fit it all in your day.');
INSERT INTO captions VALUES(2,48,'And, in class, half the students have their head down sleeping');
INSERT INTO captions VALUES(2,51,'because they''ve been up all night');
INSERT INTO captions VALUES(2,53,'trying to study for their exams.');
INSERT INTO captions VALUES(2,58,'The only support we really get');
INSERT INTO captions VALUES(2,59,'is from our guidance counselors');
INSERT INTO captions VALUES(2,61,'about which college to apply to,');
INSERT INTO captions VALUES(2,62,'what classes to take.');
INSERT INTO captions VALUES(2,64,'"You need an AP science if you want to go to nahdahdah."');
INSERT INTO captions VALUES(2,67,'But that''s not me.');
INSERT INTO captions VALUES(2,71,'I''m very, very much engaged in art history');
INSERT INTO captions VALUES(2,75,'and ''mic/mac'', which is micro/macroeconomics.');
INSERT INTO captions VALUES(2,78,'Both the teachers are extremely intelligent,');
INSERT INTO captions VALUES(2,80,'and they like to talk about what they know about the subject.');
INSERT INTO captions VALUES(2,86,'My ''mic/mac'' teacher, he sits at his desk,');
INSERT INTO captions VALUES(2,88,'and he gives not only all his opinions,');
INSERT INTO captions VALUES(2,90,'but all the videos he''s watched,');
INSERT INTO captions VALUES(2,92,'all the articles he''s read,');
INSERT INTO captions VALUES(2,93,'everything that''s going on right now');
INSERT INTO captions VALUES(2,95,'that''s somehow related,');
INSERT INTO captions VALUES(2,96,'and pulls everything out of the air');
INSERT INTO captions VALUES(2,97,'and brings it all in to the topic that we''re talking about,');
INSERT INTO captions VALUES(2,100,'which will not be tested on');
INSERT INTO captions VALUES(2,102,'but makes the topic something real.');
INSERT INTO captions VALUES(2,105,'I never imagined myself to enjoy economics.');
INSERT INTO captions VALUES(2,108,'The best way I learn is in an actual, hands-on experience,');
INSERT INTO captions VALUES(2,113,'something that can bring me out of the classroom.');
INSERT INTO captions VALUES(2,119,'I think the best kind of education');
INSERT INTO captions VALUES(2,122,'is one that teaches you how to speak');
INSERT INTO captions VALUES(2,123,'and think for yourself.');
INSERT INTO captions VALUES(2,126,'That''s much more valuable');
INSERT INTO captions VALUES(2,127,'than just passing your exams.');
INSERT INTO captions VALUES(3,0,'language:en;This is one of five student perspectives featured in the PBS special "TED Talks Education" (#TEDTalksed.) Learn more here: https://www.ted.com/promos/TEDTalksEducation Special thanks to our partners in this endeavor: WNET;PBS and the Corporation for Public Broadcasting.;Education;TED \TED PBS\;TED-Ed;\TED;Ed\;TEDEducation;#TEDTalksEd;\Melissa;Perez\');
INSERT INTO captions VALUES(3,0,'Transcriber: Andrea McDonough Reviewer: Jessica Ruby');
INSERT INTO captions VALUES(3,7,'My name is Melissa Perez.');
INSERT INTO captions VALUES(3,9,'I was born and raised in the Bronx, New York City,');
INSERT INTO captions VALUES(3,12,'but my parents are Mexican.');
INSERT INTO captions VALUES(3,16,'Two years ago, graduating high school');
INSERT INTO captions VALUES(3,18,'was not even in my plans.');
INSERT INTO captions VALUES(3,21,'I was doing horribly in school.');
INSERT INTO captions VALUES(3,26,'Going to a school that has metal detectors is stressful.');
INSERT INTO captions VALUES(3,31,'You have to stand in line 25 minutes to an hour.');
INSERT INTO captions VALUES(3,37,'I would only go in third period');
INSERT INTO captions VALUES(3,40,'and come out fourth.');
INSERT INTO captions VALUES(3,41,'I went third period');
INSERT INTO captions VALUES(3,43,'because I knew they didn''t take attendance first and second.');
INSERT INTO captions VALUES(3,46,'So, basically, if you were going third period,');
INSERT INTO captions VALUES(3,48,'they would mark you present for the whole day.');
INSERT INTO captions VALUES(3,50,'I just wasn''t interested in school.');
INSERT INTO captions VALUES(3,54,'Then, when I was 17,');
INSERT INTO captions VALUES(3,56,'I gave birth to my daughter, Madeline.');
INSERT INTO captions VALUES(3,59,'Getting pregnant changed my approach tremendously.');
INSERT INTO captions VALUES(3,64,'I thought, "I''m pregnant, I don''t have a diploma.');
INSERT INTO captions VALUES(3,67,'So what I am I going to be working the rest of my life?');
INSERT INTO captions VALUES(3,69,'$7.25 an hour?');
INSERT INTO captions VALUES(3,70,'No! I have to graduate and go to college."');
INSERT INTO captions VALUES(3,76,'Being a mom, still going to high school is, ummm,');
INSERT INTO captions VALUES(3,80,'overwhelming, I would say, at the beginning,');
INSERT INTO captions VALUES(3,84,'but I had this math teacher who encouraged me');
INSERT INTO captions VALUES(3,87,'to go on with school.');
INSERT INTO captions VALUES(3,90,'She always said that she saw something in me.');
INSERT INTO captions VALUES(3,93,'She was like, "I know there''s something inside you there');
INSERT INTO captions VALUES(3,96,'that wants to fight for it."');
INSERT INTO captions VALUES(3,98,'She always pushes me,');
INSERT INTO captions VALUES(3,99,'she always gives me the hard questions');
INSERT INTO captions VALUES(3,102,'because she knows that my favorite subject is math.');
INSERT INTO captions VALUES(3,106,'She''s always like, "Melissa, figure this out.');
INSERT INTO captions VALUES(3,107,'Melissa, figure that out."');
INSERT INTO captions VALUES(3,108,'And I''m like, "Okay, okay."');
INSERT INTO captions VALUES(3,109,'I like that about her');
INSERT INTO captions VALUES(3,110,'because she always tends to push me.');
INSERT INTO captions VALUES(3,114,'Thelma is the first teacher');
INSERT INTO captions VALUES(3,115,'that has treated me,');
INSERT INTO captions VALUES(3,118,'you know,');
INSERT INTO captions VALUES(3,119,'I guess, special.');
INSERT INTO captions VALUES(3,122,'After I had my baby, I got my head into the game');
INSERT INTO captions VALUES(3,125,'and I passed my classes.');
INSERT INTO captions VALUES(3,127,'Being a mom at a young age');
INSERT INTO captions VALUES(3,130,'either makes you or breaks you,');
INSERT INTO captions VALUES(3,132,'but, in my case, it made me.');
INSERT INTO captions VALUES(3,134,'It made me have a total different visual for my future.');
INSERT INTO captions VALUES(3,141,'I am the first person in my family');
INSERT INTO captions VALUES(3,143,'to graduate from high school.');
INSERT INTO captions VALUES(4,0,'language:en;This is one of five student perspectives featured in the PBS special "TED Talks Education" (#TEDTalksed.) Learn more here: https://www.ted.com/promos/TEDTalksEducation Special thanks to our partners in this endeavor: WNET;PBS and the Corporation for Public Broadcasting.;Education;\Shayna Cody\;TED-Ed;\TED;Ed\;TEDEducation;TED;PBS\;#TEDTalksEd');
INSERT INTO captions VALUES(4,0,'Transcriber: Andrea McDonough Reviewer: Jessica Ruby');
INSERT INTO captions VALUES(4,9,'My name is Shayna Cody,');
INSERT INTO captions VALUES(4,10,'and I go to Bard High School Early College.');
INSERT INTO captions VALUES(4,13,'I''m a twin.');
INSERT INTO captions VALUES(4,14,'I''m so competitive with my twin sister.');
INSERT INTO captions VALUES(4,18,'We compete about everything.');
INSERT INTO captions VALUES(4,21,'When my sister finishes homework before me,');
INSERT INTO captions VALUES(4,23,'she usually rubs it in my face.');
INSERT INTO captions VALUES(4,25,'We''re just so competitive about everything.');
INSERT INTO captions VALUES(4,28,'I think a fully-rounded education is not just sitting there');
INSERT INTO captions VALUES(4,32,'doing the work that''s required of you,');
INSERT INTO captions VALUES(4,34,'but actually taking the time to, like, focus in');
INSERT INTO captions VALUES(4,37,'and learn what you can out of your classes.');
INSERT INTO captions VALUES(4,41,'I feel like, whatever I do,');
INSERT INTO captions VALUES(4,43,'I want to be the best at what I do.');
INSERT INTO captions VALUES(4,47,'Failure is not an option.');
INSERT INTO captions VALUES(4,51,'My dream job when I leave school');
INSERT INTO captions VALUES(4,53,'is to become a cardiologist.');
INSERT INTO captions VALUES(4,56,'Every Saturday morning, I go to a program');
INSERT INTO captions VALUES(4,59,'called The Lang Youth Medical Program.');
INSERT INTO captions VALUES(4,62,'It''s a pre-medical program where we prepare');
INSERT INTO captions VALUES(4,64,'to become doctors.');
INSERT INTO captions VALUES(4,66,'It''s a six-year program.');
INSERT INTO captions VALUES(4,67,'I got in at 7th grade in my middle school,');
INSERT INTO captions VALUES(4,70,'and they stay with you all the way up until 12th grade.');
INSERT INTO captions VALUES(4,73,'We learn about viruses,');
INSERT INTO captions VALUES(4,74,'we learn about the body.');
INSERT INTO captions VALUES(4,77,'I feel I''ve grown a whole lot');
INSERT INTO captions VALUES(4,79,'since I''ve been in high school.');
INSERT INTO captions VALUES(4,81,'I feel like I''ve had to grow up.');
INSERT INTO captions VALUES(4,83,'You can''t be the same person');
INSERT INTO captions VALUES(4,84,'that you were when you went in there,');
INSERT INTO captions VALUES(4,86,'otherwise, you''re not going to get anything out of it.');
CREATE INDEX [idx_playlists_id]
    ON [playlists] ([id]);
CREATE INDEX [idx_playlists_time_modified]
    ON [playlists] ([time_modified]);
CREATE INDEX [idx_playlists_time_deleted]
    ON [playlists] ([time_deleted]);
CREATE INDEX [idx_playlists_time_created]
    ON [playlists] ([time_created]);
CREATE INDEX [idx_playlists_hours_update_delay]
    ON [playlists] ([hours_update_delay]);
CREATE INDEX [idx_playlists_uploader]
    ON [playlists] ([uploader]);
CREATE INDEX [idx_playlists_extractor_config]
    ON [playlists] ([extractor_config]);
CREATE INDEX [idx_playlists_profile]
    ON [playlists] ([profile]);
CREATE UNIQUE INDEX [idx_playlists_path]
    ON [playlists] ([path]);
CREATE INDEX [idx_playlists_extractor_key]
    ON [playlists] ([extractor_key]);
CREATE INDEX [idx_media_id]
    ON [media] ([id]);
CREATE INDEX [idx_media_time_deleted]
    ON [media] ([time_deleted]);
CREATE INDEX [idx_media_playlists_id]
    ON [media] ([playlists_id]);
CREATE INDEX [idx_media_size]
    ON [media] ([size]);
CREATE INDEX [idx_media_duration]
    ON [media] ([duration]);
CREATE INDEX [idx_media_time_created]
    ON [media] ([time_created]);
CREATE INDEX [idx_media_time_modified]
    ON [media] ([time_modified]);
CREATE INDEX [idx_media_time_downloaded]
    ON [media] ([time_downloaded]);
CREATE INDEX [idx_media_fps]
    ON [media] ([fps]);
CREATE INDEX [idx_media_view_count]
    ON [media] ([view_count]);
CREATE INDEX [idx_media_uploader]
    ON [media] ([uploader]);
CREATE UNIQUE INDEX [idx_media_path]
    ON [media] ([path]);
CREATE TRIGGER [playlists_ai] AFTER INSERT ON [playlists] BEGIN
  INSERT INTO [playlists_fts] (rowid, [path], [title]) VALUES (new.rowid, new.[path], new.[title]);
END;
CREATE TRIGGER [playlists_ad] AFTER DELETE ON [playlists] BEGIN
  INSERT INTO [playlists_fts] ([playlists_fts], rowid, [path], [title]) VALUES('delete', old.rowid, old.[path], old.[title]);
END;
CREATE TRIGGER [playlists_au] AFTER UPDATE ON [playlists] BEGIN
  INSERT INTO [playlists_fts] ([playlists_fts], rowid, [path], [title]) VALUES('delete', old.rowid, old.[path], old.[title]);
  INSERT INTO [playlists_fts] (rowid, [path], [title]) VALUES (new.rowid, new.[path], new.[title]);
END;
CREATE TRIGGER [media_ai] AFTER INSERT ON [media] BEGIN
  INSERT INTO [media_fts] (rowid, [path], [webpath], [title]) VALUES (new.rowid, new.[path], new.[webpath], new.[title]);
END;
CREATE TRIGGER [media_ad] AFTER DELETE ON [media] BEGIN
  INSERT INTO [media_fts] ([media_fts], rowid, [path], [webpath], [title]) VALUES('delete', old.rowid, old.[path], old.[webpath], old.[title]);
END;
CREATE TRIGGER [media_au] AFTER UPDATE ON [media] BEGIN
  INSERT INTO [media_fts] ([media_fts], rowid, [path], [webpath], [title]) VALUES('delete', old.rowid, old.[path], old.[webpath], old.[title]);
  INSERT INTO [media_fts] (rowid, [path], [webpath], [title]) VALUES (new.rowid, new.[path], new.[webpath], new.[title]);
END;
PRAGMA writable_schema=OFF;
COMMIT;

holta · 2024-09-19T13:26:58Z

@codewiz mentions that he appreciates the "declarative style" of this PR, so that de facto schema(s) are more understandable to all 👍

avni · 2024-09-20T02:35:55Z

Seeing the schema like this does make it much easier to go through. Thanks @deldesir!

Couple of questions:

book_media_map: If book_id is a primary key in this table, is this really a table for books, not a mapping table?
If though, the book_media_map table is supposed to link to the calibredb table in some way: https://manual.calibre-ebook.com/generated/en/calibredb.html, should book_id be a foreign key in the map table?
media: should playlists_id be a FK to the playlists table?
Can a media object belong to more than one playlists? If so, then it may make be helpful for the schema to have a media_playlist_map instead of storing the playlists_id in the media table.
Is there a notion of user in Calibre-Web? If so, you may want to house user_id in the history table if you need to be able to distinguish user histories.

avni · 2024-09-20T13:25:42Z

Couple of notes from today's 9/20 call:

Clarif: There are a few modifications @deldesir has made to the xklb db, primarily the addition of the book_media_map table and augmenting the captions table with a primary key and a foreign key to reference media_id but most of the schema is from the upstream xklb. Very helpful to know!
The map table book_media_map can be prefixed with iiab_ to make clear this is a new addition (@holta mentioned that @codewiz suggested this).
What is the purpose of the book_media_map table and how do you intend to use it / for what use cases?
Meta point: Consider how to keep the data across the 3-4 systems consistent, e.g., the book_media_map table is trying to map the calibredb db and xklb but there's no FK constraint on book_id. Garbage collection may be okay in certain cases, but it may be helpful to leverage RDBMS properties (such as PK/FK constraints) where possible to keep data consistent. It is possible to have a PK and FK on the same db column if needed. @deldesir noted that part of the challenge is the calibredb and xklb are in two different physical databases meaning it’s non-trivial to refer to both of them from the same application.

deldesir · 2024-09-20T14:08:57Z

book_media_map: If book_id is a primary key in this table, is this really a table for books, not a mapping table?

Technically there is a table for books, but it's in another db (metadata.db). The book_id (books.id) is read from

calibre-web/cps/editbooks.py

Line 265 in 2f7ae47

book_id = db_book.id

and inserted into book_media_map.

If though, the book_media_map table is supposed to link to the calibredb table in some way: https://manual.calibre-ebook.com/generated/en/calibredb.html, should book_id be a foreign key in the map table?

We cannot link two databases by joining there tables columns here, book_media_map was done out of the desire to keep a record of book_id with relatioship with media_id from xklb.db.

media: should playlists_id be a FK to the playlists table?

Since the goal here is to understand and replicate the official schema in order to use it in a declarative form, this needs to be done upstream by the maintainer in relationship with the overall structure of xklb.

I guess this is set as it is as a denormalization strategy necessary for xklb functionality around playlists...

Can a media object belong to more than one playlists? If so, then it may make be helpful for the schema to have a media_playlist_map instead of storing the playlists_id in the media table.

This would introduce redundancy since all playlists are computed to populate bookshelves stored in app.db.

Is there a notion of user in Calibre-Web? If so, you may want to house user_id in the history table if you need to be able to distinguish user histories.

The history table might have been generated by using xklb search feature. It's xklb related. Calibre-Web user settings are stored in app.db.

avni · 2024-09-20T15:36:07Z

This would introduce redundancy since all playlists are computed to populate bookshelves stored in app.db.

I don't understand the reasoning here. Right now, if a media object is in more than one playlist, there will be multiple rows for the same media object in the media table....

holta · 2024-09-20T16:09:13Z

This would introduce redundancy since all playlists are computed to populate bookshelves stored in app.db.

I don't understand the reasoning here. Right now, if a media object is in more than one playlist, there will be multiple rows for the same media object in the media table....

Great question but can we arrange a bookshelf/playlist design call in coming weeks, to solve these larger questions? e.g. to begin working thru the many tensions between the config/app.db view of bookshelves and the xklb-metadata.db view of playlists!

Short term, I'd ask that we keep focus on individual videos, to solve the actual problem at hand without getting carried away. In other words, first and foremost we should help @deldesir to debug this PR:

PR Query captions/subtitles [e.g. video transcripts] from xklb-metadata.db #244

Working thru the most immediate problems articulated here:

PR #244 [searching] is prone to false positives [ALSO: database integrity questions about safely/synchronously deleting "videos" from both/all DB's] #246

And if on the side that forces us to submit a draft proposal as to how portable bookshelves or portable playlists should in fact / eventually work, that's A-OK as an optional bonus, as an early blueprint for future work. 💯

avni · 2024-09-20T16:12:08Z

+1 on focusing on videos. playlists can definitely come later!

deldesir added 5 commits September 12, 2024 11:10

Create xb.py

1df37a2

Define core tables for xklb db

f96e0bd

Get book ID mapped to media ID

2ca806c

Function to get a specific book (which maps to a media entry) for a given caption

Create blank database

d5571b0

Add xklb db initialization

58c8341

holta added the enhancement New feature or request label Sep 12, 2024

deldesir self-assigned this Sep 12, 2024

holta changed the title ~~Create xb.py~~ Create xb.py [beginning to align schemas!] Sep 12, 2024

deldesir added 3 commits September 12, 2024 12:42

Add missing Boolean import

4e1fd5d

Add add_book_media_mapping function

4dc4a37

Comment stats table

e062395

holta changed the title ~~Create xb.py [beginning to align schemas!]~~ New xb.py: Possible schema to standardize xklb db's + main operations? [need to align w/ Calibre-Web's metadata.db & app.db!] Sep 12, 2024

holta mentioned this pull request Sep 12, 2024

Integrate xklb db in Calibre-Web #247

Draft

Add missing relationship

f2c76e3

holta mentioned this pull request Sep 20, 2024

PR #244 [searching] is prone to false positives [ALSO: database integrity questions about safely/synchronously deleting "videos" from both/all DB's] #246

Open

holta marked this pull request as draft September 20, 2024 15:13

deldesir added 3 commits September 24, 2024 12:32

Simplify xklb db modeling

d3cebad

Remove unused code

b3b2c23

Enhance session

1976c21

deldesir marked this pull request as ready for review September 25, 2024 02:08

This was referenced Sep 25, 2024

Begin with blank xklb-metadata.db from template? [w/ key columns added for Calibre-Web videos] iiab/iiab#3808

Closed

Model xklb-metadata.db using SQLAlchemy #259

Open

deldesir marked this pull request as draft October 16, 2024 11:12

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

New xb.py: Possible schema to standardize xklb db's + main operations? [need to align w/ Calibre-Web's metadata.db & app.db!] #253

New xb.py: Possible schema to standardize xklb db's + main operations? [need to align w/ Calibre-Web's metadata.db & app.db!] #253

deldesir commented Sep 12, 2024 •

edited

Loading

holta commented Sep 13, 2024 •

edited

Loading

deldesir commented Sep 19, 2024 •

edited by holta

Loading

deldesir commented Sep 19, 2024 •

edited

Loading

holta commented Sep 19, 2024 •

edited

Loading

holta commented Sep 19, 2024

avni commented Sep 20, 2024 •

edited

Loading

avni commented Sep 20, 2024 •

edited

Loading

deldesir commented Sep 20, 2024 •

edited

Loading

avni commented Sep 20, 2024

holta commented Sep 20, 2024 •

edited

Loading

avni commented Sep 20, 2024

New xb.py: Possible schema to standardize xklb db's + main operations? [need to align w/ Calibre-Web's metadata.db & app.db!] #253

Are you sure you want to change the base?

New xb.py: Possible schema to standardize xklb db's + main operations? [need to align w/ Calibre-Web's metadata.db & app.db!] #253

Conversation

deldesir commented Sep 12, 2024 • edited Loading

holta commented Sep 13, 2024 • edited Loading

deldesir commented Sep 19, 2024 • edited by holta Loading

deldesir commented Sep 19, 2024 • edited Loading

holta commented Sep 19, 2024 • edited Loading

holta commented Sep 19, 2024

avni commented Sep 20, 2024 • edited Loading

avni commented Sep 20, 2024 • edited Loading

deldesir commented Sep 20, 2024 • edited Loading

avni commented Sep 20, 2024

holta commented Sep 20, 2024 • edited Loading

avni commented Sep 20, 2024

deldesir commented Sep 12, 2024 •

edited

Loading

holta commented Sep 13, 2024 •

edited

Loading

deldesir commented Sep 19, 2024 •

edited by holta

Loading

deldesir commented Sep 19, 2024 •

edited

Loading

holta commented Sep 19, 2024 •

edited

Loading

avni commented Sep 20, 2024 •

edited

Loading

avni commented Sep 20, 2024 •

edited

Loading

deldesir commented Sep 20, 2024 •

edited

Loading

holta commented Sep 20, 2024 •

edited

Loading