Data Pipeline

This repository contains the code for the data pipeline of the project. The data pipeline is responsible for the following tasks:

Syncing all the data from the data sources to our own database and typesense cluster
Preprocessing and cleaning the data
Generating new JSON files for the frontend

Name		Name	Last commit message	Last commit date
Latest commit History 23 Commits
.vscode		.vscode
data		data
generated/collections		generated/collections
openai-batches		openai-batches
openiti-to-turath		openiti-to-turath
output		output
patches		patches
prisma		prisma
samples		samples
scripts		scripts
src		src
test		test
.env.example		.env.example
.eslintrc.cjs		.eslintrc.cjs
.gitignore		.gitignore
README.md		README.md
airtable-texts.json		airtable-texts.json
authors-conflicts.json		authors-conflicts.json
books-conflicts.json		books-conflicts.json
docker-compose.yml		docker-compose.yml
drizzle.config.ts		drizzle.config.ts
index.html		index.html
package.json		package.json
pnpm-lock.yaml		pnpm-lock.yaml
test.jsonl		test.jsonl
tsconfig.json		tsconfig.json
unlinked-authors.json		unlinked-authors.json

Provide feedback