Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: allow for external database drivers #2744

Open
wants to merge 8 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 7 additions & 2 deletions npm/src/db/db.ts
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
import {
DatabaseDriver,
DatabaseDriverOption,
DatabaseOption,
Encrypted,
EncryptionKey,
Expand Down Expand Up @@ -130,9 +131,13 @@ class DB implements DatabaseDriver {
}
}

const _new = async (options: DatabaseOption) => {
const _new = async (options: DatabaseOption | DatabaseDriverOption) => {
const encryptionKey = options.encryptionKey ? Buffer.from(options.encryptionKey, 'latin1') : null;

if ('driver' in options) {
return new DB(options.driver, encryptionKey)
}

switch (options.engine) {
case 'redis':
return new DB(await redis.new(options), encryptionKey);
Expand Down Expand Up @@ -199,7 +204,7 @@ const _new = async (options: DatabaseOption) => {
const g = global as any;

export default {
new: async (options: DatabaseOption, noCache = false) => {
new: async (options: DatabaseOption | DatabaseDriverOption, noCache = false) => {
if (g.__jacksonDb && !noCache) {
return g.__jacksonDb;
}
Expand Down
6 changes: 5 additions & 1 deletion npm/src/db/defaultDb.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2,10 +2,14 @@ import { JacksonOption } from '../typings';

export default function defaultDb(opts: JacksonOption) {
opts.db = opts.db || {};
opts.db.ttl = (opts.db.ttl || 300) * 1; // TTL for the code, session and token stores (in seconds)
if ('driver' in opts.db) {
return
}

opts.db.engine = opts.db.engine || 'sql';
opts.db.url = opts.db.url || 'postgresql://postgres:postgres@localhost:5432/postgres';
opts.db.type = opts.db.type || 'postgres'; // Only needed if DB_ENGINE is sql.
opts.db.ttl = (opts.db.ttl || 300) * 1; // TTL for the code, session and token stores (in seconds)
opts.db.cleanupLimit = (opts.db.cleanupLimit || 1000) * 1; // Limit cleanup of TTL entries to this many items at a time
opts.db.dynamodb = opts.db.dynamodb || {};
opts.db.dynamodb.region = opts.db.dynamodb.region || 'us-east-1';
Expand Down
9 changes: 6 additions & 3 deletions npm/src/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -164,9 +164,12 @@ export const controllers = async (
await analyticsController.init();
}

const type = opts.db.engine === 'sql' && opts.db.type ? ' Type: ' + opts.db.type : '';

console.info(`Using engine: ${opts.db.engine}.${type}`);
if ('driver' in opts.db) {
console.info(`Using external database driver`)
} else {
const type = opts.db.engine === 'sql' && opts.db.type ? ' Type: ' + opts.db.type : '';
console.info(`Using engine: ${opts.db.engine}.${type}`);
}

return {
spConfig,
Expand Down
9 changes: 8 additions & 1 deletion npm/src/typings.ts
Original file line number Diff line number Diff line change
Expand Up @@ -434,14 +434,21 @@ export interface DatabaseOption {
manualMigration?: boolean;
}

export interface DatabaseDriverOption {
driver: DatabaseDriver
encryptionKey?: string;
ttl?: number;
pageLimit?: number;
}

export interface JacksonOption {
externalUrl: string;
samlPath: string;
oidcPath?: string;
samlAudience?: string;
preLoadedConnection?: string;
idpEnabled?: boolean;
db: DatabaseOption;
db: DatabaseOption | DatabaseDriverOption;
clientSecretVerifier?: string;
idpDiscoveryPath?: string;
scimPath?: string;
Expand Down
26 changes: 21 additions & 5 deletions npm/test/db/db.test.ts
Original file line number Diff line number Diff line change
@@ -1,7 +1,8 @@
import { DatabaseOption, EncryptionKey, Storable, DatabaseDriver } from '../../src/typings';
import {DatabaseOption, EncryptionKey, Storable, DatabaseDriver, DatabaseDriverOption} from '../../src/typings';
import tap from 'tap';
import DB from '../../src/db/db';
import { randomBytes } from 'crypto';
import {SqliteDB} from '../../../packages/db-sqlite/src/index'

const encryptionKey: EncryptionKey = 'I+mnyTixBoNGu0OtpG0KXJSunoPTiWMb';

Expand Down Expand Up @@ -126,6 +127,15 @@ const sqliteConfig = <DatabaseOption>{
pageLimit: 2,
};

const sqliteDbConfig = <DatabaseDriverOption>{
driver: SqliteDB.new({
url: 'file:///var/tmp/test-sqlite-database.db',
ttl: 1,
cleanupLimit: 10,
manualMigration: true,
}),
}

const dbs = [
{
...memDbConfig,
Expand Down Expand Up @@ -190,6 +200,10 @@ const dbs = [
...tursoConfig,
encryptionKey,
},
{
...sqliteDbConfig,
encryptionKey
}
];

if (process.env.PLANETSCALE_URL) {
Expand Down Expand Up @@ -220,7 +234,8 @@ tap.before(async () => {
for (const idx in dbs) {
const opts = dbs[idx];
const db = await DB.new(opts, true);
dbObjs[opts.engine! + (opts.type ? opts.type : '')] = db;
const dbName = 'driver' in opts ? typeof opts.driver : opts.engine! + (opts.type ? opts.type : '')
dbObjs[dbName] = db;

const randomSession = Date.now();
connectionStores.push(db.store('saml:config:' + randomSession + randomBytes(4).toString('hex')));
Expand All @@ -236,10 +251,11 @@ tap.test('dbs', async () => {
for (const idx in connectionStores) {
const connectionStore = connectionStores[idx];
const ttlStore = ttlStores[idx];
const dbEngine = dbs[idx].engine!;
const db = dbs[idx];
const dbEngine = 'driver' in db ? typeof db.driver : db.engine!;
let dbType = dbEngine;
if (dbs[idx].type) {
dbType += ': ' + dbs[idx].type;
if (!('driver' in db) && db.type) {
dbType += ': ' + db.type;
}

tap.test('put(): ' + dbType, async () => {
Expand Down
1 change: 1 addition & 0 deletions npm/tsconfig.json
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@
"allowSyntheticDefaultImports": true,
"experimentalDecorators": true,
"downlevelIteration": true,
"baseUrl": ".",
},
"include": ["./src/**/*"],
"exclude": ["node_modules"],
Expand Down
27 changes: 27 additions & 0 deletions packages/db-sqlite/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
# Logs
logs
*.log
npm-debug.log*
yarn-debug.log*
yarn-error.log*
pnpm-debug.log*
lerna-debug.log*

node_modules
dist
dist-ssr
*.local

# Editor directories and files
.vscode/*
!.vscode/extensions.json
.idea
.DS_Store
*.suo
*.ntvs*
*.njsproj
*.sln
*.sw?

# Ignore coverage folder
coverage
95 changes: 95 additions & 0 deletions packages/db-sqlite/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,95 @@
# vite-vanilla-ts-template

<p align="center">
<img alt="license" src="https://img.shields.io/github/license/entwurfhaus/vite-vanilla-ts-module?style=flat-square" />
<img alt="release" src="https://img.shields.io/github/v/tag/entwurfhaus/vite-vanilla-ts-template?label=release&style=flat-square" />
<img alt="visits" src="https://hits.deltapapa.io/github/entwurfhaus/vite-vanilla-ts-template.svg" />
</p>

This is a minimalist Vite `vanilla-ts` template, for developing `ts` supported modules for publishing onto `npm`.

> Check out the older version of this template, https://github.com/entwurfhaus/vite-vanilla-ts-module.
> And always check the `package.json` dependencies - this template scaffolded from Vite 5.x, and compatible with Node 18.x & beyond

```bash
npx degit entwurfhaus/vite-vanilla-ts-template my-awesome-package
```

## Getting started

First, let's install all dependencies:

```bash
pnpm i
```

When you're ready to build and use your new `npm` package, run:

```bash
pnpm build
```

### Why `pnpm` and not `npm` or `yarn`?

This minimalist template is meant to be easily migrated to monorepo frameworks, such as `turbo` (Turborepo) and `nx` (Nx). Thus, it is why files like the `tsconfig.json` is simple.

### What is configured in this template?

We're using `vite-plugin-dts`, which is a useful `vite` plugin, to automatically generate & produce the `index.d.ts` file in the `dist`. Awesome!

Next, we're by default using `istanbul` code coverage integration with `vitest` - it's purely optional to use it, as the template can be reconfigured to `coverage-c8` (or new `c8` name if it is renamed again in near future).

### Since this template is considered minimalist, what's missing that would make this template better?

As a start `prettier` and `eslint` configurations have been left out, because I'd leave these configurations up to you - as they're not that difficult to configure. And of course, you may have a more complex configuration of `prettier` / `biomejs` and `eslint` and more, for your projects - such as for `turbo` (turborepo) or `nx` (NX).

> Bonus tip: If you want to easily manage your dependencies' versioning, check out `taze` (https://github.com/antfu/taze). `taze` is awesome, and so far it works well in a `nx` monorepo that I've been working on. There are useful commands like `pnpm up` / `pnpm up --latest` etc, or monorepo features that assist dependency management - so keep an eye out for new features, commands, packages in the near future!

## Testing

### Unit testing with `vitest`

Run `vitest` (without "watch" mode):

```bash
pnpm test
```

Or run `vitest` with code coverage report:

```bash
pnpm test:coverage
```

### Local testing only

Run below command in your new `npm` package repository. For example, `cd /var/www/my-awesome-package` then run:

```bash
pnpm link --global
```

Lastly, go to your desired application that will use your new `npm` package and run:

```bash
pnpm link /var/www/my-awesome-package
```

## Publishing

### NPM

And when ready to publish to `npm`:

```bash
npm login
npm publish
```

## References

The list of online references & to credit the open-source community, for making mini projects like these possible, **helping developers save many minutes** working.

1. Peter Mekhaeil | How to build an npx starter template - https://github.com/petermekhaeil/create-my-template
2. Netlify | Creating a TypeScript Package with Vite - https://onderonur.netlify.app/blog/creating-a-typescript-library-with-vite/
3. Jason Stuges | Github - https://github.com/jasonsturges/vite-typescript-npm-package
Loading