Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Challenges Installing and Using Library in Next.js 14 #145

Open
agusoler000 opened this issue Mar 26, 2024 · 10 comments
Open

Challenges Installing and Using Library in Next.js 14 #145

agusoler000 opened this issue Mar 26, 2024 · 10 comments

Comments

@agusoler000
Copy link

I'm facing multiple challenges while attempting to install and utilize a library in my Next.js 14 project. Here's a comprehensive overview of the encountered errors:

Module Resolution Errors: I'm consistently receiving "Module not found" errors for various modules such as npm, nock, and others. These modules are crucial for my project's functionality, but the errors persist despite their installation and presence in the project's dependencies.

HTML Loader Configuration: There seems to be a misconfiguration or absence of proper HTML loader configuration, leading to errors during the webpack compilation process. These errors specifically pertain to the handling of HTML files within the project.

Missing package.json Error: Additionally, there's an error indicating the absence of package.json, which should exist in the expected directory. This missing file might be contributing to the module resolution issues and further complicating the webpack compilation.

Webpack Compilation Errors: During the webpack compilation phase, I'm encountering errors related to missing modules such as npm and nock. These errors halt the compilation process and prevent the successful building of the project.

Despite attempts to reinstall dependencies, review webpack configurations, and ensure proper Next.js setup, these issues persist. I'm seeking guidance and assistance on resolving these errors and effectively integrating the library into my Next.js 14 project.

Any insights or suggestions from the community would be immensely appreciated.

Thank you for your support!

@mmomtchev
Copy link
Owner

What does this have to do with gdal-async? gdal-async is a Node.js-only native library without browser support?

@agusoler000
Copy link
Author

@mmomtchev
I apologize for my poor wording. I'm trying to use gdal-async on the server side (next 14) to process TIFF images, but I'm unable to get it to work due to the errors I mentioned earlier. If you have any suggestions or assistance on why this might be happening, it would be greatly appreciated

@mmomtchev
Copy link
Owner

This is likely a problem with your configuration and not gdal-async itself. First of all, if you remove gdal-async do the errors disappear?

Post the errors you are getting.

@agusoler000
Copy link
Author

Yes, I encounter errors when I use the library, not when installing it and running the application:

import { NextResponse } from 'next/server';
import { NextApiResponse, NextApiRequest } from 'next';
import  gdal from 'gdal-async'

export  async function POST(req , res: NextApiResponse) {
  try {

  const buffer = await getBufferFromRequestBody(req);
  debugger
  const dataset = await gdal.openAsync(buffer);
  // rest of the code...  

  debugger;

  } catch (error) {
    console.error('Error processing the image:', error);
    // res.status(500).json({ error: 'Error processing the image' });
  }

}

```And the first error it gave me was:



> ⨯ ../node_modules/gdal-async/node_modules/@mapbox/node-pre-gyp/lib/util/nw-pre-gyp/index.html
> Module parse failed: Unexpected token (1:0)
> You may need an appropriate loader to handle this file type, currently no loaders are configured to process this file. See https://webpack.js.org/concepts#loaders
> > <!doctype html>
> | <html>
> | <head>
> Import trace for requested module:
> ../node_modules/gdal-async/node_modules/@mapbox/node-pre-gyp/lib/util/nw-pre-gyp/index.html
> ../node_modules/gdal-async/node_modules/@mapbox/node-pre-gyp/lib/ sync ^\.\/.*$
> ../node_modules/gdal-async/node_modules/@mapbox/node-pre-gyp/lib/node-pre-gyp.js
> ../node_modules/gdal-async/lib/gdal.js
> ./app/api/tiff/route.ts
>  ⨯ ../node_modules/gdal-async/node_modules/@mapbox/node-pre-gyp/lib/util/nw-pre-gyp/index.html
> Module parse failed: Unexpected token (1:0)
> You may need an appropriate loader to handle this file type, currently no loaders are configured to process this file. See https://webpack.js.org/concepts#loaders
> > <!doctype html>
> | <html>
> | <head>
> Import trace for requested module:
> ../node_modules/gdal-async/node_modules/@mapbox/node-pre-gyp/lib/util/nw-pre-gyp/index.html
> ../node_modules/gdal-async/node_modules/@mapbox/node-pre-gyp/lib/ sync ^\.\/.*$
> ../node_modules/gdal-async/node_modules/@mapbox/node-pre-gyp/lib/node-pre-gyp.js
> ../node_modules/gdal-async/lib/gdal.js
> ./app/api/tiff/route.ts
>  ⨯ ../node_modules/gdal-async/node_modules/@mapbox/node-pre-gyp/lib/util/nw-pre-gyp/index.html
> Module parse failed: Unexpected token (1:0)
> You may need an appropriate loader to handle this file type, currently no loaders are configured to process this file. See https://webpack.js.org/concepts#loaders
> > <!doctype html>
> | <html>
> | <head>
> Import trace for requested module:
> ../node_modules/gdal-async/node_modules/@mapbox/node-pre-gyp/lib/util/nw-pre-gyp/index.html
> ../node_modules/gdal-async/node_modules/@mapbox/node-pre-gyp/lib/ sync ^\.\/.*$
> ../node_modules/gdal-async/node_modules/@mapbox/node-pre-gyp/lib/node-pre-gyp.js
> ../node_modules/gdal-async/lib/gdal.js
> ./app/api/tiff/route.ts
>  ⨯ ../node_modules/gdal-async/node_modules/@mapbox/node-pre-gyp/lib/util/nw-pre-gyp/index.html
> Module parse failed: Unexpected token (1:0)
> You may need an appropriate loader to handle this file type, currently no loaders are configured to process this file. See https://webpack.js.org/concepts#loaders
> > <!doctype html>
> | <html>
> | <head>
> Import trace for requested module:
> ../node_modules/gdal-async/node_modules/@mapbox/node-pre-gyp/lib/util/nw-pre-gyp/index.html
> ../node_modules/gdal-async/node_modules/@mapbox/node-pre-gyp/lib/ sync ^\.\/.*$
> ../node_modules/gdal-async/node_modules/@mapbox/node-pre-gyp/lib/node-pre-gyp.js
> ../node_modules/gdal-async/lib/gdal.js
> ./app/api/tiff/route.ts

___

> ⨯ ../node_modules/gdal-async/node_modules/@mapbox/node-pre-gyp/lib/util/s3_setup.js:43:20
> Module not found: Can't resolve 'mock-aws-s3'
> https://nextjs.org/docs/messages/module-not-found
> Import trace for requested module:
> ../node_modules/gdal-async/node_modules/@mapbox/node-pre-gyp/lib/node-pre-gyp.js
> ../node_modules/gdal-async/lib/gdal.js
> ./app/api/tiff/route.ts
>  ⨯ ../node_modules/gdal-async/node_modules/@mapbox/node-pre-gyp/lib/util/s3_setup.js:43:20
> Module not found: Can't resolve 'mock-aws-s3'
> https://nextjs.org/docs/messages/module-not-found
> Import trace for requested module:
> ../node_modules/gdal-async/node_modules/@mapbox/node-pre-gyp/lib/node-pre-gyp.js
> ../node_modules/gdal-async/lib/gdal.js
> ./app/api/tiff/route.ts
>  ⨯ ../node_modules/gdal-async/node_modules/@mapbox/node-pre-gyp/lib/util/s3_setup.js:43:20
> Module not found: Can't resolve 'mock-aws-s3'
> https://nextjs.org/docs/messages/module-not-found
> Import trace for requested module:
> ../node_modules/gdal-async/node_modules/@mapbox/node-pre-gyp/lib/node-pre-gyp.js
> ../node_modules/gdal-async/lib/gdal.js
> ./app/api/tiff/route.ts

__________

> ⨯ ../node_modules/gdal-async/node_modules/@mapbox/node-pre-gyp/lib/util/s3_setup.js:76:14
> Module not found: Can't resolve 'aws-sdk'
> https://nextjs.org/docs/messages/module-not-found
> Import trace for requested module:
> ../node_modules/gdal-async/node_modules/@mapbox/node-pre-gyp/lib/node-pre-gyp.js
> ../node_modules/gdal-async/lib/gdal.js
> ./app/api/tiff/route.ts
>  ⨯ ../node_modules/gdal-async/node_modules/@mapbox/node-pre-gyp/lib/util/s3_setup.js:76:14
> Module not found: Can't resolve 'aws-sdk'
> https://nextjs.org/docs/messages/module-not-found
> Import trace for requested module:
> ../node_modules/gdal-async/node_modules/@mapbox/node-pre-gyp/lib/node-pre-gyp.js
> ../node_modules/gdal-async/lib/gdal.js
> ./app/api/tiff/route.ts
> <w> [webpack.cache.PackFileCacheStrategy] Restoring pack failed from C:\Users\agust\OneDrive\Escritorio\Proyects\fase_c\avincis_fase_C\inspec-front\inspec\.next\cache\webpack\client-development-fallback.pack.gz: Error: incorrect header check
>  ⨯ ../node_modules/gdal-async/node_modules/@mapbox/node-pre-gyp/lib/util/s3_setup.js:76:14
> Module not found: Can't resolve 'aws-sdk'
> https://nextjs.org/docs/messages/module-not-found
> Import trace for requested module:
> ../node_modules/gdal-async/node_modules/@mapbox/node-pre-gyp/lib/node-pre-gyp.js
> ../node_modules/gdal-async/lib/gdal.js
> ./app/api/tiff/route.ts
>  ⨯ ../node_modules/gdal-async/node_modules/@mapbox/node-pre-gyp/lib/util/s3_setup.js:76:14
> Module not found: Can't resolve 'aws-sdk'
> https://nextjs.org/docs/messages/module-not-found
> Import trace for requested module:
> ../node_modules/gdal-async/node_modules/@mapbox/node-pre-gyp/lib/node-pre-gyp.js
> ../node_modules/gdal-async/lib/gdal.js
> ./app/api/tiff/route.ts



It seems like it doesn't install all the necessary dependencies.


**I would like to take this opportunity to thank you for the great work you do; thanks to you, my job has been much easier. I also use RLAYERS, and it's an excellent library!**

@mmomtchev
Copy link
Owner

mmomtchev commented Mar 26, 2024

The problem is node-pre-gyp which must load the native binary. I have done this before with rollup in https://github.com/mmomtchev/xc-db/ but never with webpack. You must find a way to exclude the node-pre-gyp dependencies from your project. node-pre-gyp is the build framework that includes lots of packages that cannot be packed for the web. Once the project is built, it is used only when loading the module to find the binary path:

const binding_path = binary.find(path.resolve(path.join(__dirname, '../package.json')))
const gdal = require(binding_path)

(from lib/gdal.js)

If you hardcode the binary path as ${__dirname}/../lib/binding/node-v111-linux-x64 (check your path), you won't need node-pre-gyp.

Check those links:
mapbox/node-pre-gyp#308
https://stackoverflow.com/questions/69775419/webpack-ignore-node-pre-gyp-files

I remember that I had to improve the node-pre-gyp support in rollup for it to work: danielgindi/rollup-plugin-natives#13

@mmomtchev
Copy link
Owner

@agusoler000 if you can't find a proper solution, maybe the cleanest method would be to replace the @mapbox/node-pre-gyp package with a package that implements only find() and returns the path to the .node file. Then, you will need to copy this .node file and to keep that require that loads it.

@ProductOfAmerica
Copy link

ProductOfAmerica commented May 21, 2024

@mmomtchev could my error be related to the issue you described above?

⨯ TypeError: binary.find is not a function
   at eval (webpack-internal:///(rsc)/./node_modules/.pnpm/[email protected]/node_modules/gdal-async/lib/gdal.js:5:29)
   at (rsc)/./node_modules/.pnpm/[email protected]/node_modules/gdal-async/lib/gdal.js (...\.next\server\vendor-chunks\[email protected]:59:1)
   at __webpack_require__ (...\.next\server\webpack-runtime.js:33:43)
   at eval (webpack-internal:///(rsc)/./app/(public)/api/test/route.ts:7:68)
   at (rsc)/./app/(public)/api/test/route.ts (...\.next\server\app\(public)\api\test\route.js:150:1)
   at __webpack_require__ (...\.next\server\webpack-runtime.js:33:43)
   at eval (webpack-internal:///(rsc)/./node_modules/.pnpm/[email protected]_@[email protected][email protected][email protected][email protected]/node_modules/next/dist/build/webpack/loaders/next-app-loader.js?name=app%2F(public)%2Fapi%2Ftest%2Froute&page=%2F(publ
ic)%2Fapi%2Ftest%2Froute&appPaths=&pagePath=private-next-app-dir%2F(public)%2Fapi%2Ftest%2Froute.ts&appDir=C%3A%5C%5Capp&pageExtensions=tsx&pageExtensions=ts&pageExtensions=jsx&pageExtensions=js&rootDir=C%3A%5C&isDev=true&tsconfigPath=tsconfig.json&basePath=&assetPrefix=&nextConfigOutput=&preferredRegion=&middlewareConfig=e30%3D!:15:124)
   at (rsc)/./node_modules/.pnpm/[email protected]_@[email protected][email protected][email protected][email protected]/node_modules/next/dist/build/webpack/loaders/next-app-loader.js?name=app%2F(public)%2Fapi%2Ftest%2Froute&page=%2F(public)%2Fapi%2Ftest%2Froute&a
ppPaths=&pagePath=private-next-app-dir%2F(public)%2Fapi%2Ftest%2Froute.ts&&isDev=true&tsconfigPath=tsconfig.json&basePath=&assetPrefix=&nextConfigOutput=&preferredRegion=&middlewareConfig=e30%3D! (...\.next\server\app\(public)\api\test\route.js:139:1)
   at __webpack_require__ (...\.next\server\webpack-runtime.js:33:43)
   at __webpack_exec__ (...\.next\server\app\(public)\api\test\route.js:160:39)
   at ...\.next\server\app\(public)\api\test\route.js:161:430
   at __webpack_require__.X (...\.next\server\webpack-runtime.js:121:21)
   at ...\.next\server\app\(public)\api\test\route.js:161:47
   at Object.<anonymous> (...\.next\server\app\(public)\api\test\route.js:164:3)
   at Module._compile (node:internal/modules/cjs/loader:1256:14)
   at Module._extensions..js (node:internal/modules/cjs/loader:1310:10)
   at Module.load (node:internal/modules/cjs/loader:1119:32)
   at Module._load (node:internal/modules/cjs/loader:960:12)
   at Module.require (node:internal/modules/cjs/loader:1143:19)
   at mod.require (...\node_modules\.pnpm\[email protected]_@[email protected][email protected][email protected][email protected]\node_modules\next\dist\server\require-hook.js:65:28)
   at require (node:internal/modules/cjs/helpers:119:18)
   at requirePage (...\node_modules\.pnpm\[email protected]_@[email protected][email protected][email protected][email protected]\node_modules\next\dist\server\require.js:109:84)
   at ...\node_modules\.pnpm\[email protected]_@[email protected][email protected][email protected][email protected]\node_modules\next\dist\server\load-components.js:98:84
   at async loadComponentsImpl (...\node_modules\.pnpm\[email protected]_@[email protected][email protected][email protected][email protected]\node_modules\next\dist\server\load-components.js:98:26)
   at async DevServer.findPageComponentsImpl (...\node_modules\.pnpm\[email protected]_@[email protected][email protected][email protected][email protected]\node_modules\next\dist\server\next-server.js:710:36)
   at async DevServer.findPageComponents (...\node_modules\.pnpm\[email protected]_@[email protected][email protected][email protected][email protected]\node_modules\next\dist\server\dev\next-dev-server.js:577:20)
   at async DevServer.renderPageComponent (...\node_modules\.pnpm\[email protected]_@[email protected][email protected][email protected][email protected]\node_modules\next\dist\server\base-server.js:1910:24)
   at async DevServer.renderToResponseImpl (...\node_modules\.pnpm\[email protected]_@[email protected][email protected][email protected][email protected]\node_modules\next\dist\server\base-server.js:1962:32)
   at async DevServer.pipeImpl (...\node_modules\.pnpm\[email protected]_@[email protected][email protected][email protected][email protected]\node_modules\next\dist\server\base-server.js:920:25)
   at async NextNodeServer.handleCatchallRenderRequest (...\node_modules\.pnpm\[email protected]_@[email protected][email protected][email protected][email protected]\node_modules\next\dist\server\next-server.js:272:17)
   at async DevServer.handleRequestImpl (...\node_modules\.pnpm\[email protected]_@[email protected][email protected][email protected][email protected]\node_modules\next\dist\server\base-server.js:816:17)
   at async ...\node_modules\.pnpm\[email protected]_@[email protected][email protected][email protected][email protected]\node_modules\next\dist\server\dev\next-dev-server.js:339:20
   at async Span.traceAsyncFn (...\node_modules\.pnpm\[email protected]_@[email protected][email protected][email protected][email protected]\node_modules\next\dist\trace\trace.js:154:20)
   at async DevServer.handleRequest (...\node_modules\.pnpm\[email protected]_@[email protected][email protected][email protected][email protected]\node_modules\next\dist\server\dev\next-dev-server.js:336:24)
   at async invokeRender (...\node_modules\.pnpm\[email protected]_@[email protected][email protected][email protected][email protected]\node_modules\next\dist\server\lib\router-server.js:174:21)
   at async handleRequest (...\node_modules\.pnpm\[email protected]_@[email protected][email protected][email protected][email protected]\node_modules\next\dist\server\lib\router-server.js:353:24)
   at async requestHandlerImpl (...\node_modules\.pnpm\[email protected]_@[email protected][email protected][email protected][email protected]\node_modules\next\dist\server\lib\router-server.js:377:13)
   at async Server.requestListener (...\node_modules\.pnpm\[email protected]_@[email protected][email protected][email protected][email protected]\node_modules\next\dist\server\lib\start-server.js:141:13) {
 page: '/api/test'
}

import path from 'path';
import gdal from 'gdal-async';
import { NextResponse } from 'next/server';

export async function GET({ params }: { params: { userId: string } }): Promise<NextResponse> {
	try {
		const dataset = await gdal.openAsync(
			path.resolve('C:\\NationalCSB_2016-2023_rev23\\CSB1623.gdb'),
		);

		const layer = dataset.layers.get(0);

		console.log('number of features: ' + layer.features.count());
		console.log('fields: ' + layer.fields.getNames());
		console.log('extent: ' + JSON.stringify(layer.extent));
		console.log('srs: ' + (layer.srs ? layer.srs.toWKT() : 'null'));

		return NextResponse.json([], { status: 200 });
	} catch (e) {
		console.error(e);
		return NextResponse.json([], { status: 500 });
	}
}

@ProductOfAmerica
Copy link

ProductOfAmerica commented May 21, 2024

@mmomtchev made a quick demo for you to clone: https://github.com/ProductOfAmerica/test-gdal
"CSB1623.gdb" can be downloaded here
I can't even get it working with plain express

@mmomtchev
Copy link
Owner

You are webpacking gdal, you are not supposed to do this, since gdal does not work in the browser (yet, now that there is SWIG JSE, there might be a future gdal for JavaScript version that works in the browser - but given the current state of affairs between me and the GIS community the project is unlikely in near future).

You have certainly something in your next.confg.js that makes it do this.

Try with this next.config.js, it is going to work much better:

/** @type {import('next').NextConfig} */

export default {
  webpack: (config) => {
    const externals = ['gdal-async'];
    if (config.externals)
      config.externals.push(...externals);
    else
      config.externals = externals;
    return config;
  }
};

Plain express does not webpack, so your problem must be different.

@ProductOfAmerica
Copy link

@mmomtchev I guess ignore the express errors. You solved my nextjs problem, so thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants