Listener memory leak in AI system

[colin-grant-work]

Bug Description:

There appears to be a memory leak of some kind in the AI system. When doing fairly ordinary chat interactions, I got the following warnings:

2025-02-11T19:19:17.819Z root WARN Possible Emitter memory leak detected. 1232 listeners added. Use event.maxListeners to increase the limit (175). MOST frequent listener (1057):
2025-02-11T19:19:17.819Z root WARN     at _event.Object.assign.maxListeners (...theia/examples/electron/lib/backend/packages_core_lib_common_index_js-node_modules_vscode-languageserver-types_lib_umd_sync_recursive.js:1456:54)
    at ...theia/examples/electron/lib/backend/main.js:6599:113
    at new Promise (<anonymous>)
    at [Symbol.asyncIterator] (...theia/examples/electron/lib/backend/main.js:6597:37)
    at asyncIterator.next (<anonymous>)
    at ...theia/examples/electron/lib/backend/main.js:5052:34
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
2025-02-11T19:19:22.113Z root WARN Possible Emitter memory leak detected. 1320 listeners added. Use event.maxListeners to increase the limit (175). MOST frequent listener (1145):
2025-02-11T19:19:22.113Z root WARN     at _event.Object.assign.maxListeners (...theia/examples/electron/lib/backend/packages_core_lib_common_index_js-node_modules_vscode-languageserver-types_lib_umd_sync_recursive.js:1456:54)
    at ...theia/examples/electron/lib/backend/main.js:6599:113
    at new Promise (<anonymous>)
    at [Symbol.asyncIterator] (...theia/examples/electron/lib/backend/main.js:6597:37)
    at asyncIterator.next (<anonymous>)
    at ...theia/examples/electron/lib/backend/main.js:5052:34
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
2025-02-11T19:19:25.447Z root WARN Possible Emitter memory leak detected. 1408 listeners added. Use event.maxListeners to increase the limit (175). MOST frequent listener (1233):
2025-02-11T19:19:25.447Z root WARN     at _event.Object.assign.maxListeners (...theia/examples/electron/lib/backend/packages_core_lib_common_index_js-node_modules_vscode-languageserver-types_lib_umd_sync_recursive.js:1456:54)
    at ...theia/examples/electron/lib/backend/main.js:6599:113
    at new Promise (<anonymous>)
    at [Symbol.asyncIterator] (...theia/examples/electron/lib/backend/main.js:6597:37)
    at asyncIterator.next (<anonymous>)
    at ...theia/examples/electron/lib/backend/main.js:5052:34
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
2025-02-11T19:19:29.901Z root WARN Possible Emitter memory leak detected. 1496 listeners added. Use event.maxListeners to increase the limit (175). MOST frequent listener (1321):
2025-02-11T19:19:29.901Z root WARN     at _event.Object.assign.maxListeners (...theia/examples/electron/lib/backend/packages_core_lib_common_index_js-node_modules_vscode-languageserver-types_lib_umd_sync_recursive.js:1456:54)
    at ...theia/examples/electron/lib/backend/main.js:6599:113
    at new Promise (<anonymous>)
    at [Symbol.asyncIterator] (...theia/examples/electron/lib/backend/main.js:6597:37)
    at asyncIterator.next (<anonymous>)
    at ...theia/examples/electron/lib/backend/main.js:5052:34
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

Those errors point to this code in the OpenAiLanguageModel

theia/packages/ai-openai/src/node/openai-language-model.ts

Lines 154 to 157 in c58b83c

	cancellationToken?.onCancellationRequested(() => {
	rej(new Error('Canceled'));
	runnerEnd = true; // Stop the iterator
	});

LanguageModelFrontendDeleteImpl

theia/packages/ai-core/src/node/language-model-frontend-delegate.ts

Lines 111 to 113 in c58b83c

	for await (const token of stream) {
	this.frontendDelegateClient.send(id, token);
	}

The problem seems to be the addition of a new onCancellationRequested listener for every iteration. Likely that should be extracted out to the surrounding scope.

Steps to Reproduce:

Basically the 'To test' script from #14901

Additional Information

• Operating System: MacOS Sequoia
• Theia Version: Update editors when ChangeElement added or replaced #14901, but basically master.

[JonasHelming]

@eneufeld @sdirix