SCAN IN PENDING PROCESS And CPU utilization Is Very High (99%)

[Official-BlackHat13]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

when ever I'm trying to scan a domain and scan status showing me every time in Pending even no scan is running in background And CPU utilization is very High

Expected Behavior

Must be start Scanning Subdomain .. Etc
Normal CPU utilization

Steps To Reproduce

1. Login into your Rengine account
2. go to target section and add domain and start scan
3. Scanning Process showing Status is Pending

Environment

- reNgine:2.1.3
- OS: Linux ip-172-26-6-128 6.9.7-cloud-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.9.7-1parrot1 (2024-07-18) x86_64 GNU/Linux
- Python:3.11.2
- Docker Compose: version 20.10.24+dfsg1, build 297e128

Anything else?

CPU utilization : 99.01%

Please See attached file for Proof

[github-actions]

Hey @Official-BlackHat13! 👋 Thanks for flagging this! 🐛🐞

Before we dig in, Let's make sure you have

🔍 Gone through the documentation: https://rengine.wiki
🕵️ Make sure it's not a known issue
📝 Provided us all the details related to this bug

[yogeshojha]

This is strange, I am looking into this.

[Sajibekanti]

This is strange, I am looking into this.

i faced the same error. I thought it was my VPS issue, but I reinstalled it and the problem was not solved.

[drgreenthumb93]

I've also encountered this problem. There was no scan in place, just the creation of a new project. Could only be solved by stopping via systemctl.

[2004gixxer600]

Yep same here. Stock install, no issues, not even scanning, cpu's pegged at 99-100% right out of the gate.

[thirdbyte]

Tried master and release/2.2.0. Same issue in both of them. Do we have a temporary workaround for this as of now? Using a particular release source or something else? @yogeshojha

[yogeshojha]

I tried to replicate this, I am already on both release/2.2.0 and master branch. However I don't have such higher CPU utilization. Probably its due to ollama? some pending llm stuffs going on?
or even celery

Can any of you mind attaching the logs and htop screenshot?

you can run this command htop and provide me the screenshot

also once you are in htop terminal, press F6 and use wrroes to select PERCENT_CPU and PERCENT_MEM, please attach both screenshots, this will let us know which tool is causing issues

@thirdbyte @2004gixxer600 @drgreenthumb93 @Sajibekanti Anyone of you can attach

[sanyam2602]

Same for me. Attaching the logs for the same, getting httpcore not installed error. @yogeshojha

[drgreenthumb93]

Seems like celery is causing the high CPU load. Running on Ryzen 9 5900X, 32GB RAM and a RTX 3080.

[yogeshojha]

aaah there's an exception for httpcore, @Official-BlackHat13 @Sajibekanti @drgreenthumb93 @sanyam2602 @thirdbyte Thanks guys for reporting this

I will be pushing the changes in a while, in the meantime, go to rengine/web/celery-entrypoint.sh

You will see this line

# TEMPORARY FIX, httpcore is causing issues with celery, removing it as temp fix
python3 -m pip uninstall -y httpcore

Please comment this out python3 -m pip uninstall -y httpcore

it should work, the celery is rasing exception and hasn't started so you are seeing the CPU utilization 100% because infinitely the celery tries to restart, someone test this out and let me know I will push to master then

[sanyam2602]

I tried to remove the httpcore but it is not being getting removed and still getting the same error @yogeshojha

[sanyam2602]

It is still searching for httpcore module. @yogeshojha

[yogeshojha]

@sanyam2602 sorry forgot to mention that you will have to rebuild the image.

make up && make down

should work

[sanyam2602]

Error: Invalid value for '-A' / '--app':
celery-1 | Unable to load celery application.
celery-1 | While trying to load the module reNgine.tasks the following error occurred:
celery-1 | Traceback (most recent call last):
celery-1 | File "/usr/local/lib/python3.10/dist-packages/celery/app/utils.py", line 383, in find_app
celery-1 | sym = symbol_by_name(app, imp=imp)
celery-1 | File "/usr/local/lib/python3.10/dist-packages/kombu/utils/imports.py", line 64, in symbol_by_name
celery-1 | return getattr(module, cls_name) if cls_name else module
celery-1 | AttributeError: module 'reNgine' has no attribute 'tasks'
celery-1 |
celery-1 | During handling of the above exception, another exception occurred:
celery-1 |
celery-1 | Traceback (most recent call last):
celery-1 | File "/usr/local/lib/python3.10/dist-packages/celery/bin/celery.py", line 58, in convert
celery-1 | return find_app(value)
celery-1 | File "/usr/local/lib/python3.10/dist-packages/celery/app/utils.py", line 386, in find_app
celery-1 | sym = imp(app)
celery-1 | File "/usr/local/lib/python3.10/dist-packages/celery/utils/imports.py", line 109, in import_from_cwd
celery-1 | return imp(module, package=package)
celery-1 | File "/usr/lib/python3.10/importlib/init.py", line 126, in import_module
celery-1 | return _bootstrap._gcd_import(name[level:], package, level)
celery-1 | File "", line 1050, in _gcd_import
celery-1 | File "", line 1027, in _find_and_load
celery-1 | File "", line 1006, in _find_and_load_unlocked
celery-1 | File "", line 688, in _load_unlocked
celery-1 | File "", line 883, in exec_module
celery-1 | File "", line 241, in _call_with_frames_removed
celery-1 | File "/usr/src/app/reNgine/tasks.py", line 31, in
celery-1 | from reNgine.llm import *
celery-1 | File "/usr/src/app/reNgine/llm.py", line 6, in
celery-1 | from langchain_community.llms import Ollama
celery-1 | File "/usr/local/lib/python3.10/dist-packages/langchain_community/llms/init.py", line 24, in
celery-1 | from langchain_core.language_models.llms import BaseLLM
celery-1 | File "/usr/local/lib/python3.10/dist-packages/langchain_core/language_models/init.py", line 48, in
celery-1 | from langchain_core.language_models.chat_models import BaseChatModel, SimpleChatModel
celery-1 | File "/usr/local/lib/python3.10/dist-packages/langchain_core/language_models/chat_models.py", line 28, in
celery-1 | from langchain_core.callbacks import (
celery-1 | File "/usr/local/lib/python3.10/dist-packages/langchain_core/callbacks/init.py", line 23, in
celery-1 | from langchain_core.callbacks.manager import (
celery-1 | File "/usr/local/lib/python3.10/dist-packages/langchain_core/callbacks/manager.py", line 29, in
celery-1 | from langsmith.run_helpers import get_run_tree_context
celery-1 | File "/usr/local/lib/python3.10/dist-packages/langsmith/run_helpers.py", line 41, in
celery-1 | from langsmith import client as ls_client
celery-1 | File "/usr/local/lib/python3.10/dist-packages/langsmith/client.py", line 55, in
celery-1 | from langsmith import env as ls_env
celery-1 | File "/usr/local/lib/python3.10/dist-packages/langsmith/env/init.py", line 3, in
celery-1 | from langsmith.env._runtime_env import (
celery-1 | File "/usr/local/lib/python3.10/dist-packages/langsmith/env/_runtime_env.py", line 10, in
celery-1 | from langsmith.utils import get_docker_compose_command
celery-1 | File "/usr/local/lib/python3.10/dist-packages/langsmith/utils.py", line 35, in
celery-1 | import httpx
celery-1 | File "/usr/local/lib/python3.10/dist-packages/httpx/init.py", line 2, in
celery-1 | from ._api import delete, get, head, options, patch, post, put, request, stream
celery-1 | File "/usr/local/lib/python3.10/dist-packages/httpx/_api.py", line 6, in
celery-1 | from ._client import Client
celery-1 | File "/usr/local/lib/python3.10/dist-packages/httpx/_client.py", line 32, in
celery-1 | from ._transports.default import AsyncHTTPTransport, HTTPTransport
celery-1 | File "/usr/local/lib/python3.10/dist-packages/httpx/_transports/default.py", line 32, in
celery-1 | import httpcore
celery-1 | ModuleNotFoundError: No module named 'httpcore'

This is the full error by the way @yogeshojha

[sanyam2602]

I can confirm, I am not getting the following error, and it is been fixed now.

Thanks
Sanyam

[sanyam2602]

Let me open an pull request for this.

[sanyam2602]

#1390

@yogeshojha

Thanks

[drgreenthumb93]

@yogeshojha

The make up && make down worked for me.

[vipul-soman]

I'm intermittently getting a 502 Bad Gateway when I made this change
The web-1 container seems to be restarting again and again

[yogeshojha]

@vipul-soman please attach the logs

[vipul-soman]