Hi @Wendong-Fan,

I'm interested in implementing the SemHash feature. Before proceeding, I have some questions:

1. Implementation approach:

   • Should we implement SemHash from scratch?
   • Or should we integrate the existing MinishLab/semhash library?

2. If we use the existing library:

   • Are there any licensing considerations since CAMEL uses Apache 2.0?
   • Any specific requirements for third-party dependencies?

3. Initial thoughts on integration:

   • Plan to add it in camel/utils/dedup.py
   • Will support both single and multi-dataset deduplication
   • Will maintain CAMEL's existing code style and documentation standards

Looking forward to your guidance on these points before starting the implementation.

Hi @User235514 ,

Thank you so much for your willingness to contribute!

1. We can integrate the existing MinishLab/semhash library.
2. I believe there are no licensing issues, but could @ Kaiming Hu double-check this? We use Poetry to manage dependencies, so you can add the new dependency to the pyproject.toml file.
3. The plan looks solid to me.

Feel free to reach out if you have any questions during the integration process!

hi @User235514 there is no licensing issues they use MIT license as far as I am concerned https://github.com/MinishLab/semhash/blob/main/LICENSE but you do need to include the copyright notice somewhere in our folder. @Wendong-Fan can further advise on this.

When trying to add semhash package using poetry add semhash@^0.2.0, the dependency resolution gets stuck for a very long time (>1000s) at downloading antlr4-python3-runtime.

Attempted Solutions

1. Used Tsinghua mirror with POETRY_SOURCE="https://pypi.tuna.tsinghua.edu.cn/simple"
2. Cleared poetry cache using poetry cache clear . --all

Neither of these solutions resolved the issue. The process still gets stuck at downloading antlr4-python3-runtime.

System information

• Python version: 3.11.11
• Poetry version: 2.0.1

Steps to reproduce

1. Clone and install the CAMEL repository
2. Run poetry add semhash@^0.2.0
3. The process gets stuck at resolving dependencies

Additional context

• I tried using Tsinghua mirror with POETRY_SOURCE="https://pypi.tuna.tsinghua.edu.cn/simple", but the issue persists
• The process gets stuck specifically at downloading antlr4-python3-runtime
• Total resolution time exceeds a few hours without completing

Hey @User235514 , it took me 135.8s to add the dependency, could you try with poetry add semhash?

Hi @Wendong-Fan. Thank you for your response. I just tried again with poetry add semhash, but I'm still experiencing the same issue as before. The process gets stuck at "Resolving dependencies..." for an extended period (>1000s). However, I noticed that this time it seems to be stuck at a different point in the resolution process compared to my previous attempt.

The overall behavior remains consistent with my previous report - even with a simpler command, the dependency resolution is still not completing in a reasonable timeframe.

Have we encountered similar cases before where users experienced significantly longer dependency resolution times? If so, what were the common causes and solutions?

Would you have any additional suggestions for troubleshooting this issue? Perhaps there are some verbose logging options we could enable to better understand where exactly the process is getting stuck?

Hey @User235514 , Thank you for your detailed update. Could you try waiting a bit longer to see if the resolution eventually completes? This delay might be due to it being the first time you're running the command, which can sometimes take longer as dependencies are resolved and cached

You can also try updating your poetry version by poetry self update