Add ingest folders with symlink

[spaceymonk]

Hi, I wanted to propose a simple change in ingest script.

Reason behind it: I store my documents in a separate storage and instead of copying files I symlink them to SOURCE_DOCUMENTS folder. It works when I link documents one by one (i.e. giving full path for each file). But I have documents in nested folders and linking directories not work due to followlinks flag defaults to False. Documentation

The PR solves this problem.

[NitkarshChourasia]

Perfect.
But have you tested it?

[NitkarshChourasia]

What do you mean by separate storage?

Can you specify further?

[NitkarshChourasia]

Hi, I wanted to propose a simple change in ingest script.

Reason behind it: I store my documents in a separate storage and instead of copying files I symlink them to SOURCE_DOCUMENTS folder. It works when I link documents one by one (i.e. giving full path for each file). But I have documents in nested folders and linking directories not work due to followlinks flag defaults to False. Documentation

The PR solves this problem.

Do you mean by different hard-disk/SSD? or something?

[NitkarshChourasia]

@spaceymonk just for clarification I am asking.

[spaceymonk]

• I've tested it, of course.
• By storage I meant another directory in the filesystem that I can symlink to under SOURCE_DOCUMENTS folder. For example, I store my documents under ~/Documents path and if I wanted to run localGPT on my documents, I have to copy/move al the files into SOURCE_DOCUMENTS path.

With this change I just run ln -s ~/Documents/ Documents and it automatically detects and ingests all the files under ~/Documents/.

There might be occur one problem, as the documentation stated, if you created a loop in your path, i.e. symlinking parent directory in child directory, it may run into infinite loop due to lack of storing of visited paths in Python.

[NitkarshChourasia]

Okay.

…

On Sat 6 Apr, 2024, 11:08 AM Berktuğ Kaan Özkan, ***@***.***> wrote: - I've tested it, of course. - By storage I meant another directory in the filesystem that I can symlink to under SOURCE_DOCUMENTS folder. For example, I store my documents under ~/Documents path and if I wanted to run *localGPT* on my documents, I have to copy/move al the files into SOURCE_DOCUMENTS path. With this change I just run ln -s ~/Documents/ Documents and it automatically detects and ingests all the files under ~/Documents/. There might be occur one problem, as the documentation stated, if you created a loop in your path, i.e. symlinking parent directory in child directory, it may run into infinite loop due to lack of storing of visited paths in Python. — Reply to this email directly, view it on GitHub <#748 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AVTTXYC5I3W4U2WONCL43X3Y36C5ZAVCNFSM6AAAAABDKC7SYKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANBQHE3TMOJUGQ> . You are receiving this because you commented.Message ID: ***@***.***>

[spaceymonk]

Any updates ?