markdown text splitter

[pelikhan]

---

Summary of Changes

• ✂️ Markdown Chunking Utility Added: Introduced a new chunkMarkdown function to split markdown files into manageable chunks without breaking the heading structure. This supports efficient token-based chunking for large markdown files.

• 🧪 Comprehensive Test Coverage: Added a robust test suite (mdchunk.test.ts) to validate the chunkMarkdown functionality, including edge cases like empty markdown, nested headings, large sections, and backtracking logic.

• 🚀 Global Utility Enhancement: Extended the installGlobals method to include the chunk method under the global MD interface, enabling markdown chunking with optional parameters like maxTokens and model.

• 📄 New Sample Script: Added a sample script (mdchunk.genai.mjs) demonstrating the usage of the chunk method to process markdown files with token limits.

• 🛠️ Integration with File and Token Utilities: Incorporated resolveFileContent and resolveTokenEncoder to handle file content resolution and token encoding seamlessly during markdown chunking.

• 📝 Type Definition Update: Updated the MD interface in prompt_template.d.ts to include the new chunk method, ensuring proper type support for markdown chunking.

These changes enhance the system's ability to handle large markdown files efficiently while maintaining structural integrity, improving both flexibility and usability.

AI-generated content by prd may be incorrect