-
Notifications
You must be signed in to change notification settings - Fork 26
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Does elaster use bulk indexing? #4
Comments
I managed to continue the import even with errors by commenting out the |
You might decide to introduce a special flag, smth like As for bulk, its not supported ATM, but something nice to have.
|
Thanks for replying. I will try to implement the bulk thing and send you a pull request. Its taking very long to import.. it took me around 5 hrs for 8 million documents.. I think it can be faster. |
I actually was thinking about it for quite long time ago. But, since I never really needed, I had no time to make it finally happen. Especially for those purposes, I've added this to |
I am just about to start implementing the bulk thing. I want a config option (with default 1000) for the number of docs they can index in a single bulk operation and tada.. I'm bit of a node js newb.. So I will have to hack my way through promises/streams etc.. If you have any suggestions please suggest.. Thank you. |
Sure. As I said, I see |
so this code block
changes to
and then update the postToElastic function to handle the |
I'm not sure that would be that straight forward.. but the direction is right. |
So cool.. All I have to do is this..
|
You probably also need to change Since project don't have any unit tests, please make sure it's working by manual testing. Ideally, you should test with and without batching and have a benchmarks, so we could really see what's the performance boost with |
Hello, Here are the updates I made. syllogismos@aa5d31a And it is not working.. properly. Here are the things I did.
If I do above, each item is not passing though the
and then changed the stream like below..
I have no idea how dumb what I did is, but wanted to check if it works or not.. 💃
|
So my current implementation just stops after indexing the first batch.. and it just waits.. |
https://github.com/syllogismos/elaster/blob/bugs-bulk/source/elaster.js fixed small issue.. |
Does elaster use bulk indexing? or indexes each document separately?
Also will it be possible to ignore errors? Because of some rogue documents with wrong mappings the entire export gets cancelled.
The text was updated successfully, but these errors were encountered: