-
Notifications
You must be signed in to change notification settings - Fork 0
andrepl/shotgun
Folders and files
Name | Name | Last commit message | Last commit date | |
---|---|---|---|---|
Repository files navigation
shotgun is a unix shell utility which can be used to parallelize long-running processes that operate on data from stdin. Examples: lets say we have a fast-growing log file. we also have a program which reads lines from stdin, does some CPU intensive work, and then inserts them into a database. if we can parallelize the expensive work, we can save a lot of time. thats where shotgun comes in. $ tail -f test.log | shotgun.py -p 4 -c "crunch" in the above example, shotgun launches 4 instances of the 'crunch' program. it then reads lines from the tail -f command and will distribute those lines across the pool of subprocesses, writing to their stdin.
About
unix shell utility to parallelize stream processing.
Resources
Stars
Watchers
Forks
Releases
No releases published
Packages 0
No packages published