Skip to content

andrepl/shotgun

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

2 Commits
 
 
 
 

Repository files navigation

shotgun is a unix shell utility which can be used to parallelize long-running processes that operate on data from stdin.

Examples:
lets say we have a fast-growing log file.  we also have a program which
reads lines from stdin, does some CPU intensive work, and then inserts 
them into a database.  if we can parallelize the expensive work, we can
save a lot of time.  thats where shotgun comes in.

    $ tail -f test.log | shotgun.py -p 4 -c "crunch"

in the above example, shotgun launches 4 instances of the 'crunch'
program. it then reads lines  from the tail -f command and will distribute
those lines across the pool of subprocesses, writing to their stdin.



About

unix shell utility to parallelize stream processing.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages