Need Linux support

[FreifunkerEZ]

Hi.

I needed Linux support, so I wrote this. There are surely numerous improvements to be made, but this is what I got working and I need to continue with other things.

Maybe you'd like to include this in the repo. I assume no responsibility for broken systems or dreams.

#!/usr/bin/env bash -e
if [ -z "$1" ]
then
	echo "pwned_downloader.sh accepts 1 parameter:
	The range: A single char [0..9A..F]

	All 64k files of the range will be downloaded to a temporary subdir.
	Finally, the temporary files will be combined to the current dir to '\$range.txt'.
	Expect 5GB temporary file-usage and 2.5GB size of the final file.
	An aborted process cannot be resumed.
	However, it will clean up automatically on restarts and after completed runs.
	Expect about 30 minutes per range.
	The use of low latency storage and lots of CPUs will speed the process up.
	Download the whole all 16 ranges with
	    'for x in {0..9} {A..F}; do ./pwned_downloader.sh \$x; done'
	Combine them to a complete dictionary afterwards with
	    'for x in {0..9} {A..F}; do \$x.txt >> pwned-dict.txt; done'
	"
	exit 1
fi

# start
# URLs need to be either upper- or lower-case. We take upper.
export range=`echo -n  $1 | awk '{ print toupper($0) }'`
rm -rf $range
mkdir $range
cd $range

# Create URLs for parallel downloading.
echo Generating 64k URLs for $range into file $range.urls
printf "Example: https://api.pwnedpasswords.com/range/$rangeABCD\n"
for w in {0..9} {A..F}
do
  for x in {0..9} {A..F}
  do
    for y in {0..9} {A..F}
    do
      for z in {0..9} {A..F}
      do
        printf "https://api.pwnedpasswords.com/range/$range$w$x$y$z\n" >> $range.urls
      done
    done
  done
done

# Parallely download all URLs
echo "You might need to install 'parallel'."
cat $range.urls | parallel -j 10 wget -nv {}

# Combine files.
rm $range.urls    # Delete URLs file or it will become part of the dict.
rm ../$range.txt  # Need to start fresh due to >>appending.
for file in `ls`
do
  echo combining $file
  # So, what happens here? The downloaded files contain 35 chars per line.
  # But a pwned-hash is 40 chars. The missing 5 chars come from the file's name.
  # Prefix each line with the file's name before appending it to the output file.
  cat $file | awk -v file="$file" '{print file $0}' >> ../$range.txt
done

# Clean up
cd ..
rm -rf $range

echo Range $range done.
ls -lh $range.txt
head $range.txt
echo "..."
tail $range.txt

[apjone]

I have been able to install and run on linux (ubuntu) using the Microsoft instructions https://learn.microsoft.com/en-gb/dotnet/core/install/linux to install dotnet 6 and dotnet6-sdk on my distribution and then following the instructions in the README.md . The dotnet tool installer will build the binary under ~/.dotnet/tools/haveibeenpwned-downloader so you will need to either move to a directory in your path or add the ~/.dotnet/tools directory to your path.

 ~/.dotnet/tools/haveibeenpwned-downloader -n -o

Hash ranges downloaded ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% 06:25:41 ⢿

[stebet]

Windows, Linux and MacOS are all supported (as long as the OS is supported by .NET 6.0). You just need to install the .NET 6.0 SDK and install the tool according to the instructions here: https://github.com/HaveIBeenPwned/PwnedPasswordsDownloader#installation

The .NET SDK can be downloaded here: https://dotnet.microsoft.com/en-us/download

[muzso]

For people looking for a simple (just curl !) solution for HIBP password hash downloads: #79