en-AU.vtt

WEBVTT
Kind: captions
Language: en

00:00:00.001 --> 00:00:01.567
English (AU) (Spoken) [Manually Transcribed Captions]
github.com/WizardTim/WizardTim-captions

00:00:01.580 --> 00:00:08.260
[Alarm Horn]

00:00:08.260 --> 00:00:12.480
Oh no,
no, no!

00:00:12.480 --> 00:00:14.160
Oh my god!

00:00:14.160 --> 00:00:16.160
Please God no!

00:00:16.160 --> 00:00:18.240
Please God no!
Please God no!

00:00:18.720 --> 00:00:20.700
Oh please God help me!

00:00:20.700 --> 00:00:22.160
[Alarm bell]

00:00:22.160 --> 00:00:25.439
So yeah unfortunately I've had one of my
hard drives

00:00:25.439 --> 00:00:29.519
fail on me although thankfully it's not
as bad as that scene will lead you to

00:00:29.519 --> 00:00:33.440
believe
but yeah let's go through what happened

00:00:33.440 --> 00:00:36.640
so
this here is my home server it served me

00:00:36.640 --> 00:00:41.360
pretty well i
made it back in 2017 and here's all the

00:00:41.360 --> 00:00:44.640
parts that are inside it
at the end of the day this thing's

00:00:44.640 --> 00:00:48.960
pretty much just a glorified file server
although it does a couple other things

00:00:48.960 --> 00:00:53.120
like collect data from sensors and stuff
so all this hardware is

00:00:53.120 --> 00:00:57.039
probably a bit excessive but it gets the
job done and is reasonably power

00:00:57.039 --> 00:01:01.359
efficient.
Probably the most important part about

00:01:01.359 --> 00:01:04.400
this
file server is the there's a minimal

00:01:04.400 --> 00:01:09.200
amount of jank
i mean there's a little bit of jank...

00:01:09.200 --> 00:01:14.320
...fair bit of jank, um like this chipset here
i don't know what gigabyte was thinking

00:01:14.320 --> 00:01:19.360
but this is a piss weak heatsink so i
have to glue on some extra heatsinks

00:01:19.360 --> 00:01:25.520
and uh the hard drives in it i mean uh
yeah so it's not like i haven't been

00:01:25.520 --> 00:01:29.439
warned before don't use consumer grade
hard drives in your file server

00:01:29.439 --> 00:01:34.159
especially when it's on 24/7 and doing a
reasonable amount of work

00:01:34.159 --> 00:01:39.119
so yeah i went and bought four consumer
grade hard drives from seagate and they

00:01:39.119 --> 00:01:44.079
rated 2,400 hours a year and 55 TB a year

00:01:44.079 --> 00:01:48.079
and of course over the past three years
i've put about

00:01:48.079 --> 00:01:52.079
27,000 hours on them which is 24/7 as well as

00:01:52.079 --> 00:01:55.600
5.3 PiB of reads [and writes]
yeah i

00:01:55.600 --> 00:02:01.920
don't think my warranty is valid anymore
but i mean hey what's what's the worst that can happen?

00:02:02.120 --> 00:02:05.680
So on the first of every month i

00:02:05.680 --> 00:02:11.280
do a scrub on my zfs array and
i was just minding my own business and

00:02:11.680 --> 00:02:13.680
[Metal Gear Solid 'Alert']

00:02:14.160 --> 00:02:20.080
yeah drive 'A' has 19 read errors
and it's faulted out of the array and i

00:02:20.080 --> 00:02:24.879
don't have any more parody left
fantastic although this isn't too bad

00:02:24.879 --> 00:02:28.080
because
previously i have had similar problems

00:02:28.080 --> 00:02:32.560
with zfs
either due to it correcting a checksum

00:02:32.560 --> 00:02:38.080
uh which probably caused by bit rot or
more serious problems that ended up

00:02:38.080 --> 00:02:40.460
being caused by
vibrations

00:02:40.460 --> 00:02:42.460
{Jack} The vibration, there was a vibration

00:02:42.460 --> 00:02:44.960
probably because i didn't have the hard

00:02:44.960 --> 00:02:48.160
drive screwed in properly and probably
more likely because i

00:02:48.160 --> 00:02:51.280
bought consumer grade hard drives
although

00:02:51.280 --> 00:02:54.959
looking here at the uh historical
monitoring statistics

00:02:54.959 --> 00:02:59.360
we see here that the hard drive has
actually got eight pending sectors

00:02:59.360 --> 00:03:04.159
and this happened a couple days ago
so this uh certainly isn't sort of the

00:03:04.159 --> 00:03:08.400
situation we had before
and if we take a closer look at the uh

00:03:08.400 --> 00:03:12.560
raw smart statistics
we can see here that there's some seek

00:03:12.560 --> 00:03:15.840
error rate problems we have some
uncorrectable errors

00:03:15.840 --> 00:03:21.040
the command timeout is insane we have
some high fly rights which are certainly

00:03:21.040 --> 00:03:26.159
not good
and um yeah these pending sectors

00:03:26.159 --> 00:03:30.480
although
they're not reallocated sectors yet and

00:03:30.480 --> 00:03:34.159
to make doubly sure i took the hard
drive out and put it into my windows

00:03:34.159 --> 00:03:36.959
machine
to check it with CrystalDiskInfo

00:03:36.959 --> 00:03:40.239
because smart statistics are always a
pain to interpret

00:03:40.239 --> 00:03:43.840
and it was telling me pretty much
exactly the same thing hard drive have

00:03:43.840 --> 00:03:47.360
bad day
back up now and that's exactly what i

00:03:47.360 --> 00:03:50.159
did
and rather amusingly you can see here on

00:03:50.159 --> 00:03:54.080
my network activity 
the two days i spent desperately

00:03:54.080 --> 00:03:59.920
transferring data off the server onto
my offsite backup hard drives and after

00:03:59.920 --> 00:04:05.599
those two days once i knew my
data was safe on the 3rd i decided to

00:04:05.599 --> 00:04:10.080
run a zfs scrub to see whether or not
that would fix the problem i don't know

00:04:10.080 --> 00:04:15.680
maybe maybe something weird happened
and uh it corrected 10 checksum errors

00:04:15.680 --> 00:04:19.040
it's quite good it didn't have any more
read errors

00:04:19.040 --> 00:04:23.680
but yeah the number of
pending sectors kind of went up a little

00:04:23.680 --> 00:04:27.840
bit never a good sign
and those smart statistics they they

00:04:27.840 --> 00:04:30.320
weren't looking too high either they're
kind of getting a bit worse

00:04:30.320 --> 00:04:36.080
so um maybe i should uh check
all of the sectors on this hard drive

00:04:36.080 --> 00:04:39.919
properly
and so i ran bad blocks in the

00:04:39.919 --> 00:04:43.759
non-destructive mode
not the default read only the

00:04:43.759 --> 00:04:48.320
non-destructive
read write test and yeah it

00:04:48.320 --> 00:04:54.080
returned 14 read errors
certainly not a good sign but even worse

00:04:54.080 --> 00:05:00.160
the number of pending sectors
skyrocketed to 121 and so after that

00:05:00.160 --> 00:05:04.960
i did a short and a long SMART self test
on the drive and both of them failed

00:05:04.960 --> 00:05:09.440
with read errors
yeah this this hard drive is screwed so

00:05:09.440 --> 00:05:12.960
uh time to go to the local computer
store and buy a new hard drive

00:05:12.960 --> 00:05:20.160
and uh oh, this product is unavailable,
this product has been discontinued,

00:05:20.160 --> 00:05:24.800
this product is no longer available,
brilliant what am i meant to do now i

00:05:24.800 --> 00:05:27.840
i need the
the same hard drive to put in the array

00:05:27.840 --> 00:05:31.600
i can't i can't get a
5,000 rpm drive and whack it in there

00:05:31.600 --> 00:05:35.600
it's it's going to be a big mess
i mean newegg does have it in stock in

00:05:35.600 --> 00:05:40.320
the united states
it's only AU$136 (Dollarydoos) but

00:05:40.320 --> 00:05:45.100
do i trust international shipping of a
hard drive? Noooo.

00:05:45.100 --> 00:05:50.400
So i mean hey let's have a look at
this uh hard drive again it's it doesn't

00:05:50.400 --> 00:05:55.039
sound like it's a rock tumbler
yet and it hasn't actually reallocated

00:05:55.039 --> 00:05:58.080
any sectors it's just marked them as
pending

00:05:58.080 --> 00:06:02.080
one thing that i noticed is that i could
quite perfectly fine

00:06:02.080 --> 00:06:06.720
read the sectors that were supposedly
bad and they didn't give any errors they

00:06:06.720 --> 00:06:10.639
were still there
they still had the data and reading them

00:06:10.639 --> 00:06:13.360
several times didn't reallocate the
sector

00:06:13.360 --> 00:06:19.120
so maybe we can recover it maybe there's
been some sort of weird situation where

00:06:19.120 --> 00:06:23.039
maybe one of the tracks has
had a bit of a vibration{x2} problem and

00:06:23.039 --> 00:06:26.319
gone over a different track or something
weird a little bit and

00:06:26.319 --> 00:06:32.160
i don't know so i decided to completely
reformat the hard drive which oh my god

00:06:32.160 --> 00:06:34.880
that was
that has to be one of the most difficult

00:06:34.880 --> 00:06:39.360
confirmation windows i had to okay
oh oh no i don't don't want to press

00:06:39.360 --> 00:06:43.600
that button oh
oh my data but i i did manage to get

00:06:43.600 --> 00:06:47.919
enough courage to press that button
and uh several hours later after it had

00:06:47.919 --> 00:06:51.120
completely
rewritten the hard drive with zeros all

00:06:51.120 --> 00:06:55.280
of the bad sectors were gone
no more pending sectors and i mean i was

00:06:55.280 --> 00:07:01.039
expecting it to reallocate the sectors
but they were completely gone but hey i

00:07:01.039 --> 00:07:04.720
don't really know
how it handles reformatting it it could

00:07:04.720 --> 00:07:08.880
have been a false flag and the
bad sectors were still there it just

00:07:08.880 --> 00:07:14.160
didn't flag them anymore
so i ran bad blocks again but this time

00:07:14.160 --> 00:07:17.280
in the read-only mode so it would be
faster

00:07:17.280 --> 00:07:21.199
and that returned zero errors which was
quite a surprise

00:07:21.199 --> 00:07:26.000
so evidently in this situation we didn't
have any read errors

00:07:26.000 --> 00:07:30.080
and so i decided to test with the
destructive bad blocks mode

00:07:30.080 --> 00:07:33.520
where it writes four different test
patterns and

00:07:33.520 --> 00:07:39.199
it worked perfectly so that's a surprise
maybe this really was just some sort of

00:07:39.199 --> 00:07:41.440
one-time
random problem that the drive

00:07:41.440 --> 00:07:44.479
encountered and reformatting it has
fixed it

00:07:44.479 --> 00:07:48.240
and throughout all of those tests the
current pending sector count never went

00:07:48.240 --> 00:07:52.000
up
and nor did the reallocated sectors

00:07:52.000 --> 00:07:56.479
so i thought hey let's uh let's put it
back in the array see if it'll work...

00:07:56.479 --> 00:08:02.240
Oh corrupted data! oh oh no.... okay
okay when you put it back in the array

00:08:02.240 --> 00:08:06.319
and you've erased the data
it says corrupted data because there's

00:08:06.319 --> 00:08:09.599
no data there
that the pool's still okay that that

00:08:09.599 --> 00:08:12.960
gave me a bit of a fright when i first
saw that

00:08:12.960 --> 00:08:17.440
but hey eight and a half hours later the
drive was successfully re-silvered

00:08:17.440 --> 00:08:21.440
no errors went over those sectors
perfectly fine

00:08:21.440 --> 00:08:27.120
no problem and i let it do its thing for
five days normal operation and then on

00:08:27.120 --> 00:08:31.120
the 15th i did another scrub
took five and a half hours perfectly

00:08:31.120 --> 00:08:36.159
normal
and again no problem i think we fixed it

00:08:36.159 --> 00:08:40.399
so i think we can safely say that
there's absolutely no consequence to

00:08:40.399 --> 00:08:43.039
buying consumer grade hard drives and
putting it in your

00:08:43.039 --> 00:08:46.640
24/7 server and i mean if you if you've
got

00:08:46.640 --> 00:08:50.000
problems with your pending sectors you
just reformat the hard drive

00:08:50.000 --> 00:08:54.320
everything's good again i think we've
solved it

00:08:54.320 --> 00:09:01.519
Ah come on! Ah why is it gotta read error
it was working fine just before.

00:09:01.519 --> 00:09:04.560
Ah! And the f...
pending sectors are back again although...

00:09:04.560 --> 00:09:10.880
bad this time that's not as bad but
ah it's still screwed yeah um

00:09:10.880 --> 00:09:14.000
yeah maybe i should buy some new hard
drives although and

00:09:14.000 --> 00:09:17.519
i don't know should i import new hard
drive or

00:09:17.519 --> 00:09:20.880
just completely throw pool in the bin
and actually properly buy

00:09:20.880 --> 00:09:24.640
some enterprise grade hard drives i
think i might go that route

00:09:24.640 --> 00:09:28.080
there might be a part two to this video
i don't know when i decide not to be

00:09:28.080 --> 00:09:34.080
lazy and actually fix the problem
yeah but for now don't do what i did

00:09:34.080 --> 00:09:37.600
buy hard drives that are rated for 24/7
operation

00:09:37.600 --> 00:09:42.480
and uh yeah expect them to be
discontinued

00:09:42.480 --> 00:09:46.480
and maybe having two parody drives is
actually a good idea because turns out

00:09:46.480 --> 00:09:50.640
hard drives don't just
magically instantly fail they kinda

00:09:50.640 --> 00:09:52.640
screw around with you for a while.

00:09:53.240 --> 00:10:01.840
So as always thanks for watching