undelete

Oops...

I executed the following command:

./hadoop dfs -rmr .

Everything on the DFS, including the trash seems to be
deleted. Is there 
a way to recover my data?

Thanks,
Mathijs

-- 
Knowlogy
Helperpark 290 C
9723 ZA Groningen

mathijs.hommingaknowlogy.nl
+31 (0)6 15312977
http://www.knowlogy.nl

Mathijs Homminga wrote:
> Oops...
> 
> I executed the following command:
> 
> ./hadoop dfs -rmr .
> 
> Everything on the DFS, including the trash seems to be
deleted. Is there 
> a way to recover my data?

Depending which version of Hadoop you are running ... In
0.12.x it would 
be enough to kill -9 the NameNode ASAP, and remove this
deletion 
operation from the editlog, before it gets re-integrated
with fsimage.

DO MAKE A BACKUP OF THESE FILES FIRST !!!

-- 
Best regards,
Andrzej Bialecki     <><
  ___. ___ ___ ___ _ _   __________________________________
[__ || __|__/|__||/|  Information Retrieval, Semantic Web
___|||__||  |  ||  |  Embedded Unix, System Integration
http://www.sigram.com 
Contact: info at sigram dot com

Thanks,

I stopped the namenode. How can I remove an entry from the
editlog?

FYI: The following action caused the mistake:

I first copied a directory from the DFS to local:
./hadoop dfs -get segments/20070622192310 .

Then, I edited this line (from my command history) to delete
the directory:
./hadoop dfs -rmr segments/20070622192310 .

But forgot the dot at the end... :(

Andrzej Bialecki wrote:
> Mathijs Homminga wrote:
>> Oops...
>>
>> I executed the following command:
>>
>> ./hadoop dfs -rmr .
>>
>> Everything on the DFS, including the trash seems to
be deleted. Is 
>> there a way to recover my data?
>
> Depending which version of Hadoop you are running ...
In 0.12.x it 
> would be enough to kill -9 the NameNode ASAP, and
remove this deletion 
> operation from the editlog, before it gets
re-integrated with fsimage.
>
> DO MAKE A BACKUP OF THESE FILES FIRST !!!
>

-- 
Knowlogy
Helperpark 290 C
9723 ZA Groningen

mathijs.hommingaknowlogy.nl
+31 (0)6 15312977
http://www.knowlogy.nl

Mathijs Homminga wrote:
> Thanks,
> 
> I stopped the namenode. How can I remove an entry from
the editlog?

I wrote a tool for specifically this purpose ;) but it's not
up to date 
anymore - I'm not sure how much hacking is required to make
it work 
again. See HADOOP-915.


-- 
Best regards,
Andrzej Bialecki     <><
  ___. ___ ___ ___ _ _   __________________________________
[__ || __|__/|__||/|  Information Retrieval, Semantic Web
___|||__||  |  ||  |  Embedded Unix, System Integration
http://www.sigram.com 
Contact: info at sigram dot com

Mathijs Homminga wrote:
> Thanks,
> 
> I stopped the namenode. How can I remove an entry from
the editlog?

.. I forgot to add: if you feel adventurous (or desperate)
enough, you 
can use a binary editor and remove this DELETE record from
the file. Be 
sure to carefully read FSEditLog.logDelete() and related
methods to 
figure out where the record starts / ends.


-- 
Best regards,
Andrzej Bialecki     <><
  ___. ___ ___ ___ _ _   __________________________________
[__ || __|__/|__||/|  Information Retrieval, Semantic Web
___|||__||  |  ||  |  Embedded Unix, System Integration
http://www.sigram.com 
Contact: info at sigram dot com

Since Hadoop 0.12, if you configure fs.trash.interval to a
non-zero 
value then 'bin/hadoop dfs -rm' will move things to a trash
directory 
instead of immediately removing them.  The Trash is
periodically emptied 
of older items.  Perhaps we should change the default value
for this to 
60 (one hour)?

Doug

Mathijs Homminga wrote:
> Oops...
> 
> I executed the following command:
> 
> ./hadoop dfs -rmr .
> 
> Everything on the DFS, including the trash seems to be
deleted. Is there 
> a way to recover my data?
> 
> Thanks,
> Mathijs
> 

You saved the day again.

The FSEditLogTool worked like a charm, without modifications

(http
s://issues.apache.org/jira/browse/HADOOP-915).

Here is what I did to perform an undelete of the root
directory on my 
HDFS (hadoop 0.12.2.)

- first, I ran a few tests on another dfs to make sure the
FSEditLogTool 
worked (and I understood what is was doing). Then on the
affected dfs:
- kill the namenode (already did that asap, and to be sure,
I stopped 
all other hadoop daemons)
- run FSEditLogTool -dump ${dfs.name.dir}/edits edits.txt
- remove the last line from edits.txt which contains the
DELETE entry.
- run FSEditLogTool -restore edits.txt
${dfs.name.dir}/edits
- start the namenode and datanodes again
- I had to turn off save-mode manually:
    $ bin/hadoop dfsadmin -safemode leave

o/

Andrzej Bialecki wrote:
> Mathijs Homminga wrote:
>> Thanks,
>>
>> I stopped the namenode. How can I remove an entry
from the editlog?
>
> I wrote a tool for specifically this purpose ;) but
it's not up to 
> date anymore - I'm not sure how much hacking is
required to make it 
> work again. See HADOOP-915.
>
>

-- 
Knowlogy
Helperpark 290 C
9723 ZA Groningen

mathijs.hommingaknowlogy.nl
+31 (0)6 15312977
http://www.knowlogy.nl

+1

Setting this property to a non-zero value by default will
give less 
experienced Hadoop users (like me ) a safety
net.


Doug Cutting wrote:
> Since Hadoop 0.12, if you configure fs.trash.interval
to a non-zero 
> value then 'bin/hadoop dfs -rm' will move things to a
trash directory 
> instead of immediately removing them.  The Trash is
periodically 
> emptied of older items.  Perhaps we should change the
default value 
> for this to 60 (one hour)?
>
> Doug
>
> Mathijs Homminga wrote:
>> Oops...
>>
>> I executed the following command:
>>
>> ./hadoop dfs -rmr .
>>
>> Everything on the DFS, including the trash seems to
be deleted. Is 
>> there a way to recover my data?
>>
>> Thanks,
>> Mathijs
>>

-- 
Knowlogy
Helperpark 290 C
9723 ZA Groningen

mathijs.hommingaknowlogy.nl
+31 (0)6 15312977
http://www.knowlogy.nl

Mathijs Homminga wrote:
> You saved the day again.
> 
> The FSEditLogTool worked like a charm, without
modifications 
> (http
s://issues.apache.org/jira/browse/HADOOP-915).

That's great, I was afraid it was out of sync with your
version of Hadoop.

> 
> Here is what I did to perform an undelete of the root
directory on my 
> HDFS (hadoop 0.12.2.)
> 
> - first, I ran a few tests on another dfs to make sure
the FSEditLogTool 
> worked (and I understood what is was doing). Then on
the affected dfs:
> - kill the namenode (already did that asap, and to be
sure, I stopped 
> all other hadoop daemons)
> - run FSEditLogTool -dump ${dfs.name.dir}/edits
edits.txt
> - remove the last line from edits.txt which contains
the DELETE entry.
> - run FSEditLogTool -restore edits.txt
${dfs.name.dir}/edits
> - start the namenode and datanodes again
> - I had to turn off save-mode manually:
>    $ bin/hadoop dfsadmin -safemode leave

Just for the record - this procedure should be your
absolutely last 
resort, because it's tricky and error prone. You can easily
corrupt your 
filesystem beyond any hope.

-- 
Best regards,
Andrzej Bialecki     <><
  ___. ___ ___ ___ _ _   __________________________________
[__ || __|__/|__||/|  Information Retrieval, Semantic Web
___|||__||  |  ||  |  Embedded Unix, System Integration
http://www.sigram.com 
Contact: info at sigram dot com