I have done a fuck up again (mkfs.fat on an ext4 partition)

foxone · July 2020

Hello,
while setting up my backup system (oh, the irony), I accidentally launched a mkfs.fat on my ext4 hard drive.

Of course, since i was setting up the backup system, I didn't have any backup.

I desperately ^C but it was too late and the filesystem was gone, so now i'm trying to recover the data.
Photorec of course can see everything, but with random file names. I'd be nice to be able to recover it with the correct file names, so that the files would actually have context (most are source codes and stuff referenced via file name).

I already made an image of the disk, and I'm working on that.
Using testdisk i can see partitions with the original partition name:

However, whatever partition i try to open gives me no files:

I also found online some info on how to recover the partition using backup superblocks, but all the offset i tried don't seem to work (doing the dry-run fsck gives me a superblock error).

At this point i have no idea what to do, besides using photorec (which i would like to not resort to).

InceptionHosting · July 2020

A simple method would be to boot sysrescuecd, spin up gparted and use that to do recovery.

Don't forget to burn $7 before you start for the lowend gods.

Side note: When you up fuck you really up fuck dont you

foxone · July 2020

@AnthonySmith said:
A simple method would be to boot sysrescuecd, spin up gparted and use that to do recovery.

Why gparted? How would i use it? Last time i used it, that was just a partitioning tool.
Or does sysrescuecd have something extra?

Side note: When you up fuck you really up fuck dont you

Fuck me

InceptionHosting · July 2020

@foxone said: Why gparted? How would i use it? Last time i used it, that was just a partitioning tool.

Or does sysrescuecd have something extra?

gparted has whole partition recovery tools, sysrescuecd because its easy to find and you can get additional functionality to check stuff.

nullnothere · July 2020

@foxone said: I accidentally launched a mkfs.fat on my ext4 hard drive.

Depending on how quickly you killed the mkfs.fat you may still be able to use some of the (hopefully) not-overwritten superblocks of the (original) ext4 FS and recover your files metadata.

See: https://help.ubuntu.com/community/DataRecovery (look at TestDisk and see if you can quickly undo some of the damage).

Specifically look at the -S option of mkfs.ext4 and see if that also helps you along somewhat. Now that you are working off an (disposable) image file you can boldly experiment.

foxone · July 2020

I have tried to recover using a superblock backup, but the filesystem looks so corrupted that at the end i just get an empty partition.

I used this software: https://gist.github.com/countingpine/0fd04b99058ebc910fefc910606ae6d0 to find for superblocks on the partition, then i used fsck.ext4 -n -v -b xxxx -B 4096 /dev/sda to try to recover the filesystem. It gave me a ton of errors and corrected the filesystem.

What's strange is that when i try to mount the disk, this shows:

mount: sda: more filesystems detected on /dev/sda; use -t <type> or wipefs(8).

Specifying to mount the ext4 partition -t ext4 makes it work, but i get a completely empty partition with the name of the original partition.

At this point i have no idea what i should do

MaxKVM · July 2020

@foxone

The original photo shows the error on /dev/sdb. Were the commands run on the proper device?

foxone · July 2020

@MaxKVM said:
@foxone

The original photo shows the error on /dev/sdb. Were the commands run on the proper device?

Yes! It's changing between sda and sdb since when i disconnect and reconnect it quickly (it's a clone of the original hdd in a usb caddy) it shows up as /dev/sdb.

foxone · July 2020

So, i've tried to recover from a backup superblock after remaking the fs table using testdisk, and after a shitton of errors, the partition is completely empty.

FOXO1TB: ***** FILE SYSTEM WAS MODIFIED *****
FOXO1TB: 297/61054976 files (12.5% non-contiguous), 3937795/244190208 blocks

Should i just lose hope?

ehab · July 2020

i recommend you continue to study and gain experience. Maybe i'll ask you when i run into disk crap.

AlwaysSkint · July 2020

Reclone and do a testdisc deep scan?

foxone · July 2020

@AlwaysSkint said:
Reclone and do a testdisc deep scan?

Testdisk deep scan finds the partitions but they are unrecoverable

InceptionHosting · July 2020

Well if all else fails, this probably will too but might be worth at least trying extundelete again it is available on the sysrescuecd.

It won't recover the partition but maybe some of the files.

comi · July 2020

Superblock recovery in theory should work, other than that you might want to check what fsck says about the journal, that may yield some hints.

In theory you have everything you need:
filenames are basically data on disc, they are maps ("filename", inode_number)
knowing inode_number you can get the inode itself as long as you have the superblock
if you find inode, you find the data.

You can see the details here:
https://www.kernel.org/doc/html/latest/filesystems/ext4/dynamic.html#finding-an-inode

Automatic tools do that already, so if they fail there must be something else going on, like a problem with journal...

foxone · July 2020

@comi said:
Superblock recovery in theory should work, other than that you might want to check what fsck says about the journal, that may yield some hints.

In theory you have everything you need:
filenames are basically data on disc, they are maps ("filename", inode_number)
knowing inode_number you can get the inode itself as long as you have the superblock
if you find inode, you find the data.

You can see the details here:
https://www.kernel.org/doc/html/latest/filesystems/ext4/dynamic.html#finding-an-inode

Automatic tools do that already, so if they fail there must be something else going on, like a problem with journal...

If the journal is broken, what does that cause? There might be a problem with it indeed since the hard driwe was turned off suddently.

comi · July 2020

@foxone said:

@comi said:
Superblock recovery in theory should work, other than that you might want to check what fsck says about the journal, that may yield some hints.

In theory you have everything you need:
filenames are basically data on disc, they are maps ("filename", inode_number)
knowing inode_number you can get the inode itself as long as you have the superblock
if you find inode, you find the data.

You can see the details here:
https://www.kernel.org/doc/html/latest/filesystems/ext4/dynamic.html#finding-an-inode

Automatic tools do that already, so if they fail there must be something else going on, like a problem with journal...

If the journal is broken, what does that cause? There might be a problem with it indeed since the hard driwe was turned off suddently.

if you mounted it read/write there is new journal already, so the question is whether the journal was broken. If it was and some of the metadata was corrupted it is as good as lost.

Did fsck or dmesg sad anything about it?
Did fsck or dmesg said anything else useful?
Did you make a dd copy before trying to restore?
Is it "completely" empty? No lost+found?
Did you try the -S?

foxone · July 2020

@comi said:

if you mounted it read/write there is new journal already, so the question is whether the journal was broken. If it was and some of the metadata was corrupted it is as good as lost.

Did fsck or dmesg sad anything about it?
Did fsck or dmesg said anything else useful?

Just that the superblock is not valid

Did you make a dd copy before trying to restore?

Of course, i'm working on that

Is it "completely" empty? No lost+found?

Yes.

Did you try the -S?

Yes

Anyway, i'm now scanning it with R-Linux. It looks like pretty much every software is fooled by the fact that it was a GPT reformatted as MBR with vfat - so they all find the fat partition and say "hey that's working, what's the issue then?". I'll try later to zero a couple of megabytes at the beginning to destroy the partition table and fat partition definitely.

It could find some superblocks and directories (along with real file names) correctly. So i guess the data is still there - i just need a way to find it.

nullnothere · July 2020

@foxone said: I'll try later to zero a couple of megabytes at the beginning to destroy the partition table and fat partition definitely.

If you have any options try to guide/force to look for an EXT4 FS (and related metadata). Depending on how much got written by the mkfs.fat, you still have the data blocks and possibly a fair number of the super blocks from which you can try to recover the data.

And of course you're working with a disposable image - so that's really comforting to try multiple ways without loosing anything (more).

There will be a fair bit of trial and error required (unless you get lucky) and even if you do get the actual file contents, figuring out the file name/metadata will likely be tricky. If the above diagram seems reasonably accurate, then you probably can recover most of the data even if painfully/manually.

foxone · July 2020

@nullnothere said:

@foxone said: I'll try later to zero a couple of megabytes at the beginning to destroy the partition table and fat partition definitely.

If you have any options try to guide/force to look for an EXT4 FS (and related metadata). Depending on how much got written by the mkfs.fat, you still have the data blocks and possibly a fair number of the super blocks from which you can try to recover the data.

And of course you're working with a disposable image - so that's really comforting to try multiple ways without loosing anything (more).

There will be a fair bit of trial and error required (unless you get lucky) and even if you do get the actual file contents, figuring out the file name/metadata will likely be tricky. If the above diagram seems reasonably accurate, then you probably can recover most of the data even if painfully/manually.

The scan is still ongoing. I will report once it's finished. But if it can recover the files I might as well just buy the license for 79$ and call it a day. (free version has a 256kb limit per file)

comi · July 2020

@foxone said: the fact that it was a GPT reformatted as MBR with vfat

mkfs does not touch partition table, how do you know it's a fact?

I have recreated your situation:

# fallocate -l 10G wdtfs.img
# losetup /dev/loop1 wdtfs.img 
# fdisk /dev/loop1
# mkfs.ext4 /dev/loop1p1
# mount /dev/loop1p1 /mnt/
# mkdir -p {a..c}/{a..c}
# touch {a..c}/{a..c}/{a..c}
# umount /mnt
# cp wdtfs.img wdtfs1.img
# mkfs.fat /dev/loop1p1
# mke2fs -n /dev/loop1p1
# e2fsck -b 819200 /dev/loop1p1
# mount -t ext4 /dev/loop1p1 /mnt
# ll /mnt

  total 12K
  drwxr-xr-x  3 root root 4.0K Jul 14 13:35 .
  drwxr-xr-x 19 root root 4.0K Apr  8 01:37 ..
  drwx------  5 root root 4.0K Jul 14 13:35 lost+found

# ll /mnt/lost+found/

  total 20K
  drwxr-xr-x 5 root root 4.0K Jul 14 13:15 '#131073'
  drwxr-xr-x 5 root root 4.0K Jul 14 13:15 '#393217'
  drwxr-xr-x 5 root root 4.0K Jul 14 13:15 '#524289'
  drwx------ 5 root root 4.0K Jul 14 13:35  .
  drwxr-xr-x 3 root root 4.0K Jul 14 13:35  ..

# ll /mnt/lost+found/#131073/

  total 20K
  drwxr-xr-x 5 root root 4.0K Jul 14 13:15 .
  drwx------ 5 root root 4.0K Jul 14 13:35 ..
  drwxr-xr-x 2 root root 4.0K Jul 14 13:16 a
  drwxr-xr-x 2 root root 4.0K Jul 14 13:16 b
  drwxr-xr-x 2 root root 4.0K Jul 14 13:16 c

I have lost filesystem root directory but everything else is intact.
Maybe even it is still there somewhere, but at this point it would be easier to manually identify it's contents unless you don't have hundreds of entries right at the top level.
If you are not getting the same then either there is something else going on or you are not executing it right.

foxone · July 2020

Then i guess something else is going on and i have no idea what. The exact command i used (got it from my history) was:

526 sudo mkfs.fat /dev/sda

This is a sector that looks like an ext4 superblock.

However, when trying to use it...

Am i doing something wrong? Am i not supposed to use sector numbers?

comi · July 2020

first, no, you are not supposed to use sector numbers

mke2fs -n /dev/loop1p1
mke2fs 1.45.6 (20-Mar-2020)
/dev/loop1p1 contains a vfat file system
Proceed anyway? (y,N) y
Creating filesystem with 2621179 4k blocks and 655360 inodes
Filesystem UUID: a815893f-b3f2-4491-b826-75d426613cc2
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632

you need to use one of these

second, why e2fsck -n? By default you should agree with fsck unless you know better

foxone · July 2020

@comi said:
first, no, you are not supposed to use sector numbers
mke2fs -n /dev/loop1p1
mke2fs 1.45.6 (20-Mar-2020)
/dev/loop1p1 contains a vfat file system
Proceed anyway? (y,N) y
Creating filesystem with 2621179 4k blocks and 655360 inodes
Filesystem UUID: a815893f-b3f2-4491-b826-75d426613cc2
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632
you need to use one of these

second, why e2fsck -n? By default you should agree with fsck unless you know better

Because i don't want to touch the image yet, i just want to check if it recognizes the superblock or not.

comi · July 2020

@foxone said:
@comi said:
first, no, you are not supposed to use sector numbers
mke2fs -n /dev/loop1p1
mke2fs 1.45.6 (20-Mar-2020)
/dev/loop1p1 contains a vfat file system
Proceed anyway? (y,N) y
Creating filesystem with 2621179 4k blocks and 655360 inodes
Filesystem UUID: a815893f-b3f2-4491-b826-75d426613cc2
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632
you need to use one of these

second, why e2fsck -n? By default you should agree with fsck unless you know better
Because i don't want to touch the image yet, i just want to check if it recognizes the superblock or not.

ah, oki oki

foxone · July 2020

Halfway through the disk these are the browsable partitions

comi · July 2020

526 sudo mkfs.fat /dev/sda

maan, I'm so sorry, I completely missed that you created filesystem over the entire disk not the partition - I completely ignored that

yup, you destroyed or damaged the partition table.

In theory if you can create new partition table exactly the same as it was before you can recover every partition one by one.

More specificaly same number of partitions in the same order of the exact same size.

foxone · July 2020

@comi said:

526 sudo mkfs.fat /dev/sda

maan, I'm so sorry, I completely missed that you created filesystem over the entire disk not the partition - I completely ignored that

yup, you destroyed or damaged the partition table.

In theory if you can create new partition table exactly the same as it was before you can recover every partition one by one.

More specificaly same number of partitions in the same order of the exact same size.

Sadly i have no idea how the old partition was made. This disk was my main disk (thus had /, /boot, swap and /home partitions), then i moved it to be a data disk (deleting /boot and /, but keeping /home and enlarging it to occupy the residual space), then i finally got rid of the swap partition and thus enlarged the partition to the full disk size. However, i have no idea if it was actually full disk or a couple of megs in front of it or what.

R-Linux is finding a lot of inodes and master blocks, but they all seem from android images i had stored on the drive, or the old / and /boot partitions. It looks like the original 1tb partition disappeared out of nowhere for some reason...

Ympker · July 2020

@foxone How many furries were lost?

foxone · July 2020

@Ympker said:
@foxone How many furries were lost?

None. I funnily keep those on a couple 12tb drives in my server

yoursunny · July 2020

@comi said:
In theory if you can create new partition table exactly the same as it was before you can recover every partition one by one.

More specificaly same number of partitions in the same order of the exact same size.

So the lesson is that we should email ourselves the partition table, so that we can recover data even if some files are not yet in the backup.

foxone · July 2020

@yoursunny said:

@comi said:
In theory if you can create new partition table exactly the same as it was before you can recover every partition one by one.

More specificaly same number of partitions in the same order of the exact same size.

So the lesson is that we should email ourselves the partition table, so that we can recover data even if some files are not yet in the backup.

This so much. I've just done a dd of 1gb of all my drives to keep safe just in case.

And btw, i still haven't recovered my data. I just removed the drive and i'll keep it there until i find time and will to try more.

I have done a fuck up again (mkfs.fat on an ext4 partition)

Comments