MDADM Superblock Recovery

Question

After a power cycle I found my RAID 5 Array no longer working. I tried various methods to reassemble the array but nothing has worked so far. I believe I need to recreate the superblocks and UUIDs somehow, but was reluctant to barrel into something as to not lose a bunch of data. Thanks for reading.

cat /etc/mdadm/mdadm.conf:

DEVICE partitions
ARRAY /dev/md0 level=raid5 num-devices=4 metadata=0.90 UUID=fd522a0f:2de72d76:f2afdfe9:5e3c9df1
MAILADDR root

Which is normal. It should have 4x2000GB drives (sda, sdc, sde, sdd).

cat /proc/mdstat:

Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : inactive sdd[1](S)
  1953514496 blocks

unused devices: <none>

This is a problem. It only shows one drive in the array and it is also inactive. The array should have sda, sdc, and sde in there as well. When I do a mdadm --examine /dev/sdd everything looks fine. On the other drives examine says no RAID superblock on /dev/sdX.

mdadm --examine --scan:

ARRAY /dev/md0 level=raid5 num-devices=4 metadata=0.90 UUID=fd522a0f:2de72d76:f2afdfe9:5e3c9df1

No help there.

mdadm --assemble --scan -v:

mdadm: looking for devices for /dev/md0
mdadm: no RAID superblock on /dev/sde
mdadm: /dev/sde has wrong uuid.
mdadm: cannot open device /dev/sdd: Device or resource busy
mdadm: /dev/sdd has wrong uuid.
mdadm: no RAID superblock on /dev/sdc
mdadm: /dev/sdc has wrong uuid.
mdadm: cannot open device /dev/sdb5: Device or resource busy
mdadm: /dev/sdb5 has wrong uuid.
mdadm: no RAID superblock on /dev/sdb2
mdadm: /dev/sdb2 has wrong uuid.
mdadm: cannot open device /dev/sdb1: Device or resource busy
mdadm: /dev/sdb1 has wrong uuid.
mdadm: cannot open device /dev/sdb: Device or resource busy
mdadm: /dev/sdb has wrong uuid.
mdadm: no RAID superblock on /dev/sda
mdadm: /dev/sda has wrong uuid.

From this it looks like I have no UUIDs and no Superblocks for sda, sdc, and sde.

sudo fdisk -l

Disk /dev/sda: 2000.4 GB, 2000397852160 bytes
255 heads, 63 sectors/track, 243201 cylinders, total 3907027055 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/sda doesn't contain a valid partition table

Disk /dev/sdb: 250.1 GB, 250058268160 bytes
255 heads, 63 sectors/track, 30401 cylinders, total 488395055 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x353cf669

Device Boot      Start         End      Blocks   Id  System
/dev/sdb1              63   476327249   238163593+  83  Linux
/dev/sdb2       476327250   488392064     6032407+   5  Extended
/dev/sdb5       476327313   488392064     6032376   82  Linux swap / Solaris

Disk /dev/sdc: 2000.4 GB, 2000397852160 bytes
255 heads, 63 sectors/track, 243201 cylinders, total 3907027055 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/sdc doesn't contain a valid partition table

Disk /dev/sdd: 2000.4 GB, 2000398934016 bytes
255 heads, 63 sectors/track, 243201 cylinders, total 3907029168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/sdd doesn't contain a valid partition table

Disk /dev/sde: 2000.4 GB, 2000397852160 bytes
255 heads, 63 sectors/track, 243201 cylinders, total 3907027055 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/sde doesn't contain a valid partition table

So from this it looks like none of my RAID disks have a partition table or UUID. The closest thing I found to my problem was this thread, which suggested running mdadm --create /dev/md0 -v -l 5 -n 4 /dev/sda /dev/sdc /dev/sde /dev/sdd and checking for a valid filesystem with fsck -fn /dev/md0. However, the first command spit out mdadm: no raid-devices specified. I retried the command using sda1, sdc1, etc, but then I get this:

mdadm: layout defaults to left-symmetric
mdadm: chunk size defaults to 512K
mdadm: layout defaults to left-symmetric
mdadm: layout defaults to left-symmetric
mdadm: super1.x cannot open /dev/sda1: No such file or directory
mdadm: ddf: Cannot open /dev/sda1: No such file or directory
mdadm: Cannot open /dev/sda1: No such file or directory
mdadm: device /dev/sda1 not suitable for any style of array

If I do a create and leave sda1 as a "missing" variable in the command then it just says the same thing for sdc1.

I am sure that I am making this more complicated than it needs to be. Can someone with experience please help me? Thanks for your time in advance.

*edit* When i run dumpe2fs /dev/sda i get:

dumpe2fs 1.41.14 (22-Dec-2010)
Filesystem volume name:   <none>
Last mounted on:          <not available>
Filesystem UUID:          bbe6fb91-d37c-414a-8c2b-c76a30b9b5c5
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype     needs_recovery sparse_super large_file
Filesystem flags:         signed_directory_hash 
Default mount options:    (none)
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              366288896
Block count:              1465135872
Reserved block count:     73256793
Free blocks:              568552005
Free inodes:              366066972
First block:              0
Block size:               4096
Fragment size:            4096
Reserved GDT blocks:      674
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         8192
Inode blocks per group:   512
Filesystem created:       Wed Oct 28 12:23:09 2009
Last mount time:          Tue Oct 18 13:59:36 2011
Last write time:          Tue Oct 18 13:59:36 2011
Mount count:              17
Maximum mount count:      26
Last checked:             Fri Oct 14 17:04:16 2011
Check interval:           15552000 (6 months)
Next check after:         Wed Apr 11 17:04:16 2012
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:           256
Required extra isize:     28
Desired extra isize:      28
Journal inode:            8
Default directory hash:   half_md4
Directory Hash Seed:      17e784d8-012e-4a29-9bbd-c312de282588
Journal backup:           inode blocks
Journal superblock magic number invalid!

So stuff is still there. Still researching...

score 6 · Accepted Answer · edited Oct 30 '23 at 06:31

Yikes! What a pickle. let's see if we can get you sorted. Starting with a recap of your disks and partition tables:

sda - no partition table
sdb - sdb1 [Linux] sdb2 [Linux extended] sdb5 [swap]
sdc - no partition table
sdd - no partition table
sde - no partition table

None of these are marked fd Linux raid autodetect, which is the default
You're not using partitions to organize your disk space [0]
You appear to have the entire disk formatted for ext2/3 and are using the entire disk as part of the raidset

The last point is where I think you became undone. The initscripts probably thought you were due for an fsck, sanity checked the volumes, and wiped out the MD superblock in the process. dumpe2fs should return nothing for volumes part of the RAID set.

Take my RAID for example:

root@mark21:/tmp/etc/udev# fdisk -l /dev/sda

Disk /dev/sda: 640.1 GB, 640135028736 bytes
255 heads, 63 sectors/track, 77825 cylinders, total 1250263728 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x0000ffc4
Device Boot      Start         End      Blocks   Id  System
/dev/sda1            2048  1240233983   620115968   fd  Linux raid autodetect
root@mark21:/tmp/etc/udev# dumpe2fs /dev/sda1
dumpe2fs 1.41.14 (22-Dec-2010)
dumpe2fs: Bad magic number in super-block while trying to open /dev/sda
Couldn't find valid filesystem superblock.

That you were able to recreate the RAID set at all is extremely lucky, but that doesn't change the fundamental flaws in your deployment. This will happen again.

What I would recommend is:

Backup everything on that RAID set
Destroy the array and erase the md superblock from each device (man mdadm)
Zero out those disks: dd if=/dev/zero of=/dev/sdX bs=1M count=100
Create partitions on sda, sdc, sdd, & sdf that span 99% of the disk [0]
Tag those partitions as type fd (linux-raid wiki)
Never ever format these partitions with any sort of filesystem
Create a new RAID 5: mdadm --create /dev/md0 -v -f -l 5 -n 4 /dev/sda1 /dev/sdc1 /dev/sdd1 /dev/sde1
Update new UUID in /etc/mdadm.conf
Live happily ever after

I presume from your description that sdb is your system disk, and that's fine. Just make sure you don't accidentally include that in your raid set creation. After this, you should be on the right track and will never encounter this problem again.

[0] I encountered a very nasty fault once on SATA disks that had lots of bad blocks. After using the vendor tool to reconstitute the disk. My once identical set of disks was now unique, the bad drive now had a few blocks less than before the low level format had begun, which of course ruined my partition table and prevented the drive from rejoined the MD RAID set.

Hard drives usually have a "free list" of backup blocks used for just an occasion. My theory is that that list must have been exhausted, and since this wasn't an enterprise disk, instead of failing safe and allowing me the opportunity to send it off for data recovery, it decided to truncate my data and re-size the entire disk in.

Therefore, I never use the entire disk anymore when creating a RAID set, and instead use anywhere from 95-99% of the available free space when creating a partition that would normally span the entire disk. This also gives you some additional flexibility when replacing failed members. For example, not all 250 GB disks have the same amount of free blocks, so if you undershoot the max by a comfortable margin, then you can use almost any disk brand to replace a failed member.

score 1 · Answer 2 · answered Oct 19 '11 at 07:00

1

I've had the same issue before, and I didn't document it (and was a while ago).

I recall something about using e2fsck -b <superblockSector> /dev/sdX and trying backup superblock sectors

you could also take a look at TestDisk

answered Oct 19 '11 at 07:00

Thermionix

411

I will literally paypal you a bounty if you can get this working. I ran dumpe2fs on /sda and got a valid output, but it is too long for this comment. I am adding it to my question. – Teque5 Oct 19 '11 at 15:53
Using $ mdadm --create /dev/md0 -v -f -l 5 -n 4 /dev/sda /dev/sdc /dev/sde /dev/sdd I was able to get the drive active. It is currently resyncing. When i run $ fsck -n /dev/md0 it says superblock invalid, please run $ e2fsck -b 8193 <device>. I plan on doing this once the resync is complete (3 hrs). Did you do this and recover your data? – Teque5 Oct 19 '11 at 20:04
Yep, 8193 rings a bell - although I think I actually played around with it for some time (If I recall I had to try 8192,as it kept telling me 8193 wasn't a superblock, although I think it didn't accept 8192 - maybe I used a greater sector superblock as a few of mine may have been corrupted, all I remember is a lot of messing around with e2fsck - back then I think I had to compile my own later version than what was in the repo's to get it to work ) But I Digress - I did eventually get my data back. Still on mdadm/raid5 - hoping Btrfs is useable in the next 12months or so. – Thermionix Oct 19 '11 at 22:04
Looking at the man page of e2fsck it says 8193 is for 1k block sizes. 2k is a 16384, and 4k is at 32768. I have a 6TB drive so there is no way it is using 1k block sizes. I am pretty sure i want to go with 32768 for a 4k size. How big is your raid? Do you think is is a better idea to write at the 1k block size and work my way up? – Teque5 Oct 19 '11 at 22:59
I tried both the 1k and 4k sizes and it keeps giving the same message: the superblock could not be read or does not describe a correct ext2 filesystem. If the device is valid and it really contains an ext2 filesystem (and not swap or ufs or something else), then the superblock is corrupt, and you might try running e2fsck with an alternate superblock: e2fsck -b 8193 <device> – Teque5 Oct 20 '11 at 00:23
I should use fsck.ext3 instead of e2fsck since my system is ext3 right? – Teque5 Oct 20 '11 at 00:39
Nothing detects the superblocks. I always get the Bad magic number in super-block while trying to open /dev/md0 error. I am now using testdisk to search md0 for superblocks. – Teque5 Oct 20 '11 at 01:21

score 0 · Answer 3 · answered Jun 18 '12 at 10:24

It've been a while since your post, but I will write this:

"mdadm: cannot open device /dev/sdb1: Device or resource busy"

it's good to check

cat /proc/mdstat

I guess your drive is attached to some raid ex. /dev/md126

If so, stop raid

mdadm --stop /dev/md125

and then try to reassemble you raid /dev/md0

mdadm --assemble --verbose --update summaries /dev/md0 /dev/sda3 /dev/sdb3 /dev/sdc3 /dev/sdd3

BUT: More important issue:

DO NOT USE RAID 5 WITH DISK GREATER THAN 1.5 TB !!!

Unrecoverable bit error rate

This is the rate at which a drive will be unable to recover data after application of cyclic redundancy check (CRC) codes and multiple retries. The UBE (Unrecoverable Bit Error) rate is typically specified at 1 bit in 10^15 for enterprise class drives (SCSI, FC, SAS), and 1 bit in 10^14 for desktop class drives (IDE/ATA/PATA, SATA). (so every ~1.7 TB)

So, if one of your drive fail, there is ~55% chance it WON'T rebuild (for UBE 10^-14) Good luck...

more here: http://www.raidtips.com/raid5-ure.aspx

-1 for the "unrecoverable bit error rate" hysteria. All of the articles I've seen on this topic were misinterpreting the manufacturer's numbers as "errors per bits read" whereas they are "errors per sectors read" - which makes a differentiating factor of 2^17 or more than 10^3, making it roughly 1500 exabytes. The common sense should have taught anybody spreading this rumor that with a substantial number of RAID5 arrays are running with drives >> 2 TB nowadays, the word would already have been around much faster than with theoretical calculations based on wrong assumptions. — syneticon-dj, Jun 25 '12 at 16:00

MDADM Superblock Recovery

3 Answers3