Raid inactive, can't assemble because "device is busy"

by Buzut   Last Updated June 15, 2018 12:00 PM

I have a RAID5 on 3 disks. One of the disk presented errors (SMART) so I stopped the server and got it changed. At this point, the array wasn't in a degraded state, all disks were running.

But now, the server won't boot because one of the raid volume can't be assembled. So I booted the server in rescue mode and tried to mount it manually. Two raid volumes are mounted without any problem, but the third one won't because mdadm states /dev/sdc4 is busy - skipping.

Even when I stop already assembled volumes with mdadm --stop and afterwards try to assemble the missing one, I still get the busy error message.

# mdadm --assemble /dev/md125 /dev/sd[bc]4
mdadm: /dev/sdb4 is busy - skipping
mdadm: /dev/sdc4 is busy - skipping

Here are some details:

cat /proc/mdstat

# cat /proc/mdstat 
Personalities : [raid6] [raid5] [raid4] [linear] [multipath] [raid0] [raid1] [raid10] 
md0 : active raid5 sdb1[3] sdc1[2]
      610304 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/2] [_UU]

md1 : active raid5 sdb3[3] sdc3[2]
      20955136 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/2] [_UU]

md125 : inactive sdc4[2](S) sdb4[3](S)
      477220864 blocks super 1.2

unused devices: <none>

mdadm --examine /dev/sd*

# mdadm --examine /dev/sd*
mdadm: No md superblock detected on /dev/sda.
/dev/sdb:
   MBR Magic : aa55
Partition[0] :       612352 sectors at         2048 (type fd)
Partition[1] :      1048576 sectors at       614400 (type 82)
Partition[2] :     20971520 sectors at      1662976 (type fd)
Partition[3] :    477483008 sectors at     22634496 (type fd)
/dev/sdb1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : c59f1ae7:c2db40ca:f23c39c6:b8906d82
           Name : 163-172-108-110:0  (local to host 163-172-108-110)
  Creation Time : Wed Aug 30 10:50:38 2017
     Raid Level : raid5
   Raid Devices : 3

 Avail Dev Size : 610304 (298.05 MiB 312.48 MB)
     Array Size : 610304 (596.10 MiB 624.95 MB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
   Unused Space : before=1960 sectors, after=0 sectors
          State : clean
    Device UUID : f8b8f6d5:af13231e:df576ce8:dc0987ca

    Update Time : Fri Jun 15 13:06:25 2018
  Bad Block Log : 512 entries available at offset 72 sectors
       Checksum : c0af5ca5 - correct
         Events : 107

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 1
   Array State : .AA ('A' == active, '.' == missing, 'R' == replacing)
mdadm: No md superblock detected on /dev/sdb2.
/dev/sdb3:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 82b97a85:9e2f9bf4:0ab3e92f:c2d082f3
           Name : 163-172-108-110:1  (local to host 163-172-108-110)
  Creation Time : Wed Aug 30 10:50:38 2017
     Raid Level : raid5
   Raid Devices : 3

 Avail Dev Size : 20955136 (9.99 GiB 10.73 GB)
     Array Size : 20955136 (19.98 GiB 21.46 GB)
    Data Offset : 16384 sectors
   Super Offset : 8 sectors
   Unused Space : before=16296 sectors, after=0 sectors
          State : clean
    Device UUID : 581e9da2:196f11b2:80814cca:000f1ce2

    Update Time : Fri Jun 15 13:06:25 2018
  Bad Block Log : 512 entries available at offset 72 sectors
       Checksum : c0a68c45 - correct
         Events : 1216

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 1
   Array State : .AA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdb4:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : 3c78263f:6baa0912:caa18c52:b9d80e7c
           Name : 163-172-108-110:2  (local to host 163-172-108-110)
  Creation Time : Wed Aug 30 10:50:38 2017
     Raid Level : raid5
   Raid Devices : 3

 Avail Dev Size : 477220864 (227.56 GiB 244.34 GB)
     Array Size : 477220864 (455.11 GiB 488.67 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
   Unused Space : before=262056 sectors, after=0 sectors
          State : active
    Device UUID : c2328796:4f6255c3:8d6f12b4:083f0dd5

Internal Bitmap : 8 sectors from superblock
    Update Time : Fri Jun 15 10:46:10 2018
  Bad Block Log : 512 entries available at offset 72 sectors
       Checksum : 5b3bc507 - correct
         Events : 10339

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 1
   Array State : AAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdc:
   MBR Magic : aa55
Partition[0] :       612352 sectors at         2048 (type fd)
Partition[1] :      1048576 sectors at       614400 (type 82)
Partition[2] :     20971520 sectors at      1662976 (type fd)
Partition[3] :    477483008 sectors at     22634496 (type fd)
/dev/sdc1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : c59f1ae7:c2db40ca:f23c39c6:b8906d82
           Name : 163-172-108-110:0  (local to host 163-172-108-110)
  Creation Time : Wed Aug 30 10:50:38 2017
     Raid Level : raid5
   Raid Devices : 3

 Avail Dev Size : 610304 (298.05 MiB 312.48 MB)
     Array Size : 610304 (596.10 MiB 624.95 MB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
   Unused Space : before=1960 sectors, after=0 sectors
          State : clean
    Device UUID : 9879accc:f903b32b:1bd6e67a:b1133812

    Update Time : Fri Jun 15 13:06:25 2018
  Bad Block Log : 512 entries available at offset 72 sectors
       Checksum : 9f20959e - correct
         Events : 107

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 2
   Array State : .AA ('A' == active, '.' == missing, 'R' == replacing)
mdadm: No md superblock detected on /dev/sdc2.
/dev/sdc3:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 82b97a85:9e2f9bf4:0ab3e92f:c2d082f3
           Name : 163-172-108-110:1  (local to host 163-172-108-110)
  Creation Time : Wed Aug 30 10:50:38 2017
     Raid Level : raid5
   Raid Devices : 3

 Avail Dev Size : 20955136 (9.99 GiB 10.73 GB)
     Array Size : 20955136 (19.98 GiB 21.46 GB)
    Data Offset : 16384 sectors
   Super Offset : 8 sectors
   Unused Space : before=16296 sectors, after=0 sectors
          State : clean
    Device UUID : 0c9c3d3d:03777e83:3990cfdd:3cc58aee

    Update Time : Fri Jun 15 13:06:25 2018
  Bad Block Log : 512 entries available at offset 72 sectors
       Checksum : 4ca5d6d7 - correct
         Events : 1216

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 2
   Array State : .AA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdc4:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : 3c78263f:6baa0912:caa18c52:b9d80e7c
           Name : 163-172-108-110:2  (local to host 163-172-108-110)
  Creation Time : Wed Aug 30 10:50:38 2017
     Raid Level : raid5
   Raid Devices : 3

 Avail Dev Size : 477220864 (227.56 GiB 244.34 GB)
     Array Size : 477220864 (455.11 GiB 488.67 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
   Unused Space : before=262056 sectors, after=0 sectors
          State : active
    Device UUID : e547f2d6:631ac6b2:f89347d9:bd876af2

Internal Bitmap : 8 sectors from superblock
    Update Time : Fri Jun 15 10:46:10 2018
  Bad Block Log : 512 entries available at offset 72 sectors
       Checksum : cda9ff5d - correct
         Events : 10339

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 2
   Array State : AAA ('A' == active, '.' == missing, 'R' == replacing)
root@163-172-108-110:/var/log# cat /proc/mdstat 
Personalities : [raid6] [raid5] [raid4] [linear] [multipath] [raid0] [raid1] [raid10] 
md0 : active raid5 sdb1[3] sdc1[2]
      610304 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/2] [_UU]

md1 : active raid5 sdb3[3] sdc3[2]
      20955136 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/2] [_UU]

md125 : inactive sdc4[2](S) sdb4[3](S)
      477220864 blocks super 1.2

unused devices: <none>

parted -l

# parted -l
Error: /dev/sda: unrecognised disk label
Model: ATA Samsung SSD 850 (scsi)                                         
Disk /dev/sda: 250GB
Sector size (logical/physical): 512B/512B
Partition Table: unknown
Disk Flags: 

Model: ATA SAMSUNG MZ7LN256 (scsi)
Disk /dev/sdb: 256GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags: 

Number  Start   End     Size    Type     File system  Flags
 1      1049kB  315MB   314MB   primary               boot, raid
 2      315MB   851MB   537MB   primary
 3      851MB   11,6GB  10,7GB  primary               raid
 4      11,6GB  256GB   244GB   primary               raid


Model: ATA SAMSUNG MZ7LN256 (scsi)
Disk /dev/sdc: 256GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags: 

Number  Start   End     Size    Type     File system     Flags
 1      1049kB  315MB   314MB   primary                  boot, raid
 2      315MB   851MB   537MB   primary  linux-swap(v1)
 3      851MB   11,6GB  10,7GB  primary                  raid
 4      11,6GB  256GB   244GB   primary                  raid


Model: Linux Software RAID Array (md)
Disk /dev/md0: 625MB
Sector size (logical/physical): 512B/512B
Partition Table: loop
Disk Flags: 

Number  Start  End    Size   File system  Flags
 1      0,00B  625MB  625MB  ext4


Model: Linux Software RAID Array (md)
Disk /dev/md1: 21,5GB
Sector size (logical/physical): 512B/512B
Partition Table: loop
Disk Flags: 

Number  Start  End     Size    File system  Flags
 1      0,00B  21,5GB  21,5GB  ext4

lsblk

# lsblk
NAME    MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
sda       8:0    0 232,9G  0 disk  
sdb       8:16   0 238,5G  0 disk  
├─sdb1    8:17   0   299M  0 part  
│ └─md0   9:0    0   596M  0 raid5 
├─sdb2    8:18   0   512M  0 part  
├─sdb3    8:19   0    10G  0 part  
│ └─md1   9:1    0    20G  0 raid5 
└─sdb4    8:20   0 227,7G  0 part  
sdc       8:32   0 238,5G  0 disk  
├─sdc1    8:33   0   299M  0 part  
│ └─md0   9:0    0   596M  0 raid5 
├─sdc2    8:34   0   512M  0 part  [SWAP]
├─sdc3    8:35   0    10G  0 part  
│ └─md1   9:1    0    20G  0 raid5 
└─sdc4    8:36   0 227,7G  0 part  
loop0     7:0    0 451,5M  0 loop  /lib/live/mount/rootfs/img.current.squashfs


Answers 1


Finally, after unmounting everything and rebooting the server, I realized that mdadm wouldn't use the new disk because it's not exactly the same size as the other ones: 250GB vs 256GB.

# mdadm --manage /dev/md125 --add /dev/sda4
mdadm: /dev/sda4 not large enough to join array
Buzut
Buzut
June 18, 2018 09:05 AM

Related Questions



linux + How to verify hardware or software RAID?

Updated July 09, 2017 13:00 PM


Breaking RAID 1 on Windows Server

Updated November 05, 2017 05:00 AM