QNAP RAID Expansion Christian Donner, July 8, 2013October 11, 2014 Expanding a RAID array on a QNAP NAS device is documented in detail in the user manual, but alas, the many desperate calls for help on the QNAP forums are evidence that this process does not always work smoothly. So I was not surprised when I was unsuccessful with my upgrade. I replaced 4x2TB disks with 4x3TB disks because I had run out of space in my RAID 5 array. This is exactly the scenario I was working with, except for the disk sizes (image taken from the QNAP documentation): 9 steps Steps 1 through 6 worked as expected. I was able to replace and rebuild each drive through the Web interface as described in the manual. Then I had a RAID 5 array on the new 3TB disks, but still with the old size (5469 GB instead of 8248 GB). Steps 7 and 8 are meant to expand the RAID array so that the newly added space is put to use. The Expand process never completed, though. After 24 hours it would still say Expanding… (50%). So I contacted support. What followed was a replay of all the recipes that I had previously seen on the forums. But because there was no single set of instructions for the whole process, and because there were some discrepancies here and there between what my support engineer told me and what other people did, and because I have a record of everything I did and I know it worked, I decided to publish my own version. The short version is that the UI stumbles on two potential issues, and it does not tell you. Issue number one is that it may not be able to stop all the services that access the volume, and subsequently the volume cannot be unmounted. I had a lot of things running, including Surveillance Station, EyeOS, and TigerCRM. I don’t think that QNAP tests their script with all the potential packages that people may have installed on their device. The second issue is that there is not enough swap memory. This needs to be addressed. As a result, the only way to complete the expansion successfully is via command shell. Here is what you need to do – assuming that all disks have been replaced and rebuilt, i.e. you have successfully completed steps 1 through 6 in the documentation. 1. Reboot the NAS This is a good idea, because you may have the device in some undefined state, based on prior attempts of addressing the issue. 2. Stop the services Support told me to run the following commands: #/etc/init.d/services.sh stop #umount /dev/md0 (NOTE: do not proceed if md0 cannot be unmounted!) #e2fsck -f /dev/md0 #resize2fs /dev/md0 #storage_boot_init 2 #/etc/init.d/services.sh start So I did: [~] # /etc/init.d/services.sh stop Stop qpkg service: Disable Optware/ipkg kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec] kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec] Remove link /mnt/ext/opt/videostation Create disable link to /mnt/ext/opt/videostation Remove link /share/MD0_DATA/Web/videostation Create disable link to /share/MD0_DATA/Web/videostation . Stop service: cloud3p.sh vpn_openvpn.sh vpn_pptp.sh ldap_server.sh antivirus.sh iso_mount.sh qsyncman.sh rsyslog.sh snmp nvrd.sh lunportman.sh iscsitrgt.sh smb.sh nfs crond.sh Qthttpd.sh twonkymedia.sh init_iTune.sh ImRd.sh StartMediaService.sh bt_scheduler.sh btd.sh ftp.sh atalk.sh mysqld.sh recycled.sh . [~] # umount /dev/md0 umount: /share/MD0_DATA: device is busy umount: /share/MD0_DATA: device is busy As you can see, umount did not complete. At this point support was of no help. Quote: Please try to reboot the NAS and then log right back into SSH on the QNAP and run the commands again to see if it can stop the process. It could not. I had to manually stop things, first every application and service that had a checkbox in the web interface, then by uninstalling TigerCRM and EyeOS that I was not using anyway. After several iterations of trial and error, the umount finally worked. I cannot say what had initially caused it to fail. 3. Run file system check (e2fsck) On to the next step: [~] # e2fsck -f /dev/md0 e2fsck 1.41.4 (27-Jan-2009) Pass 1: Checking inodes, blocks, and sizes Pass 2: Checking directory structure Pass 3: Checking directory connectivity Pass 4: Checking reference counts Pass 5: Checking group summary information /dev/md0: 1979044/365993984 files (3.0% non-contiguous), 1352060256/1463959200 blocks [~] # [~] # resize2fs /dev/md0 resize2fs 1.41.4 (27-Jan-2009) Resizing the filesystem on /dev/md0 to 2196523200 (4k) blocks. resize2fs: Memory allocation failed while trying to resize /dev/md0 [~] # 4. If it fails, add swap space I had seen this issue in other forum posts, but I did not have to add a storage device to address it. Support gave me the following procedure to follow that I copied and pasted ad verbatim (this is not what I ended up executing, because my partition 2 was not on md8): [~] # more /proc/mdstat Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [multipath] md0 : active raid5 sda3[0] sdh3[7] sdg3[6] sdf3[5] sde3[4] sdd3[3] sdc3[2] sdb3[1] 20500882752 blocks super 1.0 level 5, 64k chunk, algorithm 2 [8/8] [UUUUUUUU] bitmap: 0/11 pages [0KB], 131072KB chunk md8 : active raid1 sdh2[2](S) sdg2[3](S) sdf2[4](S) sde2[5](S) sdd2[6](S) sdc2[7](S) sdb2[1] sda2[0] 530048 blocks [2/2] [UU] md13 : active raid1 sda4[0] sdh4[7] sdg4[6] sdf4[5] sde4[4] sdd4[3] sdc4[2] sdb4[1] 458880 blocks [8/8] [UUUUUUUU] bitmap: 1/57 pages [4KB], 4KB chunk md9 : active raid1 sda1[0] sdh1[7] sdf1[6] sdg1[5] sdd1[4] sde1[3] sdc1[2] sdb1[1] 530048 blocks [8/8] [UUUUUUUU]e2 bitmap: 0/65 pages [0KB], 4KB chunk You want to look for partition 2. In the above it is md8 but it could be different on your NAS. Just look where the drives are assembled in partition2 such as sda2, sdb2, sdc2 ,sdd2 etc. Once you know where partition two is assembled, do the following commands below: unused devices: [~] # swapoff /dev/md8 [~] # mdadm -S /dev/md8 mdadm: stopped /dev/md8 [~] # mkswap /dev/sda2 Setting upwapspace ve srsion 1, size = 542859 kB no label, UUID=7194e0a9-be7a-43ac-829f-fd2d55e07d62 [~] # mkswap /dev/sdb2 Setting up swapspace version 1, size = 542859 kB no label, UUID=0af8fcdd-8ed1-4fca-8f53-0349d86f9474 [~] # mkswap /dev/sdc2 Setting up swapspace version 1, size = 542859 kB no label, UUID=f40bd836-3798-4c71-b8ff-9c1e9fbff6bf [~] # mkswap /dev/sdd2 Setting up swapspace version 1, size = 542859 kB no label, UUID=4dad1835-8d88-4cf1-a851-d80a87706fea [~] # swapon /dev/sda2 [~] # swapon /dev/sdb2 [~] # swapon /dev/sdc2 [~] # swapon /dev/sdd2 5. Run e2fsck again [~] # e2fsck -f /dev/md0 e2fsck 1.41.4 (27-Jan-2009) Pass 1: Checking inodes, blocks, and sizes Pass 2: Checking directory structure Pass 3: Checking directory connectivity Pass 4: Checking reference counts Pass 5: Checking group summary information /dev/md0: 1979044/365993984 files (3.0% non-contiguous), 1352060256/1463959200 blocks 5. Run resize2fs, restart the RAID, and restart the services [~] # resize2fs /dev/md0 resize2fs 1.41.4 (27-Jan-2009) Resizing the filesystem on /dev/md0 to 2196523200 (4k) blocks. The filesystem on /dev/md0 is now 2196523200 blocks long. Amen. The rest is history. [~] # storage_boot_init 2 storage_boot_init 2 ... mdadm: /dev/md0 not identified in config file. mdadm: stopped /dev/md0 mdadm: /dev/md0 has been started with 4 drives. storage_boot_init.c: Start raid device /dev/md0 successfully md0 : active raid5 sda3[0] sdd3[3] sdc3[2] sdb3[1] md0 : active raid5 sda3[0] sdd3[3] sdc3[2] sdb3[1] md0 : active raid5 sda3[0] sdd3[3] sdc3[2] sdb3[1] md0 : active raid5 sda3[0] sdd3[3] sdc3[2] sdb3[1] storage_boot_init.c: /dev/md0 is active. storage_boot_init.c: Check filesystem on /dev/md0. storage_boot_init.c: check_last_degrade_error... [~] # /etc/init.d/services.sh start Start services: recycled.sh mysqld.sh atalk.sh ftp.sh bt_scheduler.sh btd.sh StartMediaService.sh ImRd.sh init_iTune.sh twonkymedia.sh Qthttpd.sh crond.sh nfs smb.sh lunportman.sh iscsitrgt.sh nvrd.sh snmp rsyslog.sh qsyncman.sh iso_mount.sh antivirus.sh ldap_server.sh vpn_pptp.sh vpn_openvpn.sh cloud3p.sh . Start qpkg service: Error: Optware is disabled. VideoStation is disabled. [~] # Success. Now all I had left to do was to restart the services that I needed. This looks daunting for someone with little Linux experience, but there is no way around it until QNAP fixes their scripts. Related Posts:TyreWiz not working after battery changeThe Great Cat Litter Poop OffSUTAB Scam?Amazon threatens customer of 26 yearsEnphase Envoy Local Access Information Technology ExpansionfeaturedQNAPraid
Thank you so much, Christian. I used ‘lsof | grep share’ to see wich files were open on /share/MD0_DATA. Saw processes like qpkg, ajaxplorer, USBCam. Deactivating wasn’t sufficien so I removed all these pakacges – didn’t use them anymore anyhow. Then expanding with resize2fs from 1T to 3T was then a matter of only an hour. Regards, Maarten
Thank you this saved my day/weekend. Together with “lsof /dev/md0” I could resolve the umount problem quite good (later on I saw the post by Maarten with the same solution :-). Anyway, I now have 4*4TB in Raid5 mode running smoothly on my TS459 Pro. Yeah! BR, TJ
Oh man….thank you soooo much for this tutorial. I had exactly the same problems and couldn’t have solved them without your excellent guide! I’m really dissapointed at QNAP though :/ Thumbs up!
Thank you from another very relieved QNAP user. Just upgraded my TS-559pro from 5x2TB to 5x4TB drives. As with everyone else, replacing the disks according to QNAP’s own instructions went fine but stumbled on the resize steps due to not being able to unmount /dev/md0. Kudos 🙂
Thank you so much for posting this and saving me so much time! Just like to add my alternate ending to your experience, since I had a minor heart attack not able to find the “storage_boot_init” command on my TS-469L running firmware 4.0.3. So “resize2fs /dev/md0” executed successful and I executed “storage_boot_init 2” to get everything up and running again, it told me “-sh: storage_boot_init: command not found”. To cut the story shot, after contacting support and your good self Christian, I tried the suggestion of simply rebooting the NAS “/sbin/reboot”. It rebooted without problem, with my RAID5 capacity expanded and happily synchronizing now. A quick check with “more /proc/mdstat” also reveal that the swap file (referenced as md8 in the instructions) space has revert back to its original setup 🙂 Thank you and best regards, Ray
Thanks man, that guide helped. Since I’m a pretty paranoid guy, I’d furthermore recommend doing all those long-running operations inside of screen (in case the ssh connection gets interrupted). I had to set TERM to a variable that the qnap screen version would like (it’s a bit picky), so here’s what did the trick: TERM=xterm screen (and then continue to do the file system check and the resize inside of that) Cheers!
http://www.cyberciti.biz/faq/linux-command-line-run-in-background/ You can always run a command in the background to avoid the risk of a lost connection.
Christian, I am so very grateful that you took the time to write down these instructions to the public, and in such a pedagogic way. I followed them exactly (well nearly, all I changed was the name for the second partition, which in my case was md4 instead of md8, but you clearly explained how to solve that). The only thing that puzzled me a bit was Ray’s comment about firmware version 4.0.3. The thing is that I also run 4.0.3 but in my case I could execute “storage_boot_init 2” without any problems. Now I have 4*4 TB Western Digital Red, a great THANKYOU! /Anders
Christian, Thank you for posting this great piece of VERY USEFUL INFO. With the detailed description I have managed to upgrade a TS-410U from 4×1.5TB to 4x2TB in RAID 5. Drive replacement was OK, it took aprox. 14 hours per disk. With Seagate ST2000DM001 drives. THe most difficult part was unmount the FS. In my case, DNSMASQ was the reason. Just kill the process and you are ready to unmount. Thanks again. Keep up with the good work. AlfonsoP
WOW, THANKS, IT WORKS LIKE A CHARME! All other solutions I’ve tried before are useless… Successfully updated a TS-410 (Firmware v4.0.2) from 4x2TB to 4x4TB (WD Red, 64MB, SATA-3), Raid 5
BIG THANKS. GREATE WORK! I successfully updated a TS-410 from 3x3TB to 3x4TB (Seagate ST4000DM0000), Raid 5
Christian, Thank you so much for posting this procedure!! This worked perfectly for me. I have a TS-859 (v4.0.2) which was configured with 4 x 2TB (WD Green) in RAID 5 and after four years one of the drives started to fail with read/write errors. The WD Green drives were on QNAP’s recommended list when I bought the NAS + drives and were later removed likely due to the fact that you have to know to change the head park interval to it’s max (or off) using wdidle3.exe on a computer prior to installing them in the NAS. Thankfully I HAD done that or else they would have failed a lot sooner with high Load Cycle Count (LCC) SMART drive errors and constantly been dropping from the RAID array one (or more) at a time. Anyway, I decided to upgrade to 4 x 4TB (WD Red) while I was replacing the failed drive. I swapped the failed drive no problem. After rebuilding completed, I used the expand capacity capability and swapped the others one at a time as recommended. When it came time to expand the RAID to use the new space, I got the same failure: stuck at 50%. Messing with it through the web admin did not fix anything as it would repeatedly resume at 50% and then fail. Turns out it was exactly the same “memory allocation failed” error. I followed your procedure and after the “resize2fs” command completed successfully, I did a “/sbin/reboot” and I was good to go. I didn’t think it was a good idea to try to upgrade the firmware in the middle of the whole process, so I hope QNAP fixes this in a future firmware update if they haven’t already. Thanks again! Jim P.S. Sorry to hear about your 2003 Accord’s failed transmission. Sounds like we have parallel lives or something. I too bought a brand new 2003 Accord EX V6 sedan and in 2009 around 100k miles it started getting weird shifting from 2nd to 3rd (yes, I took care of the transmission recall immediately too). Not wanting to deal with it failing completely or paying for a pricey repair, I traded it in on a used 2007 Nissan Altima 3.5SE which hasn’t been perfect but has been much more satisfying overall and more engaging with a 6-speed manual.
Hello your guide was the only way that I could finally get my RAID 5 array up and running I had 3 WD Red 2.78 TB disks and added a 4th and nothing I was trying was working to expand the array as it would always fail. To be honest I don’t use Linux at all so the guide did seem somewhat imposing. However I was able to perform all steps as you had them recorded until the storage_boot_init 2 command, which gave me an unknown command error. So I did a “/sbin/reboot” and the server rebooted, but after reboot the RAID array showed as unmounted….which freaked me out somewhat. I used putty to reconnect to the server and did another reboot just using the “reboot” command, which I assume does the same thing and this time the server came up and the RAID was mounted and was now synchronizing. This was all done on a QNAP TS-669L with firmware 4.1.0. Thanks again for the guide I wouldn’t have been able to muddle through it without this as nothing else on the forums was working and it came down to the swap file problem. Dave
I’m adding to the list of the thankful 🙂 Had same issues on a TS-469L moving from 2Tb disks to 3Tb disks. After replacing the fourth one, following the procedure, tried with the GUI to expand and failed, then went through the ssh path, and got the same “Memory allocation failed” message… Your solution worked like a charm, thank you !!! A.
Thanks! Upgraded my TS-410 from 4*2Tb to 4*4Tb disks, and had the same problem when extending the file system … adding the swapspace worked like a charm. Girlfriend also happy the NAS is available again after all those days of rebuilding and expanding 😀 Pieter.
I was about to try your method, but the new firmware 4.1.1 saved me and did the trick. My system is a TS-419P+ with FW 4.1.1 and 4 WD Red 4 TB disks in raid5, giving me 10998,25 GB storage. Thanks for your page and good luck to all you expanding your storage.
Hello Christian! Thank you for your notes. I had to create more swap, too. I was very glad, that you posted the commands. Alone, I wouldn’t have stopped the swap-RAID. It worked like a charm.
Thanks for this, it may come in useful. I upgraded from (now) incompatible drives to larger RED drives. With expansion around 75% complete, the qnap lost network connection and will no longer connect at all. All the lights on the drives are green and the LCD display lights up but does nothing. I’m contacting Qnap for advice, but does anyone know. Will I be able to put the old disks into a compatible new unit and re build the original raid? It appears to be possible if I shut down the unit and just switch the disks, but does the technique used during expansion stop this happening? Cheers
Christian Donner says November 1, 2013 at 8:18 pm Thank you very much for this tip, it is VERY useful! I was looking for a way to avoid the hassle to install screen on a QNAP. 🙂 Anyways, two more steps did it for me. First, I had to kill processes with open handles for a successful umount. Run: [~] # lsof /share/MD0_DATA/ then kill the listed PIDs. Second, the resize2fs command said: The filesystem is already 732174389 blocks long. Nothing to do! So I had to manually grow the RAID to use all the available space: [~] # mdadm –grow /dev/md0 –size=max
Same experience as Fishi abouve, with the additiion that I had to turn off bitmapping because of this error on the ‘grow’ command: mdadm: Cannot set device size/shape for /dev/md0: Device or resource busy To turn off bitmapping: mdadm –grow /dev/xxx -b none
Thank you very much. Your ssh example helped me to expand capacity on a client’s QNAP. The expand function in RAID Manager is still broken in the nov 2016 version of the QNAP software, unfortunately. Best, Zip