At midnight this morning, all automated data reconstruction was completed successfully on 23 of 41 volume groups comprising the disk array. The remaining 18 of 41 volume groups will be reconstructed in a less automated fashion, as was expected. A process to export the 23 reconstructed volume groups is nearly completed. This process serves toContinue reading “Major service interruption update 7:15 am”
Normal Moodle services are now available. Disk array data recovery is proceeding successfully, but very slowly. This recovery process is being closely monitored. As soon as possible after it is completed, other services will be restored to their normal working conditions. Your continued understanding is appreciated.
Yesterday, at approximately 4 am, we experienced a major hardware failure on one of the primary storage disk arrays used by many NJIT applications. We have been working since that time to restore services. As of noon today, approximately 70% of the consistency checks had completed. It is difficult to predict an exact time, butContinue reading “Major service interruption update and review at 2:05 pm”
The procedure to reconstruct data damaged in NJIT’s enterprise disk storage array is taken much longer than initially expected. Fortunately, there are no indications of any data loss, and data integrity is the highest priority. The Gmail authentication service has been restored and restoring the Moodle authentication service is under investigation. At this time, theContinue reading “Major service interruption update at 9:15 am”
Unfortunately, after careful analysis of the disk and data rebuild progress, and after discussions with Oracle engineers about the safest strategy for the remaining processes, we’ve determined that this service restoration might not be completed before 8:00AM Wednesday morning. Our highest priority continues to be the integrity of the data. Our work with the OracleContinue reading “Major service interruption status update at 10:45 pm”
Due to the complexities of a multiple component failure, it looks like we are still several hours away from service restoration. Our current estimation is around midnight. Thanks for your continued patience.
Repairs are continuing. Thus far there are no signs of data loss or corruption, however the consistency checking is proceeding slower than anticipated. The new estimated time for service restoration is 5 pm. We thank you for your continued patience.
Equipment repairs on the enterprise disk storage array are expected to be completed by 1:00PM. Then the 100 plus systems impacted by this failure will be evaluated and brought up in a priority order, with academic-related services being the highest priority. Assuming there are no data loss and/or corruption issues which will require file restoresContinue reading “Major service interruption status update at 12:30 pm”
At about 4:00 am this morning, an enterprise disk storage system failed which is affecting many critical services including administrative email, Gmail authentication, AFS and DFS file systems, Highlander Pipeline, etc. Field engineers are expected on-site by 10:30 am with replacement parts. We expect to know by 11:00AM whether the service interruption will be extended.Continue reading “Major service interruption”