Major service interruption update 7:15 am

At midnight this morning, all automated data reconstruction was completed successfully on 23 of 41 volume groups comprising the disk array. The remaining 18 of 41 volume groups will be reconstructed in a less automated fashion, as was expected. A process to export the 23 reconstructed volume groups is nearly completed. This process serves toContinue reading “Major service interruption update 7:15 am”

Major service interruption update and review at 2:05 pm

Yesterday, at approximately 4 am, we experienced a major hardware failure on one of the primary storage disk arrays used by many NJIT applications. We have been working since that time to restore services. As of noon today, approximately 70% of the consistency checks had completed. It is difficult to predict an exact time, butContinue reading “Major service interruption update and review at 2:05 pm”

Major service interruption update at 9:15 am

The procedure to reconstruct data damaged in NJIT’s enterprise disk storage array is taken much longer than initially expected.  Fortunately, there are no indications of any data loss, and data integrity is the highest priority.  The Gmail authentication service has been restored and restoring the Moodle authentication service is under investigation. At this time, theContinue reading “Major service interruption update at 9:15 am”

Major service interruption status update at 10:45 pm

Unfortunately, after careful analysis of the disk and data rebuild progress, and after discussions with Oracle engineers about the safest strategy for the remaining processes, we’ve determined that this service restoration might not be completed before 8:00AM Wednesday morning. Our highest priority continues to be the integrity of the data. Our work with the OracleContinue reading “Major service interruption status update at 10:45 pm”

Major service interruption status update at 12:30 pm

Equipment repairs on the enterprise disk storage array are expected to be completed by 1:00PM. Then the 100 plus systems impacted by this failure will be evaluated and brought up in a priority order, with academic-related services being the highest priority. Assuming there are no data loss and/or corruption issues which will require file restoresContinue reading “Major service interruption status update at 12:30 pm”

Major service interruption

At about 4:00 am this morning, an enterprise disk storage system failed which is affecting many critical services including administrative email, Gmail authentication, AFS and DFS file systems, Highlander Pipeline, etc. Field engineers are expected on-site by 10:30 am with replacement parts. We expect to know by 11:00AM whether the service interruption will be extended.Continue reading “Major service interruption”