Ooops

A Japanese university inadvertently wiped a colossal 77TB of research data from its supercomputer after a software update pushed by Hewlett Packard Enterprise (HPE) caused a script to go rogue and delete backups.

Kyoto University said 34 million files from 14 research groups had been deleted – and nearly a third of them will not get their data back after the incident which it blamed squarely on the HPE supercomputing system.

A software update error meant the Cray/HPE system deleted almost all files older than 10 days held in large capacity disc storage backup rather than just log files. It had initially feared up to 100TB was permanently lost.

Hewlett Packard said in a letter published by Kyoto University on December 29, 2021 that it took “100% responsibility” for the issue
...
HPE said: “The backup script includes a find command to delete log files older than 10 days. In addition to functional improvement of the script, the variable name passed to the find command for deletion was changed to improve visibility and readability.”
...
The company added: “However, there was a lack of consideration in the release procedure of this modified script. We were not aware of the side effects of this behavior and released the [updated] script, overwriting [a bash script] while it was still running,” HPE admitted. “This resulted in the reloading of the modified shell script in the middle of the execution, resulting in undefined variables. As a result, the original log files in /LARGE0 [backup disc storage] were deleted instead of the original process of deleting files saved in the log directory.”