I remember those computers with the keys for them, If I recall correctly, it was to prevent powering on or off depending on the position. I’d be willing to bet this is using an IDE standard drive based on it’s age. eBay would most likely have those. If you have any MS-DOS install disks in your collection, you may be able to format the HDD and get up and running by reinstalling it., unless it’s doing the "click of death’ or spinning up and then down up then down and so forth.
The turbo button was pretty fun for the time too. IF you were running at 16mhz, you could hit the button and it would boost it to 33mzh or something like that. Sort of for the extra kick you would probably need to compile programs or run a newer game at the time.
The CPU speed setting in the BIOS would help certainly. I would always set mine to the higher speed when it gave me the option.
More or less, the Turbo button was useful for the older software which couldn’t handle the ability to run faster or ran too fast. Depress the button to toggle the speed down or up. (Nice little read about these here) https://dfarq.homeip.net/why-did-old-pcs-have-a-turbo-button/
As for the HDD since you were able to write a file, it’s probably fine. One item of note, HDD’s are not considered permanent storage as you have natural degradation of the magnetic layers on the surface (a.k.a. bit-rot) which is what you are probably seeing with the corrupted files. Simply formatting the drive should be enough for it to refresh the drive’s platter(s) and create a more viable readable and writable surface. HDD’s will be more prone to failing via motor or controller board failure. You really have nothing to lose other than a little bit of time! Then, if it fails during Win95 or Win98 install, then yeah, replace the drive.