Recovering from corrupt root filesystem

datamodel · February 1, 2005 9:44AM

Hi,

I have a broken iMac and want to check that my plan for recovery is sane. I'm from a Unix (Sun) background.

Recent action: Box is at 10.3.7, just installed Skype and PithHelmet. Haven't fsck/repaired for many weeks.

Symptoms: System becomes slow, Safari crashes on startup, every time. Uninstall PithHelmet - still does it, still slow.

Reboot box - doesn't come up - flashing question mark.

Reboot to firmware, eject CD, pop in install CD and boot it. Run Disk Utility.

Root filesystem mounts but is corrupt, "Repair Disk" fails ("command exited with an error code"). Three times. Repair permissions hangs halfway through.

So....

I can't see how to get a terminal window from the boot CD-ROM - perhaps a manual fsck would have more luck (or several of them).

So, I've bought a firewire external disk (was going to anyway). I'm intending on installing Panther onto it, and then pulling all the apps/data from the old disk onto the external one. As much as is possible anyway.

Then reinstall fresh over the internal disk.

Does this seem sensible? Is there other stuff I should be trying first?

I don't know whether to be pissed off at the iMac for dying the day before iLife '05, iWork arrive or whether to congratulate it on reading my mind and giving me a reason to buy a big external disk...

Thanks in advance,

Martin.

kickaha · February 1, 2005 10:24AM

While booting from the CD, hold down Cmd-S. If that fails, set the boot volume to the CD in OF directly, bypassing the Cmd-C requirement for CD booting, and then try Cmd-S.

datamodel · February 1, 2005 10:31AM

Quote:

Originally posted by Kickaha

While booting from the CD, hold down Cmd-S. If that fails, set the boot volume to the CD in OF directly, bypassing the Cmd-C requirement for CD booting, and then try Cmd-S.

This is for single user mode off the CD right?

Brilliant - I can try that this evening. The new disk arrives tomorrow and I'm out all evening, so if I can get it up and limping tonight that would great.

Thanks very much,

Martin.

kickaha · February 1, 2005 10:40AM

Quote:

Originally posted by datamodel

This is for single user mode off the CD right?

Yup.

Quote:

Thanks very much,

That's what we're here for.

datamodel · February 2, 2005 7:23AM

Well, I gave it a go.

I eventually settled on holding down Option to ge the boot chooser, selecting the CD and holding cmd-S while it booted. The other way kept, um, not working.

So anyway, I got single user. According to disk utility the disk is disk2s3, in single user though I have no /dev/disk2s3 (or rdisk device), and of the devices there the only thing I could fsck was the read-only root fs.

Am I missing something here? The disk utility finds and mounts the FS, but I couldn't see the device in single user.

I eventually went with:

# find `ls /dev/rdisk*`

do

fsck -y $i

done

to try the lot, to no avail.

Anyway, the external disk should be there when I get back today, so I'll give that a try.

Thanks for your help,

Cheers,

Martin.

kickaha · February 2, 2005 8:46AM

The disk numbering is completely dependent on what order the disks spin up in. In single user mode, it doesn't bother with any of them... note that when you first boot into SUM, you can't even write to disk. You have to explicitly fsck it and mount it as read/write. SUM is a safety net, so this makes sense.

Just follow the directions that appear just above the first prompt in SUM. Type in 'fsck -fy' (f forces the fsck even if the drive is journaled, which I assume yours is, it really is a big help), and let 'er rip.

You're making it much more complicated than it needs to be.

datamodel · February 2, 2005 9:17AM

Quote:

Originally posted by Kickaha

Just follow the directions that appear just above the first prompt in SUM. Type in 'fsck -fy' (f forces the fsck even if the drive is journaled, which I assume yours is, it really is a big help), and let 'er rip.

Ah... missed the -f, didn't read the message... d'oh! I got "unable to find superblock" for all the devices but the cdrom root fs.

Yep, it is journalled.

Quote:

You're making it much more complicated than it needs to be.

I was afraid I might be... normally when I'm doing this it's in a datacentre on some hooning great SPARC, with shouting people hanging about...

I only got it in December, still wearing in the mantra "Do the obvious. It will work."!

Devices are numbered in the order they spin up... whoa. I was expecting it to work like Suns, where the OF device tree is copied into the OS and named fairly straightforwardly.

Kind of unrelated but I love OF - I've used Sun's version (OBP) for years, and wondered what OF would be like. It's much better.

I couldn't walk the OF device tree though - on a SPARC I'd do:

ok cd /

ok ls

and see device paths - on OF the ls gives an error. Not important but wondering if this functionality would be expected to work.

Thanks,

Martin.

kickaha · February 2, 2005 9:38AM

Not really, OF is actually an implementation of Forth.

No, seriously. You've got a full programming environment in your BIOS. Very bizarre, very slick, and very powerful. What that means is that it's a whole different beast than the shell.

I'm not an OF guru by any means. Novice is more like it. Googling should get you where you need to be, and perhaps a poke around Apple's support pages.

As I understand it, the dynamic numbering of devices as they become available is to allow for a lot of flexibility when Things Go Wrong(tm). If a drive is simply DOA during boot on the Sun, does it keep expecting to find it?

datamodel · February 2, 2005 9:54AM

Quote:

Originally posted by Kickaha

Not really, OF is actually an implementation of Forth. No, seriously. You've got a full programming environment in your BIOS. Very bizarre, very slick, and very powerful. What that means is that it's a whole different beast than the shell.

Yep - OBP is the OpenBoot Prom - Sun's is very similar (in fact I think it's a Sun technology they opened as a standard (similar to SPARC).

The Sun one doesn't let telnet in to it though like the Apple implementation does. Apple have added a bunch of cool features.

What foxed me the first time was that on a SPARC it's "eject cdrom", and on the apple it's "eject cd". D'oh!

Quote:

I'm not an OF guru by any means. Novice is more like it. Googling should get you where you need to be, and perhaps a poke around Apple's support pages.

OK, thanks - I'll have a look at Apple's site - I've found good stuff there before.

Quote:

As I understand it, the dynamic numbering of devices as they become available is to allow for a lot of flexibility when Things Go Wrong(tm). If a drive is simply DOA during boot on the Sun, does it keep expecting to find it?

No, but it doesn't renumber the other devices either. The device names represent where the device is logically - e.g. Controller 1, Target 4, LUN 6, partition 0 on a Sun box would be /dev/dsk/c1t4d6s0 (s for "slice) and is a symlink to a much longer path under /devices which pretty much exactly represents the hardware path (but is hairy!).

Solaris can also dynamically add and remove devices, but it does insist on making everything follow the c?t?d? convention, so regardless of the underlying tech, it *looks* like SCSI...

I can go into much more detail, but you may find yourself slipping into unconsciousness (or fascinated, depending

).

Cheers,

Martin.

datamodel · February 2, 2005 10:06AM

Found the syntax for Apple OF at:

http://developer.apple.com/technotes/tn/tn1061.html

Thanks for the pointer.

Cheers,

Martin.

datamodel · February 3, 2005 3:31AM

Well, I'm running again on the external disk, still need to copy stuff back over.

My apps, etc. can I just drag from the Applications folder on the dead system disk (it reads OK, for the most part), to the Applications folder on the current system disk?

Things like Photoshop Elements ideally I don't want to have to mess about downloading again, etc.

As for the original disk, not sure what's going on there - still doesn't fsck, I bought Disk Warrior, after it's great reviews - nada. In Repair it prints a message "Speed inhibited by disk malfunction" and doesn't get any further than that (I left it overnight).

But Disk Warrior and Disk Utility report on the drives S.M.A.R.T. data and that says it's fine. So it looks odd - corrupt enough it's unfixable, yet I can still read most of the files off it.

I'll pull the files off it manually and erase/reinstall on it I think, see if anything comes up.

Disk Warrior's message had me a bit worried about the disk, sending it back to Apple, etc. but if S.M.A.R.T. reports all OK, then I guess hardware-wise I'm good?

Thanks,

Martin.

Recovering from corrupt root filesystem

Comments