AppleInsider › Forums › Software › Mac OS X › Auto defrag + secure delete = not secure?
New Posts  All Forums:Forum Nav:

Auto defrag + secure delete = not secure?

post #1 of 49
7/14/06 at 6:22pm
Thread Starter 
Mac OS X's ability to securely empty the trash is great when you have something you want to get rid off permanently. Then there's the auto defrag functionality that rewrites new unfragmented copies upon read when a file satisfies certain criteria (below 20MB, greater than x number of fragments, etc.)

My question is: Is it possible that auto defrag creates multiple copies of a file and marks the older ones as deleted (normal operation), and that when using secure delete on the last copy, the older ones are still on the hard drive?

Assume in this case that FileVault is *not* turned on.

Thanks in advance.
PM G5 Dual 2.0GHz, 2GB RAM
PB G4 1.67GHz, 1.5GB RAM
Reply
PM G5 Dual 2.0GHz, 2GB RAM
PB G4 1.67GHz, 1.5GB RAM
Reply
post #2 of 49
7/14/06 at 11:32pm
That's a VERY good question.
Hiro's Hall of Shame ignore list: Tulkas -- because we know he wasn't born dumb.
Reply
Hiro's Hall of Shame ignore list: Tulkas -- because we know he wasn't born dumb.
Reply
post #3 of 49
7/15/06 at 2:09am
Quote:
Originally posted by drumsticks
Is it possible that auto defrag creates multiple copies of a file

No.
Reply
Reply
post #4 of 49
7/15/06 at 3:50am
Thread Starter 
Ok. Perhaps I should rephrase. Auto defrag creates a new unfragmented copy and 'discards' the fragmented one, which still remains on disk.

When one finally secure deletes the final unfragmented file, one is still left with a fragmented copy! Not so secure after all!

Or is it?
PM G5 Dual 2.0GHz, 2GB RAM
PB G4 1.67GHz, 1.5GB RAM
Reply
PM G5 Dual 2.0GHz, 2GB RAM
PB G4 1.67GHz, 1.5GB RAM
Reply
post #5 of 49
7/15/06 at 4:06am
Quote:
Originally posted by drumsticks
Auto defrag creates a new unfragmented copy

No.

What happens is that the inefficiently-placed fragments are restructured to form a new, more coherent fragment.

Because srm is file system-transparent, it doesn't actually know about any of this. It doesn't ask HFS+ for where the fragments are and then overwrite them one by one. It doesn't check for physical locations. Rather, it uses HFS+'s APIs to overwrite the file, regardless of its fragments. If, in the meantime, HFS+ were to relocate fragments due to hot file clustering (what you presumably mean by "auto-defrag"), srm would be paused, because the fragment being moved would cause the file to be locked. srm would wait for unlocking, then continue.

There is no security issue here. Interesting theory, however.
Reply
Reply
post #6 of 49
7/15/06 at 5:06am
Thread Starter 
Thanks for your replies Chucker, but I still don't get it. According to http://www.kernelthread.com/mac/apme...izations/#FIVE

Quote:
When a file is opened on an HFS Plus volume, the following conditions are tested:
  • If the file is less than 20 MB in size
  • If the file is not already busy
  • If the file is not read-only
  • If the file has more than eight extents
  • If the system has been up for at least three minutes
If all of the above conditions are satisfied, the file is relocated -- it is defragmented on-the-fly.

File contiguity (regardless of file size) is promoted in general as a consequence of the extent-based allocation policy in HFS Plus, which also delays actual allocation.

That suggests to me that a new contiquous version is created to replace the fragmented version!

I guess I'm talking at the HFS+ level, while you are talking at the SRM (Storage Resource Management?) level.

PS: I've used "auto-defrag" when I probably should have used "on-the-fly-defrag".
PM G5 Dual 2.0GHz, 2GB RAM
PB G4 1.67GHz, 1.5GB RAM
Reply
PM G5 Dual 2.0GHz, 2GB RAM
PB G4 1.67GHz, 1.5GB RAM
Reply
post #7 of 49
7/15/06 at 5:17am
Actually, the very piece you quote provides your answer:
Quote:
If the file is not already busy

When securely deleting the file, it would be busy, and thus this on-the-fly defragmentation would not kick in to begin with.

Conversely, while on-the-fly defragmentation is happening, secure deletion would be paused and would wait until the file has been "relocated", as Singh calls it (which I believe is somewhat misleading). After that, secure deletion would happen to the new physical location of the file.

"But wait", you say, "what happens to the old physical location? Aren't there going to be remains?" Yes, possibly, and worse, the secure deletion utility (e.g. srm) would not know about them. To get of these unlikely, scattered and near-impossible to find traces, one would have to securely erase the entire file system.
Reply
Reply
post #8 of 49
7/15/06 at 7:37am
If you are really worried about security you could use the "Erase Free Space" feature located in Disk Utility.
Reply
Reply
post #9 of 49
7/15/06 at 7:40am
Quote:
Originally posted by troberts
If you are really worried about security you could use the "Erase Free Space" feature located in Disk Utility.

Even that isn't 100% proof, as free space can and will vary during the long operation.

Of course, you could always boot from another partition and make sure no other process write-accesses it. (For example, using single-user mode.) Now that's safe.
Reply
Reply
post #10 of 49
7/15/06 at 4:41pm
Thread Starter 
Quote:
Originally posted by Chucker
"But wait", you say, "what happens to the old physical location? Aren't there going to be remains?" Yes, possibly, and worse, the secure deletion utility (e.g. srm) would not know about them. To get of these unlikely, scattered and near-impossible to find traces, one would have to securely erase the entire file system.

Ah, that's the answer to my question... Thanks Chucker!

It's not that I have something that I really want to get rid of, this is just a hypothetical question at this stage. Am wanting to know how the insides of the OS works, that's all.
PM G5 Dual 2.0GHz, 2GB RAM
PB G4 1.67GHz, 1.5GB RAM
Reply
PM G5 Dual 2.0GHz, 2GB RAM
PB G4 1.67GHz, 1.5GB RAM
Reply
post #11 of 49
7/16/06 at 1:01am
Quote:
Originally posted by Chucker
When securely deleting the file, it would be busy, and thus this on-the-fly defragmentation would not kick in to begin with.

I don't think it's as important that they happen at the same time. I think the concern was more that when the files are defragmented over a long period of time, multiple copies are placed all over the hard drive. Then when secure delete occurs, it will only secure delete the current location of the data leaving portions of the information intact.

It's true that such information may be distributed too widely to be able to find but it may not be. I'd agree that erasing free space is probably the only way to get rid of sensitive information.
Reply
Reply
post #12 of 49
7/16/06 at 2:02am
Thread Starter 
Agreed. Thanks Marvin.
PM G5 Dual 2.0GHz, 2GB RAM
PB G4 1.67GHz, 1.5GB RAM
Reply
PM G5 Dual 2.0GHz, 2GB RAM
PB G4 1.67GHz, 1.5GB RAM
Reply
post #13 of 49
7/16/06 at 10:46am
Quote:
Originally posted by Chucker
Even that isn't 100% proof, as free space can and will vary during the long operation.

Of course, you could always boot from another partition and make sure no other process write-accesses it. (For example, using single-user mode.) Now that's safe.

Two questions:
1) Would booting from the installation and then running Disk Utility be secure?
2) What is the command for erasing free space while in single-user mode? My "The Missing Manual" doesn't cover that command, it's parameters (# of times to overwrite), or where the command is located.
Reply
Reply
post #14 of 49
7/16/06 at 11:07am
Quote:
Originally posted by troberts
Two questions:
1) Would booting from the installation and then running Disk Utility be secure?

Absolutely. (I assume you mean to call the "erase free space" option.)

Quote:
2) What is the command for erasing free space while in single-user mode? My "The Missing Manual" doesn't cover that command, it's parameters (# of times to overwrite), or where the command is located.

diskutil's man page says:
Code:

     secureErase [freespace] level device
                Securely erase a disk or freespace on a mounted volume.  Level
                should be one of the following
                        1 - Single pass randomly erase the disk.
                        2 - US DoD 7 pass secure erase.
                        3 - Gutmann algorithm 35 pass secure erase.  Ownership
                of the affected disk is required.



Don't try this out with a production volume, though, because I haven't tried it. Use some partition with random not as important stuff; perhaps back it up first.

Check the rest of the man page for device syntax. Or, just run "diskutil list" to see a list of identifiers you can use.

Note also that single-user mode doesn't mount read/write per default, and doesn't mount other partitions at all; you'll have to do that manually.
Reply
Reply
post #15 of 49
7/17/06 at 2:01pm
Quote:
Originally posted by Chucker
Absolutely. (I assume you mean to call the "erase free space" option.)



diskutil's man page says:
Code:

     secureErase [freespace] level device
                Securely erase a disk or freespace on a mounted volume.  Level
                should be one of the following
                        1 - Single pass randomly erase the disk.
                        2 - US DoD 7 pass secure erase.
                        3 - Gutmann algorithm 35 pass secure erase.  Ownership
                of the affected disk is required.



Don't try this out with a production volume, though, because I haven't tried it. Use some partition with random not as important stuff; perhaps back it up first.

Check the rest of the man page for device syntax. Or, just run "diskutil list" to see a list of identifiers you can use.

Note also that single-user mode doesn't mount read/write per default, and doesn't mount other partitions at all; you'll have to do that manually.

Thank you. I think I will stick to using my install disk or even install Tiger on my external disk so I have a working OS should something go wrong with my internal drive.
Reply
Reply
post #16 of 49
7/18/06 at 12:57am
put your super secret files on an encrypted disk image. any "residue" from relocating/copying something would mean nothing in part... then secure erase the disk image when done. that should be good enough for anyone i can imagine.
i freebase user interface
Reply
i freebase user interface
Reply
post #17 of 49
7/18/06 at 1:15am
Quote:
Originally posted by grad student
put your super secret files on an encrypted disk image. any "residue" from relocating/copying something would mean nothing in part... then secure erase the disk image when done. that should be good enough for anyone i can imagine.

Yep, I was going to suggest that. You don't even have to secure delete files off an encrypted image either 'cos the database is stored on the encrypted space and the file remains there until it is overwritten.

When you are done with the encrypted image, as said above, you could be completely secure by secure erasing just the image itself. Although, I don't think it's entirely necessary because to recover an encrypted disk image, you'd have to not only recover the entire image (this is why a bigger image is better) but you'd have to know or crack the key.
Reply
Reply
post #18 of 49
7/19/06 at 10:11am
The more important question is, what are you trying to erase?

If you are no longer using a machine, just write 0's to all the bits. Otherwise, what are you hiding?
Reply
Reply
post #19 of 49
7/19/06 at 10:14am
  • Kickaha
  • Really Fast Typing Member


You're kidding, right?

Amazing how an interesting technical discussion can be dragged into a pseudo-political quagmire by one dense comment.

How about: financial information; health information; anything that you don't want being used for identity theft...

Any of those on a laptop are an issue. Any of those on a home computer are an issue if your house is burglarized.

They should be encrypted. Barring that, they should be backed up to safe storage and then securely deleted off of machines that could be at risk.

However, in the presence of standard automated disk maintenance on HFS+, secure delete can still leave fragments of previous copies of the data lying around for recovery.

Only an idiot has nothing to hide.
My brain is hung like a HORSE!
Reply
My brain is hung like a HORSE!
Reply
post #20 of 49
7/19/06 at 10:19am
Quote:
Originally posted by jpennington
Otherwise, what are you hiding?

WTF kinda business of yours is it what drumsticks or troberts are "hiding"? Privacy rights exist for a reason, and just zero'ing out a hard drive is not enough to fully get rid of data.
Reply
Reply
post #21 of 49
7/19/06 at 10:28am
Quote:
Originally posted by Chucker
WTF kinda business of yours is it what drumsticks or troberts are "hiding"? Privacy rights exist for a reason, and just zero'ing out a hard drive is not enough to fully get rid of data.

Oh so the DOD seven pass write or the 35 pass write doesn't do a good enough job. I guess the DoD is an idiot then, since obviously their method of wiping drives isn't secure according to your standards.

I seriously doubt someone ripping off a laptop in an airport is doing it to steal your identity. You could also keep your computer secure enough so that if someone found that information they wouldn't be able to access it. You are right, it all should be heavily encrypted.

Identity thieves generally lack the skills to break into well secured systems. There are far easier and more vulnerable ways for them to do it.
Reply
Reply
post #22 of 49
7/19/06 at 10:31am
  • Kickaha
  • Really Fast Typing Member
Point.

Missed.

We were discussing the erasing of files in the presence of automated disk maintenance, and how EVEN 35-pass delete won't help this apparent issue.

The file blocks pointed to are well deleted.

The file blocks that were left behind from previous 'auto-defrag' moves are LEFT BEHIND FOR RECOVERY.

I thought that was pretty clear.

*What* is being deleted is completely, utterly, 100% irrelevant. The expectation of secure delete is broken.
My brain is hung like a HORSE!
Reply
My brain is hung like a HORSE!
Reply
post #23 of 49
7/19/06 at 10:34am
Quote:
Originally posted by jpennington
Oh so the DOD seven pass write or the 35 pass write doesn't do a good enough job.

No, you were suggesting zero'ing out, which doesn't do a good enough job.

7-pass write does. 35-pass write certainly does.
Reply
Reply
post #24 of 49
7/19/06 at 10:38am
I was under the impression that the 5220.22 standard instructed a 3+ pass writing zero's to all bits.

Kickaha -- I understand that was the point of the thread. I was just wondering what was so sensitive that he/she was only using the consumer OS feature of secure delete to remove. It was a just a question, no need to get so defensive about it.
Reply
Reply
post #25 of 49
7/19/06 at 10:50am
  • Kickaha
  • Really Fast Typing Member
Quote:
Originally posted by jpennington
Kickaha -- I understand that was the point of the thread. I was just wondering what was so sensitive that he/she was only using the consumer OS feature of secure delete to remove. It was a just a question, no need to get so defensive about it.

"What are you hiding?"

Actually, when I see another member get attacked like that, I will get defensive. It is none of your business. Or mine. If they have information, as we *ALL* do, that they prefer to keep private, then that is their business, and theirs alone.

To insinuate that because they want to secure delete data, which is a perfectly reasonable thing to do, they have something to 'hide' smacks of paranoia and snooping. We outgrew McCarthyism decades ago, I'd like to keep it that way.

So yes, it was just a question. So is "Have you stopped beating your wife?" Or "How's the kiddie porn ring going?" Or "Where's your sheet?" It was a question that was aggressive, and shouldn't have been asked, and frankly, was just plain off-topic, IMO.
My brain is hung like a HORSE!
Reply
My brain is hung like a HORSE!
Reply
post #26 of 49
7/19/06 at 10:55am
Alright, it was off topic. No one forced to your reply though.
Reply
Reply
post #27 of 49
7/19/06 at 10:58am
  • Kickaha
  • Really Fast Typing Member
Quote:
Originally posted by jpennington
Alright, it was off topic. No one forced to your reply though.

Just conscience.
My brain is hung like a HORSE!
Reply
My brain is hung like a HORSE!
Reply
post #28 of 49
7/19/06 at 11:05am
Quote:
Originally posted by Kickaha
Only an idiot has nothing to hide.

Reply
Reply
post #29 of 49
7/19/06 at 2:10pm
Quote:
Originally posted by Chucker
No, you were suggesting zero'ing out, which doesn't do a good enough job.

7-pass write does. 35-pass write certainly does.

The many pass overwiting was discussed here. Seems like it might be a bit overkill despite that it is a DoD/NSA standard.
Hiro's Hall of Shame ignore list: Tulkas -- because we know he wasn't born dumb.
Reply
Hiro's Hall of Shame ignore list: Tulkas -- because we know he wasn't born dumb.
Reply
post #30 of 49
7/19/06 at 2:24pm
Thread Starter 
Haha... I fully expected this thread to go off topic as it has. Thanks for those who replied.

I agree with Kickaha. It's not about what I or anyone have to hide. It's just a technical question. I don't even use secure delete at the moment, before or after asking the question. I just felt that this combination presented a risk of some sort. Whether the risk is relevant is not important, except that the risk does exist.
PM G5 Dual 2.0GHz, 2GB RAM
PB G4 1.67GHz, 1.5GB RAM
Reply
PM G5 Dual 2.0GHz, 2GB RAM
PB G4 1.67GHz, 1.5GB RAM
Reply
post #31 of 49
7/19/06 at 3:20pm
Quote:
Originally posted by drumsticks
Haha... I fully expected this thread to go off topic as it has. Thanks for those who replied.

I agree with Kickaha. It's not about what I or anyone have to hide. It's just a technical question. I don't even use secure delete at the moment, before or after asking the question. I just felt that this combination presented a risk of some sort. Whether the risk is relevant is not important, except that the risk does exist.

A risk isn't a risk if it is relevant.

(please note the following is a just a joke, so please don't this or any or my comments seriously)
A bear could walk into my living room right now and eat me, that is a risk I am taking by sitting here with a sliding glass door behind me. Is it relevant? No, thus I don't discuss it.
Reply
Reply
post #32 of 49
7/19/06 at 4:20pm
Thread Starter 
Quote:
Originally posted by jpennington
A risk isn't a risk if it is relevant.

It might be to some people who may have falsely believed that they have securely removed all traces of something when they might not have.

But I get your point though...
PM G5 Dual 2.0GHz, 2GB RAM
PB G4 1.67GHz, 1.5GB RAM
Reply
PM G5 Dual 2.0GHz, 2GB RAM
PB G4 1.67GHz, 1.5GB RAM
Reply
post #33 of 49
7/21/06 at 4:00am
I'm really not trying to be a dink... but I need to ask.

What kind of information could be so critical that it'd need to be written-over 35-times when erased...?

Unless you work for the FBI or you edit child-porn, I am having a tough-time thinking of anything that needs to be deleted that securely. Again, not flaming, I really am curious since I am on a desktop, so the odds of my machine being misplaced or stolen are much less than that of a laptop... and I do not have any high-level corporate information on my system.

I am just looking for an example or two so I can better understand the need.
"I drank what...?"
Reply
"I drank what...?"
Reply
post #34 of 49
7/21/06 at 4:49am
35 times is excessive (7 ought to be enough), but I would imagine corporate documents is a good example. You wouldn't want to be responsible for competitors to get ahold of internal documents.
Reply
Reply
post #35 of 49
7/21/06 at 8:17am
  • Kickaha
  • Really Fast Typing Member
Exactly. On my laptop I have literally dozens of files marked "Internal - Confidential" from my work. When I erase them, I expect them to be GONE, and so does my employer.
My brain is hung like a HORSE!
Reply
My brain is hung like a HORSE!
Reply
post #36 of 49
7/21/06 at 12:40pm
Quote:
Originally posted by Scott Finlayson
Unless you work for the FBI or you edit child-porn, I am having a tough-time thinking of anything that needs to be deleted that securely.

Secret love letters to your mistress which you'd rather hide from your wife who works for the FBI and can find the files if she wanted to

Billionaire's banking details

If you were a priest and had any form of pornography on your system (apart from child-porn which would probably get you a promotion)

If you worked in the entertainment industry and had a lot of contacts for famous stars that you wouldn't want to fall into the hands of Joe Public

If you secretly filmed your family members and friends who took showers in your house but didn't want them to find out

^ guess which are mine.

That's right all of 'em :P. I'm gonna be a rich, cheating, famous, voyeur of a pope.

There are lots of reasons. Admittedly, people who need security tend to have something to hide and more often than not it tends to be something bad. however, who is anyone to judge? We've all had a personal vault since we were born - our mind. Would you object to anyone snooping around in there? Well, unless you're a child molestor then you have nothing to hide right? Of course that's a ludicrous suggestion. Well, computers are just physical analogies of that personal space and everyone has the right to keep it to themselves.

I personally keep my letters of the love which never was and notes of how to commit suicide quickly to end the pain. Oh yeah and my online banking details. What's it to ya?
Reply
Reply
post #37 of 49
7/21/06 at 8:03pm
Quote:
Originally posted by Kickaha
Exactly. On my laptop I have literally dozens of files marked "Internal - Confidential" from my work. When I erase them, I expect them to be GONE, and so does my employer.

Yeah, I used to do it *twice* that much, that is, 35x2 just to kill any possibility of them ever coming back, even partially.
'L'enfer, c'est les autres' - JPS
Reply
'L'enfer, c'est les autres' - JPS
Reply
post #38 of 49
7/22/06 at 11:44am
One thing I'm curious about is the filesystem databases. I don't think that secure delete covers that. For example, let's say someone stored passwords in the names of text files. The database stores links to the filenames so any software that analysed the databases would recover the passwords. Norton did this under OS 9. Even if the file itself couldn't be recovered, the names of the files often were. Or does secure delete generate a random name? i know some secure delete software does this.

Another vulnerability we have to consider these days is meta data stored by Spotlight. I don't think it indexes encrypted drives but it would index a file as soon as it came onto your system. Although any deletion would remove the index from Spotlight, I don't know if it would overwrite the index.
Reply
Reply
post #39 of 49
7/22/06 at 5:20pm
I think that is a difference between secure-delete and ''total obfuscation". If the data is securely deleted then you meet the delete criteria. The spotlight indices will be deleted when the file is deleted, but the indices are probably not overwritten. But lets face it, if you are that paranoid about something you would be daft not to encrypt all of it from the get-go. AND set the virtual memory swap files to be encrypted too.
Hiro's Hall of Shame ignore list: Tulkas -- because we know he wasn't born dumb.
Reply
Hiro's Hall of Shame ignore list: Tulkas -- because we know he wasn't born dumb.
Reply
post #40 of 49
7/24/06 at 9:33am
The question I have about this is, is Apple aware of this, and can they fix secure delete to address all of the security problems discussed so far in this thread?

Seems to me that it's a very useful feature that ought to be toughened up to really be secure. I don't have enough knowledge on this topic to write to Apple and ask for this feature to be looked at and improved. Has anyone contacted Apple with their concerns on this security matter?
MacPro2,1 2 x 3GHz Quad Core Intel Xeons, 13GB RAM, 1.5TB Storage, 30" Apple Cinema Display pushed by an NVIDIA Quadro FX 4500
Reply
MacPro2,1 2 x 3GHz Quad Core Intel Xeons, 13GB RAM, 1.5TB Storage, 30" Apple Cinema Display pushed by an NVIDIA Quadro FX 4500
Reply
New Posts  All Forums:Forum Nav:
  Return Home
  Back to Forum: Mac OS X
AppleInsider › Forums › Software › Mac OS X › Auto defrag + secure delete = not secure?