or Connect
AppleInsider › Forums › General › Genius Bar › moving thousands of files
New Posts  All Forums:Forum Nav:

moving thousands of files

post #1 of 27
Thread Starter 
I have a that have several layers of sub-folders upon sub-folders in it. Inside all of these folders are JPEGs that I want to get out and put into a single folder.

I've tried doing a find for .jpg files in the main folder (both in the finder and with easyfind) but the resulting list is just too long (anywhere from 6000 to 17,000) and I have trouble moving the files. I was thinking about using the terminal to move the files but I can't figure out the syntax. Whatever I try copies the folder structure as well; that's not what I want.

I tried FileBuddy but I can't figure out a way to make it select only the jpegs inside all these sub folders and move them; I have to go one folder at a time or click the disclosure triangle on every single folder to get the JPEGs to display and then I can select them... that takes much too long. It also shows the insivible files which I don't need to see in this case but that's a minor issue.

I don't mind using the terminal to do this but I would prefer some utility as the other person doing this doesn't know enough to use the terminal. Moving around a folder with 17,000+ items in the finder ain't easy but that's something FileBuddy or the terminal can do easily once everything is in one folder I think.

Can anyone suggest a fast and easy way to do this?

Thanks
post #2 of 27
Thread Starter 
well don't I feel stupid..... I can use the search feature in filebuddy!

Other than FileBuddy, is there any other way to do this?

Thanks
post #3 of 27
Can Automator do that?
post #4 of 27
Thread Starter 
That's a good idea except I can't figure out how to get Automator to do it. I can't figure out how to have it search a folder full of sub folders for JPEGs and then copy all the found items to a specified folder. My find finder items seems to turn up zero results all the time so the next action, copy finder items, doesn't copy anything.

If you know how to use Automator, please share your secrets
post #5 of 27
Quote:
Originally Posted by FreddyFredFred View Post

That's a good idea except I can't figure out how to get Automator to do it. I can't figure out how to have it search a folder full of sub folders for JPEGs and then copy all the found items to a specified folder. My find finder items seems to turn up zero results all the time so the next action, copy finder items, doesn't copy anything.

If you know how to use Automator, please share your secrets

IMO Automator is one of the most poorly constructed applications of all time.
~jvb
Reply
~jvb
Reply
post #6 of 27
Quote:
Originally Posted by jvb View Post

IMO Automator is one of the most poorly constructed applications of all time.

I'm glad I'm not the only one who thinks so. It's a dumbed-down programming tool for non-programmers that can only be used by programmers. Terrible.

Proud AAPL stock owner.

 

GOA

Reply

Proud AAPL stock owner.

 

GOA

Reply
post #7 of 27
Quote:
Originally Posted by SpamSandwich View Post

I'm glad I'm not the only one who thinks so. It's a dumbed-down programming tool for non-programmers that can only be used by programmers. Terrible.

I also love when Automator is mentioned in interviews as "the best programming tool for Mac". It makes absolutely no sense. I must be missing something.
~jvb
Reply
~jvb
Reply
post #8 of 27
I haven't used Automator much, but it looks to me like this might work:

- launch Automator
- Select Finder
- Drag "Find Finder Items" to the workflow
- Drag "Get Selected Finder Items" to workflow
- Drag "Move Finder Items" to workflow (in this order)

Then go into your options for each of the three steps to select the files you want and move them to the location you want. Not sure if you need the middle step (Get) or not.

Try some test runs and see if it's doing what you want it to do.
post #9 of 27
Create a SMART folder that searches for .JPG files from the top level folder. Open the SMART folder and you will see all of the files in there. Move them as you would normally move them via Finder.
Most of us employ the Internet not to seek the best information, but rather to select information that confirms our prejudices. - Nicholas D. Kristof
Reply
Most of us employ the Internet not to seek the best information, but rather to select information that confirms our prejudices. - Nicholas D. Kristof
Reply
post #10 of 27
Half the time it's easier to write a shell script than make an automator script. You could write a recursive script that pulls all jpegs (or whatever) into a target folder, and deletes the source folder if empty.
Cat: the other white meat
Reply
Cat: the other white meat
Reply
post #11 of 27
Thread Starter 
Well..... nothing was working, even the smart folders until I remembered I put that folder of images in the spotlight privacy tab. Took it out and things started to work a little better.

Quote:
Originally Posted by F1Turbo

I haven't used Automator much, but it looks to me like this might work:

- launch Automator
- Select Finder
- Drag "Find Finder Items" to the workflow
- Drag "Get Selected Finder Items" to workflow
- Drag "Move Finder Items" to workflow (in this order)

Then go into your options for each of the three steps to select the files you want and move them to the location you want. Not sure if you need the middle step (Get) or not.

Try some test runs and see if it's doing what you want it to do.

This does work and I don't need the "get selected finder items" action.


Quote:
Originally Posted by talksense101

Create a SMART folder that searches for .JPG files from the top level folder. Open the SMART folder and you will see all of the files in there. Move them as you would normally move them via Finder.

This one works too and I think I like it better than the automator solution because I can see the entire list of files which helps me find problems with naming. It doesn't seem to work as nicely when there are duplicate names though. I tried to select 3 files (2 duplicates, 1 not) and the finder refused to copy any of them.

Is there any way to use smart folders to find duplicate file names in that big list of file names I end up with?

Quote:
Originally Posted by Splinemodel

Half the time it's easier to write a shell script than make an automator script. You could write a recursive script that pulls all jpegs (or whatever) into a target folder, and deletes the source folder if empty.

I like that solution except I can't write the scripts. About all I know is to do ls -R and that's about it. I don't know how to write the script to do it and it's not a user friendly solution because only I would be able to modify it. The other person doing this wouldn't know how to modify or use a shell script and the last thing I need is for them to have a typo in there that messes up everything.


It looks like a Smart Folder or FileBuddy is the way to go. A smart folder is free and gets the job done but file buddy seems to be better at finding and fixing the little problems before I put everything into a single folder.

Speaking of which, I do a search for .jpg in filebuddy and come up with 6208 items. I do a search for creator type of JPEG and come up with 6201. A Smart folder determines there are 6205 files with a name containing (or ending with) .jpg. Who knows where the problem is?
post #12 of 27
Thread Starter 
Unless the finder can do this, I think I'm going to have to go with FileBuddy. The difference between the 6201 nd 6208 in filebuddy were 7 tiff files. I added the type column to the list in filebuddy and sorted by type. Filebuddy showed me 7 files with a file type of tiff that had an ending of .jpg.

Still don't know how the finder is getting 6205 though It's 3 less than filebuddy shows when searching for a .jpg.

The fun continues....
post #13 of 27
Quote:
Originally Posted by FreddyFredFred View Post

I like that solution except I can't write the scripts. About all I know is to do ls -R and that's about it. I don't know how to write the script to do it and it's not a user friendly solution because only I would be able to modify it. The other person doing this wouldn't know how to modify or use a shell script and the last thing I need is for them to have a typo in there that messes up everything.

You have to start somewhere. . . Look up "Bash scripts" or simply "shell scripts" (you are probably running the bash shell). If you are willing to learn, it might be worth the time. If you have a little bit of programming experience, this will be very quick. If not, shell scripting is not a bad intro to programming.

And yes, you should practice on duplicated subsets of the data, which in this case are the folders of jpegs.

Lastly, you would most likely run the shell script as a file i.e. "./elscripto" so it behaves just like a program. Just make sure to give it odd permission bits (e.g. chmod 777 elscripto) so that it's executable. All of this should be covered in any basic shell scripting primer.
Cat: the other white meat
Reply
Cat: the other white meat
Reply
post #14 of 27
This little script will copy the items over. It will skip files that would overwrite files with the same name in the destination folder. The first choose folder is for the destination folder which must exist (make sure it is not inside the source folder) and the second choose folder is the folder containing the items to be copied.

Test it out. Click the link below to load it into Script Editor on your machine.

CLICK HERE to automatically place this script into Script Editor.
Quote:
set theDestination to (choose folder)
tell application "Finder"
\tset theList to items of (choose folder)
\ttell me to copyTheItems(theList, theDestination)
end tell


to copyTheItems(theList, theDestination)
\ttell application "Finder"
\t\trepeat with theItem in theList
\t\t\tif kind of theItem is "folder" then
\t\t\t\tset theNewList to items of theItem
\t\t\t\ttell me to copyTheItems(theNewList, theDestination)
\t\t\telse
\t\t\t\ttry
\t\t\t\t\tduplicate theItem to theDestination
\t\t\t\tend try
\t\t\tend if
\t\tend repeat
\tend tell
end copyTheItems
--Johnny
Reply
--Johnny
Reply
post #15 of 27
Thread Starter 
Starting with lundy's most wonderful and helpful script, I managed to hack the rest of what I wanted into it.

I found a bit of code on apple's web site to help me log errors to a file so now I will know which files have duplicates so I can track down the problems (or track down any other file that may have failed to duplicated I guess). It's not perfect (if I forget to close the file before running it again, I don't think it updates the file) but it's not the end of the world.

Here's the code I'm using. It might not be pretty but it looks like it will do the job. If anyone sees something horribly wrong with it, please let me know.

Now to find a way to use finder and command+F to perform all the same tricks as filebuddy

Code:


on write_to_file(this_data, target_file, append_data)
try
set the target_file to the target_file as text
set the open_target_file to ¬
open for access file target_file with write permission
if append_data is false then ¬
set eof of the open_target_file to 0
write this_data to the open_target_file starting at eof
close access the open_target_file
return true
on error
try
close access file target_file
end try
return false
end try
end write_to_file


display dialog ("Select destination folder")
set theDestination to (choose folder)

display dialog ("Select error file location")
global errorFileLocation
set errorFileLocation to (choose folder)

tell application "Finder"
display dialog ("Choose source folder")
set theList to items of (choose folder)
tell me to copyTheItems(theList, theDestination)
end tell


to copyTheItems(theList, theDestination)
tell application "Finder"
repeat with theItem in theList
if kind of theItem is "folder" then
set theNewList to items of theItem
tell me to copyTheItems(theNewList, theDestination)
else
try
duplicate theItem to theDestination
on error error_message
set this_error to "Error: " & error_message & ": " & name of theItem & return
--on error error_message number error_number
--set this_error to "Error: " & error_number & ". " & ¬
--error_message & theItem & return
set the log_file to ((errorFileLocation) as text) & "Script Error Log"
my write_to_file(this_error, log_file, true)
end try
end if
end repeat
end tell
end copyTheItems
post #16 of 27
I considered adding code to move all duplicates to a separate folder, but you could again have dupes, so I think your method of making a list is better in that regard. You will still have the files that "would have been dupes" remaining in the source folder, and you could add the line of code in your "on error" block

set label index of theItem to 2

Which would label it Red. Then you could easily find the files that were not moved.
--Johnny
Reply
--Johnny
Reply
post #17 of 27
All right you script kiddies (as long as you're on this)...

What would I have to do to clean up my iPhoto files?

After one of the previous Apple sytem updates, and my trying to locate a file, I mistakenly managed to erase my entire library of photos. After manually recopying them to my computer (from a backup DVD) I found iPhoto keeps Originals and smaller (file size) duplicates with the same names.

Here's the question: Is there a way to eliminate the secondary unneeded smaller dupes and keep the originals (and also the slightly edited for "rotation") versions? I'm wondering if I could recover Gigs of HD space by carefully doing this with a properly conceived script.

Proud AAPL stock owner.

 

GOA

Reply

Proud AAPL stock owner.

 

GOA

Reply
post #18 of 27
Nice to see AppleScript put to use, although it confuses the hell out of me. Applescript has always been a little bit mystical to me, kind of like makefiles.
Cat: the other white meat
Reply
Cat: the other white meat
Reply
post #19 of 27
Yep. Those are most likely Thumbnails. The trick is defining a way to identify them. The best way is to go by the file size and dimensions. The file size of a thumbnail is usually 19 KB and its dimensions are either 180 by 240 or 240 by 180. The file size isn't available directly from iPhoto, so you have to get the Finder to do it, after getting the pointer to the file from iPhoto. Here is a script that will loop through all of the photos and place those that are less than 20 kilobytes or have the dimensions 240 by 180 into a separate album. Then you can peruse those and see if in fact they are all thumbnails and not just small photos that you actually want.

CLICK HERE to automatically place this script into Script Editor.

Quote:
property minSize : 20 * 1024
property thumbAlbum : "Thumbnail-Sized Photos"
tell application "iPhoto"
\tif (not (exists album thumbAlbum)) then new album name thumbAlbum
\tset numThumbs to 0
\trepeat with theCurrentPhoto in photos
\t\tset theCurrentPhotoPath to image path of theCurrentPhoto
\t\ttell application "Finder" to set sizeOfTheCurrentPhoto to (size of (info for (theCurrentPhotoPath as POSIX file)))
\t\tset theDims to dimensions of theCurrentPhoto
\t\tif (sizeOfTheCurrentPhoto is less than or equal to minSize) and (theDims is {240.0, 180.0} or theDims is {180.0, 240.0}) then
\t\t\tadd theCurrentPhoto to album thumbAlbum
\t\t\tset numThumbs to numThumbs + 1
\t\tend if
\tend repeat
\t
\tif numThumbs is 0 then
\t\tremove album thumbAlbum
\t\tselect photo library album
\t\tdisplay dialog "No thumbnail-sized photos were detected." buttons {"OK"} default button 1 with title "Thumbnail Search Results"
\telse
\t\tif numThumbs is 1 then
\t\t\tset {pluralS, wasWere, letterA} to {"", "was", "a "}
\t\telse
\t\t\tset {pluralS, wasWere, letterA} to {"s", "were", ""}
\t\tend if
\t\t
\t\tselect album thumbAlbum
\t\tactivate
\t\tdisplay dialog (numThumbs as text) & " photo" & pluralS & " appeared to be " & letterA & "thumbnail" & pluralS & " and " & wasWere & " moved to the Album " & quote & thumbAlbum & quote & "." buttons {"OK"} default button 1 with title "Thumbnail Search Results"
\tend if
end tell


Quote:
Originally Posted by SpamSandwich View Post

All right you script kiddies (as long as you're on this)...

What would I have to do to clean up my iPhoto files?

After one of the previous Apple sytem updates, and my trying to locate a file, I mistakenly managed to erase my entire library of photos. After manually recopying them to my computer (from a backup DVD) I found iPhoto keeps Originals and smaller (file size) duplicates with the same names.

Here's the question: Is there a way to eliminate the secondary unneeded smaller dupes and keep the originals (and also the slightly edited for "rotation") versions? I'm wondering if I could recover Gigs of HD space by carefully doing this with a properly conceived script.
--Johnny
Reply
--Johnny
Reply
post #20 of 27
Fantastic. Thanks for the very helpful bit of programmery. One question about this, though... if they are thumbnails, maybe I should not remove them. Does this detect duplicate images based on the name and file size?

Again... thanks! This could be an extremely useful thread for us incompetent wannabe scripters.

Proud AAPL stock owner.

 

GOA

Reply

Proud AAPL stock owner.

 

GOA

Reply
post #21 of 27
Quote:
Originally Posted by SpamSandwich View Post

Fantastic. Thanks for the very helpful bit of programmery. One question about this, though... if they are thumbnails, maybe I should not remove them. Does this detect duplicate images based on the name and file size?

Nope. But nothing is getting deleted - it just moves the "suspected" thumbnails to their own album so that you can peruse them.

Finding dupes would be another script, much more complex and would take a long time on big libraries. Better to use the Sort menu item and Sort by Title.
Quote:
Again... thanks! This could be an extremely useful thread for us incompetent wannabe scripters.

My pleasure. Keep 'em coming. I've been coding Applescript almost every day since it came out in 1993 with System 7 Pro. It still sometimes takes me an hour to figure out a particular syntax. Thank goodness for Script Debugger from Late Night Software.
--Johnny
Reply
--Johnny
Reply
post #22 of 27
Thread Starter 
Quote:
Originally Posted by lundy View Post

I considered adding code to move all duplicates to a separate folder, but you could again have dupes, so I think your method of making a list is better in that regard. You will still have the files that "would have been dupes" remaining in the source folder, and you could add the line of code in your "on error" block

set label index of theItem to 2

Which would label it Red. Then you could easily find the files that were not moved.

It's a good thing you didn't put the code in to move the files from their original location.

Duplicate file names either means someone incorrectly named a series of images or someone has duplicated a folder of images and put it into another folder but left the original there.

The fun part is to figure out what happened in each case and how to fix it so just knowing what file name is the problem is more than enough to track down the issue.

I wish I knew AppleScript so I can automate all kinds of fun things but I've never been able to learn the language (I'm more familiar with C++ and mostly text based stuff). Sometimes I try and hack away at something but I usually give up because I can't figure out the syntax and how to use the variables the way I want.

Maybe we'll see about expanding this script in the future but Phtoshop/Imageready actions are working well for the other part of this at the moment

Thanks for the help lundy, much appreciated.
post #23 of 27
Quote:
Sometimes I try and hack away at something but I usually give up because I can't figure out the syntax and how to use the variables the way I want.

It fools you because it leads you to believe that it "understands" natural language. It was based on Hypertalk and I have always admired how the Apple software engineers chose syntax terms that were incredibly appropriate to how a natural-language command would sound.

However, the coolest part is that it is a true object-oriented syntax that is easier to understand because of the "tell" verb.

The key to rapidly advancing to the stage where you can just whip out the code is to understand the object hierarchy - and the application dictionaries are terrible at this. So I use Late Night Software's Script Debugger which will let you stop and see what the elements of each object are and how they are referred to.

What other language could you write a cool statement like

tell application "Finder" to set theList to every item of desktop whose name does not end with "jpg"
--Johnny
Reply
--Johnny
Reply
post #24 of 27
Thread Starter 
Quote:
Originally Posted by lundy

It fools you because it leads you to believe that it "understands" natural language.

Ain't that the truth. I don't know all those natural language words to use: every item of desktop who name does not end..... gah!! I find C++ much easier but I don't know enough so maybe I just haven't seen any of the really horrible stuff yet.

Quote:
The key to rapidly advancing to the stage where you can just whip out the code is to understand the object hierarchy - and the application dictionaries are terrible at this.

That's where I get messed up. I can use the dictionary or record something but then I don't know all those natural language words to put them together. Oh great, that program has a command to do what I want.... now how do I use it?

Script Debugger is quite an investment to make without knowing applescript. I think I'll have to learn it the hard way.
post #25 of 27
Quote:
Originally Posted by lundy View Post

Nope. But nothing is getting deleted - it just moves the "suspected" thumbnails to their own album so that you can peruse them.

Finding dupes would be another script, much more complex and would take a long time on big libraries. Better to use the Sort menu item and Sort by Title.


My pleasure. Keep 'em coming. I've been coding Applescript almost every day since it came out in 1993 with System 7 Pro. It still sometimes takes me an hour to figure out a particular syntax. Thank goodness for Script Debugger from Late Night Software.

BTW, lundy, what's your view of Automator?

Proud AAPL stock owner.

 

GOA

Reply

Proud AAPL stock owner.

 

GOA

Reply
post #26 of 27
Quote:
Originally Posted by SpamSandwich View Post

BTW, lundy, what's your view of Automator?

People have to understand what the purpose of Automator is. It is not a general programming language. It is for the purpose of automating repetitive tasks that consist of several clearly defined steps - things like resizing all the photos in a folder and then passing the resized photos onto something else for further processing and then putting them in a folder - like an assembly line. If the task gets slightly complex or needs custom filtering or other decision-making steps, then you have to insert an Applescript step or just go with Applescript to begin with.

Automator has one very useful feature, however - and that is to put a workflow into the Contextual Menu. Unfortunately it only will do this for the Finder, PDF dropdown menu, and some others. So what I often do is make an Automator workflow for the sole purpose of wrapping an Applescript, so that it can appear in the Contextual Menu of the Finder.

Automator is used extensively by Leopard itself, in some of the new features, especially in the developer tools.
--Johnny
Reply
--Johnny
Reply
post #27 of 27
Quote:
Originally Posted by lundy View Post

What other language could you write a cool statement like

tell application "Finder" to set theList to every item of desktop whose name does not end with "jpg"

It's weird when I think about it but I feel more comfortable writing something like:

list = []
for item in Finder.getDesktopItems():
____if item[-3:] == "jpg":
________list.append(item)

and yet when you see them side by side it makes you wonder why programming languages are so obscure. Maybe it's got something to do with it being more international. It might be incredibly hard for a foreigner to figure out an English Applescript.

Quote:
Originally Posted by freddyfredfred

I think I'll have to learn it the hard way.

Nope, Leopard is going to have Python and Ruby application scripting so Applescript shouldn't be a requirement any more.

On the subject of Automator, I've grown to like it but as lundy says you have to realise that it's a workflow builder. Once you understand what it's for, it can be quite useful. One thing it would be nice for is building application installers - move file x here and file y there etc.

One of its biggest flaws is that it's too linear though. I would have much preferred a node based setup where you join inputs like Shake does. This way you can split one output to do multiple things instead of making multiple separate workflows.
New Posts  All Forums:Forum Nav:
  Return Home
  Back to Forum: Genius Bar
AppleInsider › Forums › General › Genius Bar › moving thousands of files