File Management

How to Find Duplicate Files by Content Before Deleting Anything

Group files by size, compare their SHA-256 hashes, and review which physical copies you want to keep.

Two camera folders can each contain a different IMG_0042.jpg. A camera original and a delivery copy may share their contents under entirely different names. Use filenames to locate candidates. Their byte counts give you the first useful comparison.

Start in one directory you control. Leave your backup drive and any synced working folder outside the scan unless you intend to review that location separately. Download online-only files that belong in the comparison and let downloads finish first. Close any app that's still writing the files, or make stable scratch copies before you compare them.

Narrow the list by size

Group files by their exact byte count. Different byte counts rule out an exact duplicate. Matching byte counts give you a list worth hashing. In an illustrative case, a three-file group with identical lengths may contain two copies of the same image and one unrelated image. A size filter by itself would flag all three.

PowerShell's Get-FileHash reads file content and reports a SHA-256 hash by default. A literal path keeps punctuation such as square brackets from being treated as a wildcard. Run this read-only comparison on two selected files from a scratch directory.

Get-FileHash -Algorithm SHA256 -LiteralPath '.\imports\frame-04.jpg', '.\delivery\cover.jpg'

Look at the whole hash string. Equal hashes give you a strong duplicate candidate. If the pair contains an important original and you want a direct byte comparison too, Python's filecmp.cmp supports the shallow=False option. Run a fresh comparison after any edits, since the library can cache results based on file metadata.

from filecmp import cmp

print(cmp("imports/frame-04.jpg", "delivery/cover.jpg", shallow=False))

Give every matching group a keeper

Record the path, byte count and hash for each candidate. Then add one reason why each copy exists. For example, a verified archival copy belongs in a second location even if it matches your working copy exactly. A duplicate in a forgotten import folder can go into a review folder after you confirm the delivery app opens the copy you plan to retain.

A project can still depend on the exact path of one copy. Check the project, script or external editor that uses it before moving the matching file. Files can have different permissions or other filesystem metadata outside their contents. Move a candidate to a holding folder outside the batch if you control the paths involved, then test the actual project before deleting anything. This workflow finds byte-for-byte duplicates. Similar photos, resized copies and re-encoded videos need a separate review.

Divide the candidates before deciding which files matter

Start with one folder you understand. Make a list of the candidate paths and their sizes in bytes. Avoid relying on a file manager's rounded sizes. Two files shown as 4 MB can have different lengths. Files with the same length may still contain unrelated photos. Use the size groups to decide where a content check is worthwhile. If you have an image with a small edit applied, place it beside its original to see why a similarity review needs different evidence.

Imagine the import folder has two files named “frame-04” at different paths, each shown as 1.8 MB. The delivery folder contains a renamed copy and a smaller preview. First identify which exact lengths match. Compute a hash for each plausible matching item while the files remain stable. Record the algorithm next to the value so the next comparison uses the same one. If two hashes are different, the two files did not match under the comparison. If they match, inspect why each path exists before acting.

For a high-consequence cleanup, make a byte comparison on the suspected pair and repeat after either file changes. Use the full path and check the file identity first. If the software can cache a comparison, use a new run when a file was saved during the check. Avoid comparing a placeholder or a partially downloaded local file. For synced content, the files-on-demand check explains how to verify that you have a real local copy before comparing it.

Record what a cleanup might disturb

A duplicate copy can have a different job. A delivery file is the authorised output for a client, while the same bytes in a second location may be the copy that a project opens. An archive stored separately may be part of the backup plan. Keep entries under their exact paths and write which application or person relies on them. If a file has a unique permission, modification history or filesystem attributes that matter, decide how to preserve those as well. Byte equality concerns the file data at the paths you compared.

Reason a copy exists What to test before moving it
It is the current delivery Open the link from the handoff and see which path it uses.
It belongs to a verified backup Keep the copy according to the backup plan.
It is a project asset Open the project with the candidate held aside in an authorised scratch test.
Its original purpose is unclear Preserve it in a review location and ask the record owner before deletion.

A sync service can reproduce a move or deletion elsewhere. Run the initial holding test outside a shared or synced workflow if the work allows it. For a copy inside a managed folder, check the authorised retention and restore route before changing anything. A backup file can be identical to a working original and still provide assurance that you can recover it.

Keep the decision reversible while you confirm

For each group, record the keeper and the paths you plan to set aside. Verify the keeper opens at the place expected by the app and another person can find it when needed. If a dependent tool references a changed name, keep the old path in the record and use the batch rename mapping approach to confirm the new reference. Take one test group through the actual workflow. For photo work, open the deliverable; for a media project, try the edit that reads the asset.

Leave candidates in a known holding location until the retention period or verification step your work actually needs has passed. If someone finds another use for a copy, restore it with its recorded path. Keep the hashes and decisions with the inventory. When someone adds a similarly named file later, compare its data from scratch before including it in the earlier group.

Sources