So I didn’t do it in the CLI direct, but I had a whole lot of files in a collection that obviously had duplicates.
So I first used fdupes, which got a lot of them. But there were a lot of duplicates still. I noted a bunch were identifiable by having identical file size, but just some different metadata, so I made a quick work of presenting only files with identical stuff and went about reviewing and deleting.
Then I still see a lot of duplicates, because the metadata might be slightly different. Sizes were close, but non dupes also were close. I might have proceeded to write a little something to strip it the metadata to normalize, but decided to feed it to an LLM and ask to identify likely duplicates. It failed to find them all, and erroneously declared duplicates, but it did make the work go faster. Of course in this scenario a missed duplicate isn’t a huge deal, so I had to double check their results and might have missed some things, but good enough for the effort.
Sometimes my recall isn’t quite good enough for ctrl-r, but maybe an LLM could do better. Of course a better “search engine” also could do well. Also a mind numbingly obvious snippet could be generated without the tedium. Again, having to be careful to reviee because the LLMs are useful, but unreliable.
So I didn’t do it in the CLI direct, but I had a whole lot of files in a collection that obviously had duplicates.
So I first used fdupes, which got a lot of them. But there were a lot of duplicates still. I noted a bunch were identifiable by having identical file size, but just some different metadata, so I made a quick work of presenting only files with identical stuff and went about reviewing and deleting.
Then I still see a lot of duplicates, because the metadata might be slightly different. Sizes were close, but non dupes also were close. I might have proceeded to write a little something to strip it the metadata to normalize, but decided to feed it to an LLM and ask to identify likely duplicates. It failed to find them all, and erroneously declared duplicates, but it did make the work go faster. Of course in this scenario a missed duplicate isn’t a huge deal, so I had to double check their results and might have missed some things, but good enough for the effort.
Sometimes my recall isn’t quite good enough for ctrl-r, but maybe an LLM could do better. Of course a better “search engine” also could do well. Also a mind numbingly obvious snippet could be generated without the tedium. Again, having to be careful to reviee because the LLMs are useful, but unreliable.