Big problem with corrupt files and songs skipping/dropping

I thought I had been pretty through about testing, testing and more testing before turning Beets lose on my collection (over 100K tracks or 9K+ albums). I seemed to get the directories just how I wanted them and several of you helped get me rolling. Thanks!

I use Roon for playback and it has a random feature once you finish plying your selections. As it jumps around I’m running into many problmes.

I’m finding about 1 out of every 3 tracks skips, drops or fails to play. When I find a song that has a problem I try playing the entire album and typically, several tracks on that album will have a problem. Some I am very familiar with and I am certain did not have issues before processing with Beets. I’ve copied over some of my backups, pre-Beets, for those bad albums and they play fine.

I discovered a feature on Roon where I can look for corrupt files. Please see below. However, I have problems with other tracks not reported as corrupt form Roon. Problems like skipping, dropping and even playing small bits of other songs before going back to the current, correct track. The corrupt ones typically won’t play at all or go to the next track.

What have I done wrong or how can I correct this moving forward? Fortunately I have backups before I processed with Beets, but the thought of starting over from a few weeks back is daunting. Should I be looking at Beets or something on my network, computer, etc. Files are on the same NAS as where Beets is installed and configured to run in a Docker/Container.

That certainly seems troubling! It would be good to understand more about exactly what kind of corruption these files are seeing.

Have you tried playing or inspecting the files with any other software? Maybe try a few other common tools, like VLC or similar. If that’s inconclusive, maybe you can share a sample so others can take a look?

Are the same files always corrupt when checking with Roon?

Have you tried copying the files to a local drive and checking?

It could be a network issue

I’ve got a good example with 3 versions of the same file:

  1. My original file that plays fine pre-Beets
  2. That same track after my first round of Beets and at roughly the 1:08 minute mark it plays a distinctly different bit of music for 2-3 seconds
  3. Same track that after I saw the corrupt notation in Roon I re-processed the entire album

I don’t see an obvious way to upload the files to my post, just the typical editor stuff for bold, quote, imbed picture, etc. How can I make the Depeche Mode files available to you?

Using Beyond Compare I can see that it does not think the files are the same. First screen shot the difference for the entire album, the second comparing the actual file “guts”. Note that in the first shot below that it only shows the different tracks for an album. Track #01 (Depache Mode) and Track #03 (Figures on a Beach) are reported in red while track #02 is not listed because the two version are the same

When I saw the problem yesterday I tracked down every album that contained a bad track (reported by Roon) and reprocessed them in the same way, that is on my NAS where the files and docker/container reside. So far the new files seem to play fine, though I can only realistically try a hand full.

That is fine, I guess, for the ones that Roon sees as corrupt. However there are many more that while I’m listening to randomly that just skip, drop, end, play other bit of music with no indication in Roon that they are bad.

I’ve copied all versions of The Depeche Mode track I mentioned in the post above to my local Windows PC. I’ve played in Foobar, Windows media player (locally) and Roon (after re-import). They all act the same. The original and the 2nd round of Beets both play fine. The one I originally processed in Beets has the odd music playing for a few seconds around the 1:08 mark.

I fear I have 100s or 1,000s that Roon won’t identify as corrupt but will not play correctly.

Interesting! Following the “network issue” hypothesis, are these files being accessed over a network filesystem (NFS, Samba, etc.)? That is, are the files on the NAS but beets is running on a client? That could imply that the network filesystem is to blame… it might be worth considering attempting to run beets directly on the NAS.

I must have not described clearly. NAS has all music files. NAS also has the ability to run Container app/services. So Beets is installed as a Docker image on my NAS’s Container “stuff”. I’m not an Image/Docker/Container expert but I think that is about right. At any rate Beets is running on the same NAS as where my music is stored.

I’ve freed up enough space on the NAS to have Original files stay put. Then I made an entire copy for Beets to process, once I thought I had all the kinks worked out. The only thing I’m using the network for is so my Windows PC can SSH into the NAS via PuTTY to run Beets via the command line.

Import Section of Config file looks like this:

import:
    quiet: yes 
    write: yes 
    copy: yes
    move: no
    delete: yes 
    resume: ask
    incremental: no
    quiet_fallback: skip
    log: /config/beet.log
    duplicate_action: remove #“remove” means remove **old** item
    from_scratch: yes

My thinking being that My original files never get touched. The copies start off in one place and gets processed by beets in “quiet” mode with the “duplicate_action” remove keeping my most recent version (deleting any originals from previous tests). If a match is found they get “write” yes to new location I know as being processed by Beets. The “delete” yes frees up drive space as they are processed and since “quiet_fallback” is skip, I should only end up with problem files in the starting directory for the copied files.

Does that all make sense?

So I don’t think there can be any network problems as NAS has both music files and it is where Beets runs.

Should I not be moving and deleting as I process? I’d like my workflow to be…Start with a bunch of files in one place. Process them. End up with good files in a new location. Have bad ones Beets could not match (based on my criteria) in the original starting location.

Really mysterious! I too use the “copy/move to a new directory structure” approach; I think it’s likely the most common way to do it. Your configuration looks perfectly normal.

That leaves it a mystery as to why the corruption is happening! Is there any chance this is behavior you can reproduce reliably in some way?

@47West63rd I would second that request: since you have identified specific tracks that have this issue, you could try backing up a bad track, then removing it from your beets library, then reimporting the original again using the same process, and finally see if the newly imported track is bad in a binary-identical way to the previous bad file.

The problem where you hear “small bits of other songs” is totally weird. I don’t see how there could be a bug in e.g. one of the metadata modification libraries which would result in such an outcome. Are the “other songs” only ones that were imported during the same operation? If not, it makes me think your NAS storage may have a serious problem.

One way to test whether your NAS is failing could be to first develop a reproducible operation that corrupts a file, then start a fresh beets library on a different destination filesystem not on the same physical storage, import again, and seeing whether it still happens.

Finally, maybe I missed it, but I didn’t see a post from you verifying that you tried to play these files with multiple other audio players, and heard the same corruption from all. It is important to rule out a faulty audio player.

I have not been able to reproduce it. That said I’m doing small test of the problem files. Not large batches or 5,000+ at a time. Is there any best practice for how much to process at a time? I like to get things set up after work and let it run all evening/night into the morning.

In another post I asked how I could identify files with a specific date. You suggested ImportAdded. I’m doing this below and end up with the original file dates once processed. Example I have an album that was originally ripped/tagged in October of 2012. Beets processes them, makes the directories as I want, updates tags because I can see MusicBrainz ID which was not there back in 2012 and finally moves to location where I want it (all on same NAS). Could the date manipulation have any impact?

importadded:
    preserve_mtimes: yes  # After importing files, re-set their mtimes to their original value. Default: no.
    preserve_write_mtimes: yes # After writing files, re-set their mtimes to their original value. Default: no.

I have played the “bad” songs on several players copied locally to windows box and played with Windows Media Player and Foobar. Same problems as Roon in the same spots. Re-processing with Beets from original files yields good, playable files all with Beets info present.

I’ve got plenty of them. Actually I seem to have a bunch of different problems

  • Songs that Roon identifies as corrupt and won’t play
  • Songs that play fine then have a hiccup of weird sounds then abruptly stops
  • Songs that play then have a few seconds of different music then go back to original song

Should note that Roon might identify 3 or 4 tracks as corrupt but when I play the whole album it might have 6 out of 10 tracks that do one of the three things above. There does not seem to be any pattern to a “bad album”. Some tracks good others have one of the 3 problems above. Other albums processed in the same batch are fine.

More info that might help. I live out in the country a bit. No broadband access. So all my music has to be local to me. No streaming, no NetFlix, Tidal, Qobuz, HBOMax, etc. I’ve got an industrial strength (CradlePoint) cellular modem. So all my internet comes from a cell tower…crazy in 2022 I know. Is there some relation to Beets doing lots of calls to MusicBrainz in the lager 5,000+ batches that could impact things? Seem unlikely, as I think Beets would just wait for the API call to MBrainz or how ever it gets it info.

I’m starting to notice a pattern that all the bad files are in like letters of the alphabet. Many of them are concentrated in the “D’s” and many are in U through Z. There are plenty in other letters of the alphabet but lots in those batches. Does that help at all? Almost 3/4 of the Ds. And I know I did the Ds as one batch and U-Z as one batch. Helpful?

NAS is nothing special but no slouch if we think that the problem could be overloading the NAS. QNAP TS453B with Celeron® J3455 4-core 1.5 GHz processor and max RAM, 8GB. It is only used for Music storage of FLAC and DSF files as well as Movies in .MKV format. Roon has its own NUC which is an i7. Prior to installing Beets on the NAS’s Docker/Container it did not run any apps like Kodi, Plex not even the backup/snapshot stuff from QNAP. It’s all been disabled. 4 X 10TB Seagate IronWolf drives so no consumer drives. All of that said, I’m not having problems so far with smaller batches. It will take me forever to get them all process at this rate. Nobody accessing the NAS when I’m process in Beets. I might be listening to music from the NAS when processing but nothing major like it serving up video. Everything is Cat-5 hardwired and enterprise D-link switches, no consumer products in the network chain.