Understanding Singletons behavior

Help
1

I am fairly new to Beets and just blown away at its capabilities. The level and quality of documentation alone is amazing, but I do still have questions.

I have been slowly going through my collection and cleaning things up. I first focused on full albums and learned a lot through that process and I am very happy with the results. Now I am turning my attention to partial albums and singles.

I was unsure of what my workflow would be for taking this on so I did a backup of the files and then a quick test. I took some counts of the numbers of files for reference so that I could validate what I believed would happen. My belief was that by turning on the Singleton option, I would have imported files going into the directory set up for ā€œnon-albumsā€ and the album side would be unaltered.

I picked one artist that had a couple dozen files that comprised parts of several albums - none of which were complete. When I ran the import, it appeared to work correctly. Each track was evaluated individually and I saw that they were moved to the non-album folder with the correct naming convention. Before I cranked up the process, I went back and validated my counts.

Turns out that the total number of files in my complete albums directory went down. Investigating this, I found that a handful of tracks that were in the complete albums were pulled into the non-albums folder and correspondingly removed from the beets database as being part of the complete album. For example, a greatest hits album that was complete, now is missing one track and that track is in the non-albums folder.

This was unexpected (validates backups and testing). Any ideas about what I might be doing wrong or insight into where my understanding is flawed?

Thanks for your help in advance.

2

Thatā€™s a very interesting problem! Can you elaborate a bit on how your configuration is set up?

I think the process would work how you expect if you were doing the ā€œstockā€ configuration, which does this:

  • The ā€œsourceā€ (unorganized) files are in one directory, and the ā€œdestinationā€ (organized beets library) is in another one.
  • Weā€™re exclusively either copying or moving files from the ā€œsourceā€ to the ā€œdestination,ā€ and we never import from the ā€œdestinationā€ library directory.

Correct me if Iā€™m wrong, but it sounds likeā€”one way or anotherā€”youā€™re importing singletons from a directory containing files already tracked by beets. You might reasonably expect beets to treat these as if they were any other file, copying/moving them to the ā€œNon-Albumā€ directory. However, beets actually triggers a ā€œreimportā€ when it sees you importing files it already knows about:
https://beets.readthedocs.io/en/stable/reference/cli.html#reimporting

If you reimport a song thatā€™s part of an album as a singleton, it would indeed get disassociated from its original album. Is this possibly whatā€™s going on?

3

From a file structure standpoint, I havenā€™t deviated from the suggested defaults. I have a music folder with Album and non-Album subfolders. The source directory is one that I randomly named Junk since I was moving incomplete albums and true singles into that folder while I was doing the initial complete album imports.

The specific test that I did was for the artist Alan Jackson. I have 17 complete albums of his that were migrated into beets and appropriately stored into the Albums directory. In the Junk folder, I had a subfolder named Alan Jackson with a couple of subfolders there that represented incomplete albums.

I did a simple command line of ā€œbeet import ./Alan\ Jacksonā€ from the Junks directory so all configuration was from my config.yaml file. I am posting up the full config file below.

As you describe, once I noticed the count discrepancies, I looked for where the missing files came from. Most were from one album which is a greatest hits sort of album (Genuine: The Alan Jackson Story). This clearly has tracks that span many original albums, but this album was imported successfully previously as a single album. It turns out that the missing songs are now in non-Album directory. Also when I query beets for the original album, it is updated with the songs removed.

What I canā€™t say with confidence (I didnā€™t back up the Junk folder before I started) is whether the songs in question were in the Junk folder as potentially duplicates to what was already in beets as a complete album, so I canā€™t say definitively whether the original track was moved from its location in Albums over to non-albums or if maybe the song was reimported and beets viewed the original track as a duplicate and deleted it.

I will see if I can construct another test to pin that question down.

Here is my config.yaml

directory: /Volumes/share/music
library: /Volumes/share/beets/library1.db

import:
copy: no
move: yes
write: yes
log: /Volumes/share/beets/import.log

group_albums: true

duplicate_action: remove
default_action: skip
singletons: yes
asciify_paths: yes

per_disk_numbering: yes

bell: yes
incremental: yes
incremental_skip_later: yes
resume: yes

paths:
default: Albums/$albumartist/$album%aunique{}/$disc-$track $artist - $title
singleton: Non-Album/$artist ($year) - $title
comp: Compilations/$album%aunique{}/$disc-$track $artist - $title
albumtype_soundtrack: Soundtracks/$album/$disc-$track $artist - $title

#replace:

ā€˜[\/]ā€™: _

ā€˜^.ā€™: _

ā€˜[\x00-\x1f]ā€™: _

ā€˜[<>:"?*|]ā€™: _

ā€˜.$ā€™: _

ā€˜\s+$ā€™: ā€˜ā€™

ā€˜^\s+ā€™: ā€˜ā€™

ā€˜^-ā€™: _

match:
strong_rec_thresh: 0.2 # match 60% or better for auto import
max_rec:
missing_tracks: low
unmatched_tracks: low
preferred:
countries: [ā€˜USā€™, ā€˜GB|UKā€™]

media: [ā€˜CDā€™, ā€˜Digital Media|Fileā€™]

  original_year: yes

#using the lastgenre plugin for genre setting
musicbrainz:
genre: no

#plugins: badfiles duplicates embedart fetchart info lyrics mbsync scrub unimported web
plugins: scrub embedart fetchart lyrics lastgenre replaygain #convert

scrub:
auto: yes

embedart:
compare_threshold: 70

fetchart:
cover_names: front art album folder
sources: filesystem coverart amazon albumart

lyrics:
auto: yes
google_API_key: AIzaSyBsVfIIN1tOZLA9Q3FXyp0X8qs2O4cziog
google_engine_ID: 009217259823014548361:lndtuqkycfu
sources: google musixmatch genius tekstowo

lastgenre:
auto: yes
canonical: no
count: 3
fallback: none
force: yes
min_weight: 10
prefer_specific: no
source: album
whitelist: yes
title_case: yes

separator: A separator for multiple genres. Default: ', '.

replaygain:
auto: no
backend: command
per_disc: no

convert:
auto: yes

#unimported:

ignore_extensions: png

ignore_subdirectories: NonMusic data temp

4

Sorry for the formatting on the config.yaml file. It looks like when I used the block quote function it took lines that are commented out and increased the font and bolded them. Those can be ignored.

5

I did another test. This time with Al Green and I backed up the source directory this time so that I could analyze the questions from before. I got similar results. took counts before and after and saw a difference of 3 files reduction in my completed albums directory.

Looking at what was imported, the 3 that are now missing in the completed albums folder were files that were being imported as singletons and already existed in the completed albums folder. As for the mechanics of whether the now missing files were moved or deleted, I canā€™t determine, but it seems like they are being recognized as being dupes and the original file is being removed out of the complete albums folder and being captured into the non-albums folder.

It is ā€œalmostā€ a good outcome. If the result of importing singletons that are dupes of existing files was to not import the new file, that would be fine, but removing the file from the complete album to have it exist as a singleton is less desirable.

Hopefully I am simply doing something wrong (most likely the case). Any insight you can provide is much appreciated.

6

Lastly, the log seems to prove what I was thinking. The problem for me is in duplicate handling.

duplicate-replace /Volumes/share/newmusic/xJunk/Alan Jackson/Everything I Love/01-01 Little Bitty - Alan Jackson.mp3
duplicate-replace /Volumes/share/newmusic/xJunk/Alan Jackson/Everything I Love/01-04 Between the Devil and Me - Alan Jackson.mp3
duplicate-replace /Volumes/share/newmusic/xJunk/Alan Jackson/Everything I Love/01-05 There Goes - Alan Jackson.mp3
duplicate-replace /Volumes/share/newmusic/xJunk/Alan Jackson/Everything I Love/01-06 A House With No Curtains - Alan Jackson.mp3
duplicate-replace /Volumes/share/newmusic/xJunk/Alan Jackson/Who I Am/01-01 Summertime Blues - Alan Jackson.mp3
duplicate-replace /Volumes/share/newmusic/xJunk/Alan Jackson/Who I Am/01-02 Livin' on Love - Alan Jackson.mp3
import started Wed Mar  8 10:12:20 2023
import started Wed Mar  8 12:51:25 2023
duplicate-replace /Volumes/share/newmusic/xJunk/Al Green/Compact Command Performances_ 14 Greatest Hits/01-05 How Can You Mend a Broken Heart - Al Green.mp3
duplicate-replace /Volumes/share/newmusic/xJunk/Al Green/Compact Command Performances_ 14 Greatest Hits/01-11 Love and Happiness - Al Green.mp3
duplicate-replace /Volumes/share/newmusic/xJunk/Al Green/Compact Command Performances_ 14 Greatest Hits/01-12 Take Me to the River - Al Green.mp3
duplicate-replace /Volumes/share/newmusic/xJunk/Al Green/More Greatest Hits/01-13 How Can You Mend a Broken Heart - Al Green.mp3
7

Any suggestions for what I might be doing wrong here? I am at an impasse. I canā€™t process my files that are not full albums since many might be parts of existing completed albums. Given the behavior, I might end up with a bunch of singletons, but my complete albums will be devastated. I am sure that I am simply doing something wrong, but backing up, testing, reading the docs, testing some more, etc. hasnā€™t led me to an answer yet. I am sure that I am simply doing something wrong.

8

I notice your configuration has this in the import section:

duplicate_action: remove

Thatā€™s whatā€™s causing beets to remove old files that are identical to old files. I would strongly recommend disabling this optionā€”it seems like youā€™re hoping to keep duplicates, which a different setting:
https://beets.readthedocs.io/en/stable/reference/config.html#duplicate-action

But I would probably just remove this altogether, if it were me, so I get prompted every time thereā€™s a duplicate.

9

Thanks for helping me out. Once I saw the log file, I went back and changed the duplicate action. My thinking originally was that the ā€œremoveā€ would discard the duplicate being introduced. I was wrong, it deletes the duplicate file that already exists. ā€œAskā€ sounds like the way to go. My guess is that when it does ask, it will want one of skip, keep, remove, or merge. The option of merge sounds like it would essentially take the new files and fill in the blanks in the old and discard the dupes. Is that correct?

Here is the output of my testā€¦

beet import ./Genuine_\ The\ Alan\ Jackson\ Story\

/Volumes/share/newmusic/Alan Jackson/Genuine_ The Alan Jackson Story (59 items)
Tagging:
Alan Jackson - Genuine: The Alan Jackson Story
URL:
https://musicbrainz.org/release/1b433295-be7a-43a2-ac5b-9d98461330be
(Similarity: 99.8%) (tracks) (3xCD, 2015, US, Arista Nashville)
CD 1

  • Gone Country (3:52) ā†’ Gone Country (4:20) (length)
    CD 2
    CD 3
    This album is already in the library!
    Old: 52 items, MP3, 314kbps, 191:33, 516.5 MiB
    New: 59 items, MP3, 315kbps, 216:48, 585.9 MiB
    [S]kip new, Keep all, Remove old, Merge all? m

I selected the option merge. It output a list of all the 52 existing files (not pasting in here) then showedā€¦

Tagging:
Alan Jackson - Genuine: The Alan Jackson Story
URL:
https://musicbrainz.org/release/1b433295-be7a-43a2-ac5b-9d98461330be
(Similarity: 80.9%) (unmatched tracks, tracks) (3xCD, 2015, US, Arista Nashville)
CD 1

  • Gone Country (3:52) ā†’ Gone Country (4:20) (length)
    CD 2
    CD 3
    Unmatched tracks (52):

Note the existing tracks were identified as ā€œunmatched tracks (52)ā€. As shown above, the the source folder (which is a backup of the original folder) has 59 files, 52 of which are identical to those 52 currently in beets, the other 7 are the ones that are missing in beets. I then presented a list of the ā€œunmatchedā€ 52 tracks. At the end of the list of ā€œunmatched tracksā€ (again for brevity, not pasting that list here) I was given the optionsā€¦

Apply, More candidates, Skip, Use as-is, as Tracks, Group albums,
Enter search, enter Id, aBort? a

I selected Apply. I donā€™t know if that is right, but the others did not appear to be good solutions. After selecting apply, I see in the beets database that I now have the album duplicated and a new folder created with the same name, but the album_id appended to the end of the folder name. The original album reference in beets still does not have the missing files and the original folder is unchanged. I can clearly remove the first album and delete the files manually, but then I could just have done that to start. I donā€™t see where the purpose of the merge option is.

I can certainly do all of this work in manual steps, but that defeats the purpose. I must be doing something wrong.

10

If youā€™re adding new songs, and hoping not to disturb existing albums, the duplicate option you want is ā€œkeep all.ā€

If youā€™re hoping to integrate new tracks into incomplete albums that are already in your library, thatā€™s what ā€œmergeā€ is for. But for a more controlled experiment, you might try removing (beet rm -a) the old album and then importing everything together.

11

I really appreciate your patience with this. I am not generally a needy sort, but here I am. For the missing tracks due to my playing, I did indeed simply remove the existing from the files structure and beets and reimport the backup. That is what backups are for.

As much as I would like it to be otherwise, I have a plenty of missing files in my ā€œcompletedā€ albums. If I were to come across a missing tracks (in my singletons for example) and I would like to insert that song into the correct album, what would be the appropriate way to do that? I would assume simply ensuring singletons is no and ā€œbeet import ā€, but that canā€™t be right since a single version of a song could exist on multiple albums (all the greatest hits and other compilations for example) so how would it know how it should be slotted? It could be that the source track fingerprint might actually point it to a specific album, but I would like it to be applied to another (maybe that greatest hits album). How would this work?

I would think this is a common dilemma. During my initial clean up of completed albums, I would allow beets to do the ā€œquietā€ import first so that albums that were most accurately identified would be automatically imported, but everything that beets question, I would load into Picard and I would pull everything together there. Then I would update the tags, and then import into beets. This would (mostly) turn the problematic imports into clean imports. While slow, it did allow me to get a degree of accuracy that I was happy with, but I did import some albums without all of the songs. I also have a lot of singletons; either purely singletons or partial albums not close enough to easily clean up. I expect that in some cases, the singletons will hold the missing tracks for some of the albums. If so, I would like to be able to put those together. How does that happen in beets?

In a non-beets world, I would simply copy the file into the appropriate directory. With beets, I need to update the database too. Again, with complete albums, I have taken he position that beets will manage the filesystem. I placed files into (essentially) a staging directory, pointed the import there and fired up the import. How would that work with a single file? If it was identified as an album with all the other tracks missing, would I be able to ā€œmergeā€ in to the existing album?

12

Unfortunately, the most reliable way to do this is to remove the album from the library and re-import it including the new tracks. The ā€œmergeā€ option at the duplicate prompt does something similar, but relies on beets guessing the right album to associate with.

Again, the most reliable way is to remove and import again. Itā€™s not convenient, but it is predictable. Itā€™s also possible to try a ā€œreimport,ā€ but crafting the right set of files to import that way can be tricky.

13

Thanks Adrian. I have almost completed the transition of my library into beets, finished off the singletons yesterday. Your insight and some testing allowed me to figure out a workable workflow and that task is done. The last task is about how to identify singletons that might complete an album and move those over. That appears to be largely a manual effort (canā€™t get everything automated).

Looking for another hobby so I have pulled down the beets code. Probably more than I can accomplish, but I am always looking for something to do. The learning curve is steep; I am an old school programmer (C, C++, Delphi, etc.), Python canā€™t be too hard to pick up right!?

Anyway, I know herding lost sheep through the wilderness is a thankless job, but you and a couple of others have put the effort in to help and I do appreciate it.

1 Like