Time to rethink path formats?

Development
1

The path formats feature is great for basic use, but it gets messy quickly. I want my music library organized just-so, and the way I do that in beets is cumbersome. Maybe it’s my Python bias speaking, but I think a pure Python solution would be easier to follow than a translation layer between text and the underlying Python.

In my config, I define a fake comment and fake tab using regex, that allows cleaner line breaks that line up with the surrounding lines.

Then I spin a tangled web of if-elses, which are hard to understand. Without line breaks, it’s impossible.

Granted, nobody but me does fake comments. Can people with advanced path formats share their configs? Do you create them in your YAML, or do you have to conceptualize them elsewhere because the if-else chains are too messy and there are no comments?

Should I put all the logic into inline's album_fields and item_fields? The complicated functions go there but my base logic is still in the normal path formatter. Are template functions available when using inline?

2

Interesting! I’d be interested to hear how others tackle this sort of thing too. It is perhaps worth exploring the “all-inline” approach to see how far it can take you.

I don’t think it’s currently possible to call template functions from within inline expressions, but that would be a cool thing to add to support this sort of use case!

3

Ah, I had forgotten we had recently discussed this possibility and found a tricky way to do it:

4

Kinda off-topic, but sometimes I wished that the path: entries were Jinja2 templates. But that could open another can of worms

1 Like
5

I tested it and feel good. Need to do more tests. I have a few hundred imports coming eventually™ and it should be a good chance to torture test this concept. After this initial test, I feel better about Python/some other system and worse about the current setup. However, forcing users to do it manually is a no-go, if my experience is an indication.

pros

  • incredibly readable: I can set up my paths like a regular python script and loosely follow PEP8
  • Python engineering is available. I can autoformat, peek function definitions in VS code, etc.
  • comments and line breaks are VIPs and don’t need regex hacks
  • possible to write a script to convert from existing path format to Python???
  • conservatively, 2x faster than my old system. see below.

cons / notes

  • took 4ish hours. including some non-mandatory light re-engineering, but also stuff like remapping $myfunc{album} to myfunc(album) in the definition and the function call.
  • confusing importing my own plugins (could be my sloppy, ancient dev environment)
  • how do i import base beets path functions (aunique)?
  • repeated string appends (s += albumartist): this is probably where a dedicated template language thrives
  • sequential line breaks are not allowed in the “: |” format I was using; so no space between functions etc.
  • VS code doesn’t know anything about beets vars (items, album, albumtype, etc.). it underlines all these variables in red because this would be invalid python. get around this by importing something from beets? (The “# Item fields.” and “album fields” sections of inline.py look promising.)
  • syntax highlight breaks when i actually paste it into config.yaml (everything is highlighted as a long string). get around this by dumping everything in a Python/beets plugin and just calling that with inline? Update: Getting the imports working with beet fields like album is non trivial, at least for me.
  • pdb.set_trace + dir() doesn’t show anything for item fields? (works perfect for album fields). all the item fields are available but you must know them by name (ex. mb_releasetrackid).
  • Is there anything that base Python3 handles worse than the beets path formatter? Unicode? Case sensitivity? Dupe handling I think happens one layer above, after the path is already set.
  • exceptions / parse errors don’t indicate where the error occured. The error mentions “line 70”, so the error output could just be improved to visually show where errors are. (Something to improve in inline)
  • only tested on windows
  • mixing album / item fields is annoying. I have to call 4 functions: album1, item1, album2, item2, because of how I set up the field order. (I want media in the folder name, even though it’s item-level).
  • The / path separator is interpreted as a regular /, so subfolder creation is broken if you do it within Inline.

Performance

i’ve moved all my album fields to a single python inline call. is there something about inline that might cause poor performance invoking 10ish functions? 8ish of them are very basic 5 line string manipulation functions. is compile_inline the culprit? Edit: Or maybe commenting out aunique does it? Or the regex in the path formats?

Comparison

Here’s what I have now. I rewrote the item functions and everything seems to work on 2 album moves. The Python version is a tiny bit bigger, but much more readable. The length isn’t directly comparable anyway. Like I said, I changed the code a bit as part of the conversion.

6

Thinking about this again. Disclaimer: This all looks confusing to me, which is coloring my thoughts, but I don’t tend to work at this low of a level. Maybe everything is fine. I believe I found where the magic happens.

Note that there are a few spots that mention how hacky the template functionality is. I don’t get the sense anyone wants to support this. It’s an interesting idea to lower the barrier to entry but I don’t think I would design it this way today. Easier entry, much harder advanced usage. And with all the Python guides and software today it’s arguably harder this way than with pure Python. For that reason, I don’t think templating stuff like Jinja2 should be used either. It seems like it arbitrarily cuts the available support for little/no benefit.

Some of the functions show another reason why we should go Python. They’re just another layer of abstraction and confusion onto simple Python concepts that we should be helping users build intuition and Google-ability for. If I want to know the ins and outs of Python’s string.upper there are hundreds of guides on Google, a thriving dev community, and probably the source code. If I want to know the ins and outs of Beets’ upper() I have to search through Github and follow a web of the same function being passed through complicated renames (it’s inside DefaultTemplateFunctions and is called templ_upper). If I want help there’s maybe 20 people in a given month who have any hope of helping me. The user also never sees the docstrings we have because as I said in past posts, there’s no beets integration with your IDE to peek at the definitions.

Hot take: path formats should be stored in something like beets_paths.py, to get all the autoformat and peeking goodness from your IDE without clunky copy pasting at the end. Doing Python in YAML is borderline putting a round peg into a square hole. A compromise would be having this be an optional feature. I guess the closest concept we have now is plugins - every user should have their own local plugin where they define their paths??

I think we should go pure Python. Steps to do this:

  1. Ensure no functionality is lost by doing so
  2. Export(?) the default functions (aunique, left) to something like beets.pathfuncs so people can do import beets.pathfuncs and call aunique on their paths
  3. Create a seamless way to convert existing path formats to Python (it’d be messy, but in theory this is already done since functemplate.py already converts everything to python in the background?). The ‘test coverage’ on this should be pretty good- beet mv before and after the conversion should have 0 new paths. If that test fails we put a note asking to pretty please raise an issue on github/discourse/email.
  4. Automatically set this up, comment out the old way in config.yaml, and leave a warning that the old way is deprecated.
  5. Some time later, remove the old functionality.