I don’t think it would be an issue. Beets only activates plugins that you list in your config file, so just having the files around shouldn’t be harmful. And pipx installs to an isolated location, so I think they won’t have an effect, in any case.
Just a little messy & taking up space on your hard drive, but that’s life when using any of the mainstream packaging systems (projects like NixOS and rpm-ostree/Fedora SilverBlue are trying to solve this, but I’m not sure they’re ready to recommend for most people yet).
If you do end up running into issues down the line, you can look into what files are provided by a package to make sure they’re fully uninstalled (as long as you remember the package name).