Think I have found the issue.
It's most likely the way I use `removeObserver`. I need to pass the observer token to `removeObserver` to make sure the correct observers are actually removed. This means I need an array to keep track of the many tokens from all the different formatting notifications.