Bitcoin is compact because the data is immutable and has to be stored by all nodes. Nostr only has to store data where people need it — size is not as important when eliminating redundancy is cheap and pipelined, adding mere microseconds of latency.
I'm not disagreeing in principle with the benefits of binary encoding. But between SIMD hex codecs and hash functions, flate compression, and the broad support without any of the complexities that gRPC/protobuf ties you to as a developer, I would argue to skip the "use a good existing format" step.
The binary format I already use inside the Badger database is, as I mentioned, extremely compact and includes a mechanism to string-intern the pubkeys (this is a huge saving for follow lists). It is already extremely fast, the algorithms are small and simple, and already battle-tested on WoT database usage.
Signature verifications, however, are problematic. A canonical encoding used to create the signature is necessary, no matter how you wire- or disk-encode it. It's the biggest friction point for Nostr moving to a better, binary format.
It's also why it was so critical to make that codec as fast as it can be. Part of the trick of the performance is that it's a subset of JSON that it encodes — only the part relevant to the exact syntax of Nostr events. It will swallow incorrectly formatted, minified JSON happily, if the extraneous characters are not expected next in the mostly-forward, no-lookahead decode process.
Lookahead is the biggest reason why most JSON codecs are slow. It's also a consideration I took into account when I was pondering the idea of a human-readable but compact format that uses linebreaks instead of brackets, which saves a byte for every branch in the syntax tree.