There's nothing wrong with Signal's centralization model in a worrying sense. It acts only as a clueless message relay, and it has near-zero information on any of its users, even as it delivers messages from person to person. The only information Signal knows is if a phone number is registered and the last time it connected to the server. There is great care taken to make sure everything else is completely end-to-end encrypted and unknowable, even by subpoena.
The only real issue with Signal's centralization is that if Signal the company goes down, then all clients can no longer work until someone stands up a new server to act as a relay again. Signal isn't the endgame of privacy, but it's the best we have right now for a lot of usecases, and it's the only one I've had any luck converting normies to as it's very polished and has a lot of features. IMO, by the time the central Signal server turns into an actual problem we'll hopefully have excellent options available to migrate to.
Also TMK, the only reason you still need a phone number for Signal is to combat spam. You can disable your phone number being shown to anyone else in the app and only use temporary invite codes to connect with people, so I don't count the phone number as a huge problem, though the requirement does still annoy me as it makes having multiple accounts more difficult and asserts a certain level of privilege.