I went through essentially the same thing a couple months ago. Tried Calibre (and Calibre server) since everyone recommended it.
Really disliked it. Calibre is great for converting ebooks, but has shit management and webserving capabilities.
I ended up with Kavita and am super happy. On the web client, both management and actual reading are a pleasure. Any phone/tablet client supporting OPDS works perfectly to read/download your manga/books from the server.
And a select few clients go a step further, supporting Kavita's API, which allows for 2-way sync (effectively, syncing reading progress between all your devices).