I think it's the opposite, they got rid of the jack to simplify their design and follow the general industry movement, and as a compensation they offered earbuds, and later wireless headphones. I also dislike it, but they are still the most ethical option we have.

They wrote about it: https://support.fairphone.com/hc/en-us/articles/9836188988049-Audio-Jack-3-5mm

Factors that have tipped the balance towards exclusion of the headphone jack are: modularity and its influence on the phone’s size and weight, market and legislative trends and longevity. We had to sacrifice the headphone jack to make the modules as accessible as possible. It was a tough decision to make but there was simply no space for everything and we had to make some choices. In the end, in order to build a modular, 5G device, this was the best way to go for the majority of users. If the phone were bigger than it already is (and we get a lot of feedback on it being too big), it wouldn’t be commercially viable and that would limit the impact we could have on the industry.

Any decision we made would have supporters and detractors. Our main goal, however, was to ensure that we made a product that combines repairability with industry-competitive dimensions. On top of that, we wanted to create a product that lasts for at least five years. We also wanted to attract a growing audience, which would allow us to make a bigger impact on the industry.

In the current user feedback (source: Trustpilot) we see more people being critical about the current Fairphone dimensions than about the fact that they have to use an adapter for using devices with headphone jack. Hence, at this stage we would not consider reinstating the headphone jack in our upcoming devices. [more details in the article]