As a Firefox dev (I'm still working at Mozilla, although not much on Firefox atm), I have seen many, many occurrences in which I couldn't optimize codepaths, or even in some case fix bugs, because the old extension mechanism made it impossible.
Consider the necessary steps:
1. realize that an internal API is broken;
2. come up with a new non-broken API;
3. port all the internal code using the non-broken API;
4. add a compatibility layer between the broken API and the non-broken API;
5. check all the existing add-ons to find out which ones use the broken API;
6. hope you didn't forget any add-on;
7. attempt to get in touch with all the add-on developers;
8. repeat 7. many, many times, until you are sure that the add-on developers that do not respond have simply abandoned their add-on;
9. negotiate a transition plan with the add-on developer with whom you have managed to get in touch;
10. land the patch that you have written now 3-4 months ago;
11. maintain both the broken API and the non-broken API (and their tests) for ~1 year, until you are reasonably sure that all add-on developers who intend to migrate have done so;
12. maintain (and test) a downgrade path for people who switch between versions of Firefox;
13. finally land your code;
14. realize that you still have accidentally broken some add-ons and people are (rightfully) unhappy because "Firefox broke my add-on";
15. it's 18 months since you wrote your 2-lines patch, you can finally get rid of the dead code and tests and move to something else.
This was one of the reasons for which the Chrome teams managed to be faster and more efficient than the Firefox teams (well, that and a bazillion dollars to hire way more people). The add-on architecture is the main reason for which projects such as multi-processes only landed ~8 years after we had working prototypes and some other performance projects never landed at all.
So, yes, removing the add-on architecture is definitely painful for a number of Firefox users, but I believe that we could not postpone it any further, even if it meant that some useful addons could not be ported immediately. Also, for what it's worth, we have postponed it by something like 7 years already