There should be world specific mods in the world save so it would be easier to share modded maps. Something like how world saves can now load resourcepacks so the mod dir would be something like,
-saves
--worldofworlds
---mods
and yes I know that some mods wouldn't load like CustomMainMenu and it would also most likely make the saves massive and slow to download but it would still be pretty cool as a feature also it would help a little with people using different minecraft directories just to separate mods.