This setup was my original thought as well, but, like you said, I have no idea if this would be "safe" or not, ha.
I'm fairly sure this would put the power sources in parallel, which seems to be a bad idea?
Edit: Just realized, technically isn't every fuse really coming from the battery...