What version of Fedora are you using?
F24 x86_64 (clean install). It doesn't use that much memory for me - I have 16 GB, and
after a day or so it gets to as high as 2%, restarting gets it down to about 0.8%. I got
into the habit when the machine had less memory and it would get up to around 10%. My
original post was just to mention restarting, in case the OP was unaware of it - it's
easy to do, unlike logging out or rebooting. Even if it used a lot more, I wouldn't
bother changing desktops because of it, unless it got wedged somehow if I forgot to
restart.