I have removed it, but only to re-apply thermal paste. The side of the case is kept off to try keep temps down, but has no effect anymore.

Don't think dust would cause that much problem, but I cleaned it off a bit when I applied the thermal paste.