Bread molds faster in room temp since bacteria has a better living environment than in the fridge. You see the fridge keeps the component cool disallowing bacteria to form as fast, however the coolness would eventually cause the bread to stale. I would prefer keeping the bread outside in room temperature in order to have it nice and soft and ready to be toasted (and sacrifice the 2 slices which might mold). Of course all of which in comparison to keeping bread in the fridge which will get hard and stale as the movement of taking it out and back putting it back in occurs.