It isn't necessarily the algae on the glass that is the problem, it's the free floating algae that turns your water green that you have to be concerned with.

What kind of fish do you have? Some fish like to graze on fixed algae such as can be found on underwater surfaces such as the glass. Check that out before you clean the glass except for an area to be able to see the fish that is.