If effect doesn't work check “Image processing” via Install Tool. Probably “combine” command have failed. In this case go to the “All configuration”,  find  property "im_combine_filename" and set this value to "composite" . Then clear cache and try rounded corners  again.

No way to use this in RTE.Could be some problems with animated gif-files.