To do this you need to go into the group that you manage and then look for a 'group members' link in the left hand menu. From there if you click on the group members profile image a dropdown will appear with the option to remove that member.
Unfortunately there is no way to override notifications for users.
One way around it is to upload the files with an access level of private initially and then go back and edit the file and change the access level to what you actually want it to be. Setting it to private initially will bypass the notification.