Giving no access to "Open" for that case type for all roles should work. This will create/modify Access of Role to Object instance for the case type class and giving 0 to Open instances access control.
You can try the same setting manually. Let us know if this helps
Access Manager should be straight forward and work as per configuration. If not, try this:
Each case type eventually have one starter flow. So, you can go to your starter flow (Pega 7 and above it's pyStartCase if you leverage Case Designer) rule definition, Process Tab and use Start Conditions and/or security options. If you use security option, only users who have at least one of the privileges listed to use this flow.