Hi, there's no OOTB way to do this. Customer Service uses a hybrid document model, so that limiting the DC to 1 document wouldn't work either. Since each service process is a separate document, that limit would not allow you to run any service processes.
It's probably not as simple as a single place to add this in. Basically, any place where you invoke the CreateWork API for the interaction, you would need to check if there's already an interaction open. That check can be as simple as checking the number of iFrames that are in the Window object. If there's only one iFrame, then that is the home tab and implies that there are no interactions open.
For a phone call, you can check the ScreenPopInteractionStarter HTML Fragment for an example. In the latest release, we had added some code to check some other properties before launching a call. A similar pattern could be followed. If the customer is using some other mechanism to start the interaction, the same check would need to be added there as well.
If you are looking for a rule to add some new JS functions in, you can create a new file to hold the JS and add it to the pega_cpmInteractionPortal_scripts bundle. Also, all of these comments are assuming you are using a fairly new version of Customer Service (like post 7.1.3). Otherwise, none of this applies.