For closing modal dialog you can use pega.u.d.hideModalWindow java script function. However can you explain your exact use case what you are trying to achieve here. May be we can help with a better solution than using custom js.
we have three buttons on on modal dailog save,submit and cancel and some required fields . on click of submit button required fields valdation are not thrown and our activity is called and pop up getting closed we have tried save/update/close actions onClick noting is working so i am writing an pragraph rule for this