I have implemented forgot password functionality in Pega 7.
Following are the steps that were followed-
Create a Service HTTP rule that accepts HTML post data as input (which is typically the username).Configure your service activty to set a custom password or random password.
Next insert a link in Web-Login to open a static HTML (not Rule-Obj-HTML) on click of that link.
In the static HTML configure it to be submitted to the Service HTTP URL.
As per response set from the Service HTTP rule you can display screen if password reset is success. (Before displaying screen you will most likely want to send a mail to the user which can be done in the service activity rule).
Use DSS combined with if then else to dynamically determine the URL for the various environments.