Try using the below Java Script from any of Pega HTML rules and see if this works for you or not. Not sure whether your client allows to use this ActiveX concept from the browser stand point.Usually clients won't accept to make browser side settings on ActiveX but it is up to you.
var oShell = new ActiveXObject("Shell.Application");
var commandtoRun = <%= tools.findPage("workPropPage").getProperty("pyWorkPage.AppPath").toString() %>;
I don't think you can rely on browser's allowing this kind of thing going forward: it's regarded (quite correctly I think) as a security risk for a browser to run arbitary, unsigned-code / local executables on the client desktop.
At the very least - It won't work outside of Internet Explorer either of course (ActiveX-based).
My system blocked the HTML when I tried it on IE11 on Windows7 (I'm glad it did: since 'notepad.exe' could be something worse!)
I tried this Java Step in an Activity (it's a mangled-version of the Code-Pega-PDF.VIEW activity) on a PRPC6210 system (which I happened to have running - it should still work on later versions of PRPC).
The second scenario - where you launch notepad from a browser; and save the data back - would to have to be done via a (CA-signed) Java Applet / ActiveX process and is much more complicated for me to come up with I'm afraid !
(My question really should have been : what is the point in launching notepad.exe from a browser; if you can't save the data back somewhere?)