I am not assigning it to "pyStream" , for txt file i am using show-property in activity to display the contents of the REST service response in browser ,which opens up correctly , issue is with files other than type txt .
Not sure I understand. Show-Property wasn't meant to do what you are trying to do. SendFile API sends the file to the browser directly. Storing the output of the API and displaying it with show-property won't work. The appropriate headers needed for this to work won't get sent. The response to your REST request should be to respond directly with the sendFile API.
You should NOT have to set up the HTTP Headers map: you should let 'sendFile' build these automatically given the filename (including the file extension).
You just need to have a Byte Array loaded with the contents of your Binary File; and then pass this along with a filename to the 'sendFile' API.
Take a look at the OOTB Activity 'Code-Pega-PDF.View' to see how it is able to stream a PDF back to the browser (the flags govern whether the 'attachment-disposition' are set or not - 'Code-Pega-PDF.View' sets them so that the PDF is launched by an external viewer [usually] , rather than replacing the current content in the browser).