From your question looks like you need a Connect-REST at Pega end. Through Integration wizard when we run it for REST integration, I get follwoing error when I test the response. This means only XML or JSON response is supported currently.
"The response was not recognized as JSON or XML and cannot be saved."
Another way would be to use Java Script (ajax) to download the excel in server and then use MSOParseExcel to parse the excel file and create work object.
If my understanding is correct, Connect-REST is used when Pega needs to access an external service. Pega sends a request to the service, and service sends a response to Pega.
The problem statement in my case that Pega should host the service. Some other application should call the service hosted by Pega and in that request, they should send an excel file as attachment. Now the problem is where and how do I map the incoming attachment?
I would be curious to know how the clients will POST the excel file through a REST API.
If you want to build a REST-Service then as a service owner you can decide what type of Request (usually Query string ) and Response (JSON, XML, EXCEL, CSV) best suits for you.
I have not heard of a RESTful API accepting EXCEL as input/Request (I have heard the other way as Response ). There is no OOTB way in Pega REST-service as far as I know.
One of the standard ways would be to go with a REST-Service which accepts an query string (xml could be a large string as request, pega REST service parses it and creates the work object and responds with work object ID and or Status and any other details.