Amit, when you say you are "not getting expected result”, do you mean the NTLM authentication is going through fine but the service isn't returning the correct response or have you started getting 401 (unauthorized) error?
No, i am not getting unauthorized error. The service is supposed to perform some logic and then return a boolean. In our case it is always returning false.
Same is happening even from SOAP UI.
The headers from SOAP UI indicate that the request which is sent is as expected and there is something wrong at the service side.
But what is interesting is that when the same service is called from the their existing application (Biztalk server) then it is working fine with similar set of data. So i was looking for if there is some Biztalk experience available to help us debug and understand what is sent from Biztalk server.
Are the requests sent from Pega SOAP connector, soapUI and Biztalk server exactly the same? As John recommended, have you tried sniffing the requests using TCPMON? If the authentication is going through fine, then I suspect there is some difference in the request. Either the difference is in the actual SOAP envelop in the request body or in the HTTP request headers. TCPMON should give you this information.
Another question for you: is the request from Biztalk server going directly to the IIS service or is it going via a proxy? If it is the latter, perhaps the proxy is adding something to the request that makes it work?
It is sometimes useful (and sometimes necessary) to get at the HTTP traffic itself for troubleshooting a SOAP/HTTP issues such as this.
I usually use the (very old, not-maintained, but still useful) Apache 'tcpmon' tool for this: basically you can proxy your PRPC SOAP CONNECTOR through it and examine the response : kinda like using 'Fiddler' for Web Services.
The basic approach is to fire up tcpmon as a 'listener' (not a proxy) and change the host/port configuration on your PRPC Connector to talk to that endpoint. *
You then configure 'tpcmon' to forward it's requests to the original host:port of the original endpoint: and then read off the HTTP traffic (including HTTP responses and headers etc).
*[ Sometimes this simplistic approach doesn't work : the endpoint may insert the original endpoint hostname (via HTTP headers) in the traffic, which will cause the client to attempt to contact the original endpoint...defeating the object of this diagnostic.
In that case: you can usually get round this by having three physical hosts (prpc, endpoint and tcpmon): edit the 'hosts' file of the prpc host, to 'lie' about the 'endpoints' IP address - instead you map that to the 'tcpmon' host.
From the 'tcpmon' host, you just set up the normal forwarding. ]