I can tell you are using PRCUSTOM style authentication.
The error: "No Failure response set by custom authentication" only occurs when the out parameter pyFailMessage is blank or null and no clipboard page was returned in the out parameter pyOperPage. Sending no value for pyOperPage with a failure message is pyFailMessage is how you do error handling. When both are blank it's an invalid state so we throw the exception.
From the tracer output you provided I can't tell you what is going on with out looking at the activity. You have a customized version of AuthenticationLDAPVerifyCredential in you unauthenticated ruleset.
It's either logic skipping the Property-Set step that sets the param.pyOperPage = @java("myStepPage")
In your error handling block there is no step taking the local.errorMessage and setting it to param.pyFailMessage.
Your solution worked for me. inmy case "error handling block there is no step taking the local.errorMessage and setting it to param.pyFailMessage" this was not happened. i have resolved the issue. Thank you Chris.