If you are using custom authentication then you can use 'pyChallenge' parameter with value as @java("PRAuthentication.DEFAULT_FAIL_STREAM") and provide HTML stream rule that you want to show on failure in 'Credential challenge stream' field in custom' tab of your authentication service rule.
You can use different values to handle the authentication challenges, I am attaching a file that includes different authentication parameters can be used to handle challenges.
I am getting an exception from Pega when I try this scenario.
I was getting an exception even when I made no changes to the number of login attempts. The param pyWasTimedOut does not have any value in the Authentication Activity . And the code succesfully completes the Auth Activity , but throws an exception in the web-login page.
I saved the AuthenticationTimeout (ID : AuthenticationLDAPTimeout ) into my ruleset . This activity is not being called as per the logs .
From the documentation, I assumed that this activity should be called if I set the Timeout options in the LDAP (webldap3) authentication service. Do I have to set some flag or anything specific so that the timeout activity is called with the param pyWasTimedOut correctly set ?