The behavior is actually expected. Here is a quote from SME from an internal post with the same question:
The Test Connectivity button on the Connect SOAP rule form has had very limited applicability since PRPC 5.5. Prior to that, SOAP client initialization was a complex process that involved generating and compiling third-party java components. All of those "Not Attempted" status values are related to deprecated functionality from pre-5.5 generated connectors.
Furthermore, the Test Connectivity button doesn't actually connect to the SOAP endpoint URL. It merely tests whether the URL is malformed and stops there. Pinging a URL is not as simple as pinging a socket.
You should actually run the actual connect-soap to test it out the real functionality.