So if you have RSA please configure a connection with such keys, and then test connectivity. I found couple more thing which may help you to configure the connection, please take a look:
Host Name - Enter a Domain Name Server (DNS) name for the FTP server, for example, ftp.myserver.com. Alternatively, identify an IP address. This field supports the Global Resource Settings syntax (=PageName.PropertyName). The default property name for this field is pyHostName. For more information, see Using the Global Resource Settings feature.
Port Number - Optional. Identify a TCP/IP port for this FTP server. The system automatically selects standard ports at run time if they are not specified, such as port 21 for FTP and FTP Secure (FTPS), and port 22 for SSH FTP (SFTP). This field supports the Global Resource Settings syntax (=PageName.PropertyName). The default property name for this field is pyPortNumber.
Use Authentication - Select to enable authentication.
Authentication Profile - Select the authentication profile to use that contains the user name and password for the account that Pega 7 Platform uses to connect to the FTP server. Click the Add icon to create a new authentication profile. For more information, see About Authentication Profile Data instances.
Select FTP, SSH FTP (SFTP), or FTP Secure (FTPS) based on your FTP server configuration.
SSH FTP (SFTP) Settings
Truststore – Provide the SSH public key of your FTP server in a Truststore record to restrict connectivity to this known host.
Note: The default behavior is to trust all hosts.
Keystore – Identify your FTP client by storing your private certificate in a Keystore record.
FTP Secure (FTPS) Settings
Lowest allowable SSL/TLS version - Select the lowest allowable SSL/TLS version. Pega 7 Platform will display a guardrail warning for versions that are considered vulnerable.
Truststore - Select the Truststore record that contains the server certificates to use for FTP server verification or click the Add icon to create a new one.
Keystore - Select the Keystore record that contains the client certificates to use for Pega client application verification by the FTP server or click the Add icon to create a new one.
Encrypt data channel - Enabled by default for encryption of FTP authentication and data channels. Select to encrypt data; authentication over FTP always uses encryption. If you are using mutual SSL or a secure network, you may want to deselect this option as performance can be impacted when using it.
Test Connectivity -
After you have completed this tab and saved the rule, click to confirm that Pega 7 Platform can connect to the server by using the information on this tab.
The system presents test results in a separate window, identifying the parameters used in the test, the steps attempted, and the outcome of each step.