You could possibly use Sybase as an external database to the Pega platform and some customers have done that. The following support article showcases that. That said, we do not explicitly test our features with Sybase as it is not part of our platform support guide.
There was a scenario a few weeks ago with a customer who had Pega 7.2 on WebSphere and DB2. They had configured a data source in WebSphere Admin Console against a Sybase database (specifying their Sybase driver - jconn3.jar, I believe). Then they created an external database rule in Pega. They were running stored procedures that had cursors in them. Their issue was resolved by adding a custom property on the WebSphere data source - skipCheckForUnprocessedResults and setting it to "true".