The easiest way to build assertions in a PegaUnit Test Case is to assert values on the primary page of the rule under test. So, try this:
Implement another "wrapper" rule 'W' that calls your Data Transform 'D'. 'W' could be a Data Transform or an Activity.
Just like in your real application - you will need to turn on "Pass current parameter page" from 'W' when calling 'D' to be able to access the output parameter value set by 'D'.
Have the parameters to 'W' match the parameters of 'D'.
In rule 'W', map your output parameter to a Boolean or Text property on the Primary page of 'W'.
Implement your Test Case on 'W' instead of on 'D', and assert the property you have set in #3 above holds the expected value of the output parameter.
A similar approach can be taken to implement PegaUnit test cases on Function rules. Implement a wrapper rule to call it, post the return value to the wrapper rule's Primary page and implement the PegaUnit test cases on the wrapper rule.
Implement Wrapper rules in the same Test ruleset as your Test Cases, as these rules will never run in Production and only be invoked as part of executing test cases.
Consider also whether it would be reasonable to design your Data Transform so that it sets the 'result' to the clipboard instead of to the Parameter Page. This is a more natural usage of Pega and would make 'D' testable without a wrapper.