We can hide all the OOTB flow action buttons on selecting the option: Hide the default action section buttons in the respective flow action.
If you don't want to display the cancel button alone, then you can apply the visibility condition for the respective cancel button in the section: pyCaseActionAreaButtons. As this is an available rule please save this rule into your application ruleset after the customization.
For testing purpose, I am using the when condition: Never inside my customWhenCondition to demo the things. As per your requirement, you can use build some expression or when condition as the visibility condition to not to display this button for your use-case.