Close -> Provides an explicit close action for an overlay. Overlays, configured by specifying the Launch>Local ActionTarget as an Overlay, close automatically when the user clicks outside of the overlay.
Cancel -> Closes the current form without applying any changes.
This is what the documentation says about these two actions, hope it might be clear for you.
Thanks for replying. If I use a on click close action in a button inside a section without configuring any pop ups, will it close the work object? (Instead of "close", if I use "cancel" the work object is getting closed)
No it doesn't close the work object. Cancel button has the Transition options like None, Move Backward, Move Forward, Go Home, i suspect you might be using with default value None that is the reason it is closing the work object. But in case of Close action you wont have these options to navigate through the states of work object.
Series of OOTB rules gets called when user clicks on close and cancel buttons respectively in context of an Overlay or modal dialogue mentioned as below ...
As its clear from the above points that Cancel calls workunlock activities , so its deals with releasing lock for the case closing current harness etc, while close works on the context of the main WO from which the flow action has been invoked and once the current screen closes , the focus stays on the main screen without releasing lock for the WO.