That is actually a really good clue. Upon closer inspection of your code I see you are explicitly setting a message within the edit validate itself. Can you please try removing this as a quick diagnostic and only returning true or false accordingly?
I'm not sure if this test was done properly or if we misunderstood each other. So the first test, if you change your custom LongNumberValidation to remove the additional explicit addMessage
catch (java.lang.NumberFormatException ex)
You reported the issue was still present. Is that correct?
If so, what I want you to do is private checkout or save as the OOTB NotFutureDate and put in your original code there. Does it reproduce when invoking NotFutureDate? If yes, it seems tied to this code. If no, then checkout your custom LongNumberValidation and put in the OOTB code for NotFutureDate to see what the results are.
I'm not sure if you performed this secondary test as described. One of these two outcomes should help drive next steps.
This is just a work around, if the above issue is still unsolved puzzle with OOTB vs Custom code.
While you are launching modal dialog window in repeating grid,
1) Utilize 'Using Page' option. Specify a temporary page. Now, you modal dialog window works on the temporary clipboard page.
2) When clicks on submit, if temporary page has no messages associated to the page, then you can copy temporary page data to your Work Object's page. If there error messages associated to temporary page, then do not copy. You may write these steps in Post activity of flow action corresponding to Modal dialog. Anyway
3) When you do cancel for whatever reasons, your modal dialog goes away and thus the errors also. If you re-launch the modal dialog , make sure you do Page-New / Page-Remove of temporary page.
If you need to retain data on temporary page (in case of abnormal close of WO), you may need to embed this temporary page to your WO.
Hope this helps,
If you elect to clear down the messages on Cancel, be careful of the data on the clipboard. "Cancel" from a flow action modal will not revert clipboard changes made. You will find that if you have posted values on the clipboard (which you evidently have because of the error on the clipboard), then these will be kept. Again you would need to revert your changes through the extension activity on modal "Cancel"...