Pega File listener sometimes not able to move the processed file to completed folder and the different nodes are picking the same file again and again and processing it. Due to this we had duplicate entries , sometimes even the file processing by file listener is getting failed but not hint about the reason.
Initially we thought files is not being processed at all, may be due to larger files size (270MB) but read that there is no such max cap on it.
Later identified that file is processed multiple times but not able to move it after successful run. File permissions, name (unique from other files) etc. looks fine.. also other files are getting processed successfully. The issue always lying with only some files. data, header, trail everything is fine.
When I read the article below on the same issue, it is advised that issue may be because Linux operating system holding the lock. We haven't opened the files, no other process is consuming this file. What might be chance that Linux operating system holding the lock on this file and hence Pega is not able to move the file?