We also faced similar issue and did below changes for Email Account, advanced section of receiver:
Set email/partialfetch dynamic system setting is set to true. This setting controls whether to use IMAP partial-fetch.
Set the email/fetchsize dynamic system setting to 1024000 bytes in advanced receiver. This setting controls the size of the fetch if IMAP partial-fetch is enabled.
For Pega Platform 8.3 and later, the default value is 1024000 bytes. For Pega Platform versions prior to 8.3, the default value is 16 KB.
Using this we were able to attach 15MB email attachments(max outlook limit) also to case using listener in lesser duration.