You might be able to do this by having two MQ listeners configured with different Service Rules that cater to the respective message types but the issue you will run into is that the listeners won't distinguish between the two message types when they read them end up discarding the message unless retry is configured to replay the message until the right listener gets to it.
So, a neat solution will be to have a separate queue for the different message types.
I am facing the issue which you have mentioned. This is upgrade project we are working on (from 5.2 to 7.1.7 ). So We are listening from the queue where multiple systems posts messages. some of them are putting it as TEXT and others are putting as bytes.
Any suggestion on this other than creating new queue.