If you increase the replication factor, then more nodes need to be contacted when you do a "write" for instance. So it is going to be chattier in the network. There would be more "write" overall in the cluster so the cluster will be slower. You also lose in terms of space. You only win in terms of availability.
For instance we a consistency level set to 1 and a RF of 5, the data will still be available in 1 node even if 4 are failing.
If you decrease the replication factor then the opposite happens.