Mentally screen out distractions, like background activity and noise. In addition, try not to focus on the speaker's accent or speech mannerisms to the point where they become distractions. Finally, don't be distracted by your own thoughts, feelings, or biases.
When listening for long stretches, focus on (and remember) key words and issues.
When dealing with difficult people, spend more time listening than speaking.
When in doubt about whether to listen or speak, keep listening.