Mail.app is capable of handling a huge amount of emails. But, just in case it slows down, the Mail Slimmer applet is what you need.
Mail Slimmer is a free and easy to use, AppleScript based utility which will allow you to both rebuild Mail.app`s database indexes and trash all saved e-mail attachment files with a click of a button.
When a mail attachment is opened by double-clicking on it directly from mail body, its file is saved in “Mail Downloads” folder. All saved attachments can be safely trashed, because the original attachment is still located in the body of the original email.