As I continued my investigation of the ToggleButton issue mentioned in my previous post, I discovered that if you set the Focusable property of each ToggleButton, the border left on the pressed button when pressing any of the other buttons disappears.
The issue though with this approach is that the ToggleButton may no longer be selected using the keyboard Tab button. If this is acceptable in your project, then this is a great way to solve the issue.
Again, this issue is reproduced on Windows 7 machines. I'll investigate if this holds for Windows Vista/XP machines. If not, I might submit a bug report.