Ok when you get PT running go to "setup" (next to option and window on at the top of the PT window) and "playback engine" and select the 11r as the play back engine, now go to setup and I/O in that window select "input" tab at top left now click "default" button at bottom left, then click the "output" tab and click the "default" buttom at bottom left, on bottom right make sure the Monitor path and the audition path are 11 rack as well as any others there, click the "buss" and "insert" tabs and do the same....

any time you change the playback engine you need to go back and do this. there are times that I have used AISO4ALL2 for mixing on the road when I didn't want to lug around the 11r.

my first thought when reading your post was that maybe you have a bad stick of ram or you need to reinstall PT... make sure you have the "Avid Application Manager" installed it keeps all you PT files plugins and drivers up to date and will install them for you.