OS update fixed it. Weird.... thanks for all the help! Went from 10.7.4 to 10.7.5 and suddenly is working. So far all shares have been rejected. but now I can at least play with the settings and see what works best instead of having to reboot every …
Same issues with work size256 and -g 1. RAM and CPU usage are being allocated to CVMCompiler actually, which looks like it's part of the OSX OpenGL framework?
Just realized you wanted me to switch algorithms... Switched to the panda pool BTC SHA-256 and it's working fine on GPU. Getting around 300Mh with standard settings. So it looks like scrypt is the problem.