IndiesHD

Before writing this post, I searched apple forums, KB articles and other websites about AppleTV 5.1 playback and encoding. I had some confident after reading this wonderful article “Ten Myths of the Apple TV: 5.1 Audio”  from roughlydrafted.com.   Here is my solution to playback 5.1 audio via optical port from Apple TV.

1.  Encode Video separately via Compressor 3 default AppleTV settings  available in settings tab under Apple Devices -> H.264 for Apple TV or QuickTime Pro 7.3 export “Movie to Apple TV” settings.
2. Encode Audio separately via Compressor 3 “Add Surround Sound” with the following settings. Under Audio -> QuickTime Surround 5.1 -> Format (AAC) & Channels ( 5.1(C L R Ls Rs LFE)  & Target Bit Rate (160 kbps) in Show Advanced Settings )
3. Open encoded m4v and AAC 5.1 in QuickTime. In audio file window edit-> select all and copy then move to m4v video window and place the cursor in the first frame and go to edit -> Add to movie. Save the m4v video window with embedded audio 5.1 as flattened QuickTime movie. 
4. Sync with your AppleTV thequicktime movie with 5.1 audio in iTunes.
5.  Connect the audio via optical out. Thats it. No more steps. You can enjoy 5.1 audio from your Apple TV.