I finally got Blu-ray video working!

I'm not sure if it was because the source video was 1080 interlaced or telecine encoded but forcing the framerate during conversion fixed the playback problem with my blu-ray player.

Because of one of the two above, some players thought the framerate should be 60 (or more specifically, 60000/1001) so adding "--fps 30000/1001" to the x264 conversion fixed things.

The only oddity is that I had to add about a 275ms delay to the audio to get it in sync for playing on my computer (vlc or totem) but for the blu-ray playback it was in sync without any delay.

For Thomas:

I tried adding the dvd_obs=64k option but it burned at 4X anyway... I'm not sure what's going on there, but it works so I'm leaving it alone. I've decided that for BD-R's that it's best to fix your problem if you're getting bad burns, not rely on defect management. 

To summarize the process:

Use x264 through ffmpeg or directly to convert the video to H.264 per the recommendations on www.x264bluray.com. Add the -fps flag if you need it. Unless you have a REALLY fast machine or an eternity to wait, use the slow preset instead of veryslow.

Extract the audio, I used ffmpeg to extract the existing AC3 audio.

Use tsMuxeR to create the proper blu-ray folder structure and mux'ed stream (unfortunately not FOSS, but free)

Use ImgBurn under WINE to create the image for burning, which supports UDF 2.50 for blu-ray (and 2.60, but that's not needed).

A couple of side notes:

My ASUS BW-12B1ST blu-ray burner has worked great without issue, can't recommend it enough.

Using cdrskin which is a frontend to libburn has also worked great. No need for the official cdrtools. Not that there's anything functionally wrong with cdrtools, but it keeps me out of the whole cdrtools vs cdrkit debacle. 

Thanks,

Richard