StereoscopticTV - we need a new output script.

[Rhoel]

Last year I posted on this subject and the need for developing a tool to facilitate 3D output: Just had a conversation with some techie friends in LA. In addition, we have a guy in from Hong Kong with a 3D real-time digital recorder and edit suite. There are some major developments happening right now which we need to be aware of.

3D movies have been around since the 50's and the fad died out. The reason was mainly a failure in technology and the difficulty in cinema projection. Digital cinema, 3D enabled TV sets and HMD have all resolved those problems.

As a result, the big movie producers are pumping big money into HD3D film production ... in the last year, there were 80-90 features in production or finished. Dreamworks, Cameron and Spielberg are both heavy into 3D features. As one studio exec commented, 2d cinema is like a ipod with one earpiece. Its the natural progression from HDTV and will be as big as the conversion from silent to talkies or black and white to colour.

Animation is by far the easiest format to output as 3D. The subject is static and the spatial offset easy to calculate: Left and right output are simple rendered on separate passes then encoded later downstream. Currently the TDVision codec (used in the Interview mode of H.264) Looks like the front runner for the standard - it can accurately display 1080p60 ... that's 8Gbit/sec.

Anime Studio can output 1080 easily. I have already tested stereo output and the results are good - the Z axis of the program is a godsend, making the creation of stereo images very easy. The basics are simple. The camera is offset to the left for the left output, then reset to the right for the right output.

The complication comes when the camera is rotated horizontally or rolled. Then the amount of offset changes. Units are taken off the X axis and added to the z or y axis. The maths are not too difficult, basic sin(x) calculations work very well to work out the +/- positions.

What has beaten me is the AS scripting of the camera channel. I need assistance with this.

Are there any script kiddies here who are up for an OpenSource approach solution to build an new tool which will be used with AS to output 3D image streams. The GUI will be simple, an output left or right, and the interocular separation. The option of automatically rendering both passes to a directory would be a boon: The key need is to have a standard stereo render panel, which remembers its offset settings.

If there is interest, I'll make the threat a sticky or ask Mike to generate a new topic area for stereo development. For without a doubt in my mind, from what I have seen and talked over with the CE developers this week, stereoTV and cinema is going to be huge. The technology is already here.

Be good to be part of that revolution.

Rhoel

[synthsin75]

Is the stereo render option in AS not sufficient? Or are you looking to automate several renders with differing offsets? If so, AS doesn't provide any scripting access to the stereo render.

[heyvern]

Rhoel,

Have you done any scripting on this yet? Do you have a starting point to look at so we aren't reinventing the wheel? I am always interested in seeing someone else's math solutions because I'm so bad at it. If you have the math formulas maybe I could experiment to see how to apply those offsets correctly to the camera rolls and rotations.

Here's a crazy idea:

Don't do that with the camera. Put all the layers into a "roll/rotation" layer. When you need to rotate the camera, rotate the top layer instead. This won't effect the camera offset at all.

Not a great solution due to the fact that layers can seem to rotate "funny" and it's hard to control. But it might be a short term solution until a script solution is devised. Synth is right though.. not a single itsy bitsy teeny weeny reference to the stereo renderer in the script interface so any of this type of 3D must be done by manipulating the camera and rendering "twice". If only we could access those render frames from the stereo renderer somehow.

I wonder too if there might be a way to script the "double render" that is needed as well? I tried writing a script to "pause" the time line while playing so I could do "interactive tutorials", click or do something, then continue. I got sort of close but not quite. And it only would render each frame once. Not a big deal since you could just shift the camera for a second version of the file.

-vern

[Rhoel]

Many thanks for the feedback: The AS stereo option is Anaglyphic - the red/green 3D viewing system. While useful, is not a patch on what I am now seeing in development/deployment. That is full colour renders, up to 60 fields per second, full HD. It's seriously good. There is no discernible flicker on these 3D ready flat screens as the refresh rate is 120 hertz (giving you 2 * 60 fields per second refreshes). The flicker-shutter glasses from Samsung are under $50 and they are smart and robust. With the IR transmitter, the cost is still under $75. The thinking in Hollywood is with the financial downturn, people will spend more time at home, watching the expensive flat screens they bought before the crash. With the glasses costing so little, they think they can shift megabucks worth of Blu-ray discs. The argument over which format to encode is still being fought but the first into production (as always) will win. TDVision have a good system (codec) which looks like being adopted before SMTPE etc, get off the starting blocks.

Hevern, glad to have your input on this ... you are the leading light on scripting AS. My strength is mathematics. I can write python, pascal and php code okay but my Lua is not strong. Writing the formula is tough but do-able. We need to figure out if using a look-up table is the best way of doing this. Doing all the calculations separately then adding the sum might be the best approach: Ideas welcome.

From the tests I have done already, I know the left-right shift becomes a forwards-backwards shift as the camera rotates. Worse, as the camera rolls, the left to right shift reduces until the camera is 90 degrees, when the shift becomes 0. At 180 degrees, the left-to-right shift becomes a full right-to-left

The idea of an automated renderer is wonderful and probably a phase II development. Being able to hit the render button once and have two images streams created would be a great production tool. In phase I, doing it manually will simplify the design process.

Over the next few days, I'll start presenting the maths behind the calculation, so we can talk over how to best code the formula. Tonight I am on the sick - I have the flu and I;m shivering worse than a jelly on an unbalanced spin-dryer. Not the best condition for writing formula.

If Mike has any ideas or input on this, please make it: I have no idea what is already planned for ASP6.

Hopefully by Monday, I'll be clear-headed enough to upload some sin(x), cos(x) stuff. This approach should work well - I have used it before for converting angles to offsets. It's basic trig functions (Radian mathematics in fact). Working out the angle in 3D space just complicates things a little as it adds two extra dimensions (we have a roll, tilt and rotate).

Back to bed and shiver this fever out.

Thanks again for the fast feedback.

Rhoel

[Rhoel]

Okay, still suffering the effects of the anti-flu pills but I've made real progress ... confident enough to say the formula core is done.

Firstly, I built a 3D rig in Blender so I could see from all angles what was really happening during a rotate+roll. It was clear there was a combination of two axis involved in the move: My heart sank as this meant a compound calculation.

Sometime during Sunday, a little light lit up inside my drugged-up tiny mind. Yes its compound but I don't need to know how or why ... all the angular information I need to figure out the cosine stuff is actually contained in the AS camera. It's parallax in the other cameras so all I have to do is use the Roll or rotate angle and I know where the Left camera is on the X axis.

This is what is happening graphically.




The Rotate (Y) angle (in radians) drives the cosine(x) and sin(z) functions: It really is a simple as that. The same is true for Roll(z) and Tilt(x) functions.

Using that I knocked up a quick test and got the following (1 is maximum offset - these figures match the x-axis of the right camera rotating)

degree - x position
__0 ... 1.000
_15 ... 0.966
_30 ... 0.866
_45 ... 0.707
_60 ... 0.500
_75 ... 0.259
_90 ... 0.000
105 ... -0.259
120 ... -0.500
135 ... -0.707
150 ... -0.866
165 ... -0.966
180 ... -1.000
195 ... -0.966
210 ... -0.866
225 ... -0.707
240 ... -0.500
255 ... -0.259
270 ... -0.000
285 ... 0.259
300 ... 0.500
315 ... 0.707
330 ... 0.866
345 ... 0.966
360 ... 1.000

That 0.707 at 45 degrees is highly significant, it tells me the calculation is spot on. (its one of those magic numbers in mathematics: Anyone interested in photography, multiple f32 by 0.7071 a few times ..... cool huh! Now you know why I like seeing 0.707 )

To find the actual offset is easy, just multiply (interocular_distance/2) by the cosine value: Assuming the interocular is 66mm, we multiple 33 x 0.707 (at 45 degrees) to know the camera position is 23.331. We can find the height by doing the same for the other axis.

This simple formula means we can find the numbers on the fly, no need for a look-up tables (less work, I like).



Next step:
Vern, can you look at the AS_camera and tell me the names of the camera controls. Once I have those, I can start writing the full formula so you can put it into Lua.

There are a couple of other controls to add:
The interface GUI will have an interocular distance, divergence (default to 0 which most people should use) and the option of render left or right. That's all the information which is needed.

for those interested, the core calculation is

x_offset = cosine(radian(y_axis angle in degrees))
z_offset = sin(radian(y_axis angle in degrees))


I'll add another graphic later on the radian/sin/cosine relationship later. I'll also build a test scene which will test the 3D depth in roll and rotation.

darn I'm pleased about this, simple and elegant = it will work!


Rhoel (on a roll)

[heyvern]

This sounds great Rhoel!

Here are the camera variables that control movement. They are located in the section of the script reference called "MohoDoc". All the way at the bottom of that page.

fCameraTrack (AnimVec3)
the camera tracking animation channel

fCameraZoom (AnimVal)
the camera zoom animation channel

fCameraRoll (AnimVal)
the camera roll animation channel

fCameraPanTilt (AnimVec2)
the camera pan/tilt animation channel

Each camera thingy has an argument that is either, 1, 2 or 3 values that it needs depending on what it represents. So translation needs 3 values, X, Y and Z, even if one of those values is "unchanged" it must be set on a key frame.

So if we focus on fCameraTrack, The value must be an AnimVec3 which you create this way:

Code: Select all

local v = LM.Vector3:new_local()
v:Set(val1, val2, val3)

Then you can set that "v" value to the camera like this:

Code: Select all

moho.document.fCameraTrack:SetValue(frame, v)

In that example the frame is given as the first argument to indicate where the key is placed. It isn't listed in the script reference that the frame is required... but it is used in the "Orbit Camera" menu script that comes with AS.

AS works with radians, not degrees. However you can easily convert back and forth. However radians are "faster" in the code. Doing those conversions take more processor time... hardly noticeable from my experience though. I would stick to radians though. It might be simpler than converting back and forth.

Code: Select all

math.deg(radian value)
math.rad(degree value)

If you aren't familiar with lua math in AS here's my cheat sheet on the web:

http://lua-users.org/wiki/MathLibraryTutorial
---------------------------------------------------------

I would like to see what you think about how and when this script would be applied.

Would this be a a script run like a menu or buttons script?
for example:

You have a finished project and are ready render in stereo. You do a save as of the "normal" or default view, then you apply this script which offsets the camera, to this second file. Now you could render both in batch mode or in command line mode. This would probably make the most sense.

Another thing that needs to be done is to make sure there are translation keys at the right key frames to avoid any camera "drift". If you only key the rotation of the camera, and then run this script, it might not have enough translation keys to prevent the camera from drifting. Basically if you are animating the camera you should include keys for all the rotations and translations.

-vern

[heyvern]

Camera zoom seems to be an odd duck to handle. Camera tracking has a Z value which is completely different from the zoom value.

-vern

[Rhoel]

Many thanks for this.

Zoom:
The Zoom(Z) I am hoping will not be a major problem ... effectively I am anticipating to use camera_original_z_position + z_offset. If the camera position is in fact keys with non-saved inbetweens, then we might have to make a left and right file with keys on all frames.

Multiple Cameras:
I wondered too if AS can have more than one camera - this is something for Mike to look at. Most 3D programs have the capacity for multi cameras. This way, it would be much easier to have separate camera views.

Stereo Camera as Layer:
The other option would be to have a new stereo render layer (bit like a switch layer). That way we can have just one file with all the data in one bundle. It would mean a 'generate' button to populate the layer but once done, the layer could be saved. It would alleviate the problems of keeping 3 files current.

Face the Camera:
Another issue is the Face Camera. How one gets it to face the central camera and not alight to the left camera on one pass, then realign to the right camera on the second run. That will lead to eyestrain as the objects will be different positions in the shot.


BTW, do you have Mike's email .. I did have in Chiang Mai when I was working on Machine Masters but that computer is not in Cambodia. You might want to make him aware of this thread - he might have valuable input, especially if the stereo renderer is to be adopted in AS6. If multiple cameras are to be a future feature, then we need to construct the code blocks to allow for that to incorporated later.

I'll start on the code tonight (or during the day job if it remains a sluggish day);.

Rhoel

[Rhoel]

Each camera thingy has an argument that is either, 1, 2 or 3 values that it needs depending on what it represents. So translation needs 3 values, X, Y and Z, even if one of those values is "unchanged" it must be set on a key frame.

So if we focus on fCameraTrack, The value must be an AnimVec3 which you create this way:

Code: Select all

local v = LM.Vector3:new_local()
v:Set(val1, val2, val3)

Then you can set that "v" value to the camera like this:

Code: Select all

moho.document.fCameraTrack:SetValue(frame, v)

This is Object orientated which I find I usually mess up, then spend the next three months figuring out why the entire program has crashed.

I will concentrate on writing the actual formula, to give you the values for 'Set(val1, val2, val3)'. Having Lua using Radians is a slight pain as the rotate values appear to be in degrees. However, having math.rad(rotation) will be simple enough. I'll keep the naming real and logical, more left_camera_z_position to make it readable.

Mike uses LM for his code, you have a favourite for this? HV (HeyVern), HR (HevernRhoel) VT (VernTim) or SR (StereoRenderer)?

K, back to work.

Rhoel


edit
----

I was doodling with the positions tables over lunch and discovered something interesting. The Tilt appears not to affect any of the calculations: This is because it has no displacement. So its value is always 0. ie, it is rotating about its center. Might make the formula even simpler.

I'll generate a real set of positions and test in AS before compiling the formula: I'll manually feed the x,y,z positions into AS and render for left and right, see if the output is as predicted.

R

[heyvern]

As for what we might get in a future version. It would be great if we could get Mike or SM to get on board with this. If this were part of the render or camera settings, with more options like rendering separate files instead of red/blue etc.

I have Mike's email and will send him a note about this but I don't know how important this might be to him or where he is with any development for the next version. We may want to put this "on hold" until we know for sure. If there is more features to support this in an upcoming version we could waste a ton of time for nothing.

-------------
Continuing on with the discussion though...

I didn't think of a "center" or "main" camera. I just imagined there would be two files, one for the left one for the right. A center or main camera "file" is a good idea though. It makes sense.

There is another option for this as well. I'm still working on that "consolidation" script that writes a new file and fixes asset links. The stereo camera script could create two files and save them to a specific location.

This script could be a menu script that would pop up a window with settings for eye spacing and "depth", for instance how far "out" or "in" from the screen the 3D effect is.

--------------

As for the initials for the name of the script, makes no difference to me. Could use your HV or ST or RL or whatever. Or no initials at all. Can always put our names in the comments of the script file.

-vern

[Rhoel]

I have Mike's email and will send him a note about this.

Great, be good to know he is up to speed on this. I suspect the original red/blue anaglyphic output is part of the render library he uses, not his own code. So whether we have access to it is anyone's guess.

I didn't think of a "center" or "main" camera .... A center or main camera "file" is a good idea though. It makes sense.

This approach means you use the centre/normal camera to set the scene up, do everything normally. The script is then used if you need 2D output. This way its retro-compatible with older files.. Using a center camera also means that any rotations will orbit around the center line, and not around one camera.

This script could be a menu script that would pop up a window with settings for eye spacing and "depth", for instance how far "out" or "in" from the screen the 3D effect is.

The eye spacing is called interocular distance and the in/out setting is called convergence/divergence. There is a school of thought which says in animation (2D/3D) cameras should always be in Parallel - so divergence/convergence = 0. I am happy to have the divergence setting there in the code and GUI, but maybe loose it from the GUI in the production version if testing shows poor results (ie, headaches and eye-strain).

The eye spacing is usually set between 60-66mm, the iris-to-iris spacing of most people. In Moho though, the x units are units of ???. I think we have the GUI in mm, but use a variable in the code which 'guesstimates' the value of a mm: This would make it much more user friendly.

Or no [module prefix] initials at all.

I thought Mike encourages scripts to have a prefix as more than one team might produce a hand-held script or whatever. I was going to suggest BB to identify the Boggit Brothers I have no preference at all to this. Chose the initials of your kid if you like.

Tim
.

[Rhoel]

Vern,

Quick question: Tomnight I started roughing out the lua code. I used the Orbit script as a base for the new formula: I renamed the file then modified the display title: Discovered that despite being able to change the variable values and other things within the script body, the GUI titles remain the same.

I even used a find replace on the file and changed all orbit camera references to Stereo camera but nada, Anime Studio continues to display "Orbit Camera" as the GUI title. Even the "Duration (frames)" label when changed didn't update.

I remember this baffling me before and not finding a solution. Clearly, there is no "Orbit" name in the script. Is is coming from within LM.GUI or is AS holding a cache someplace?

I noted there are things like Mikes Alert script which displays Anime Studio Pro which is nowhere in his example script (Which kind of defeats the value of a sample script if you cannot see how the code works).

Ideas on the GUI labels are welcomed.

Rhoel

[heyvern]

Check the funky code that references a "string" file of tool names etc. I think that is what might be happening.

I did the exact same thing you did, modified the orbit camera tool and the name did change. I think to sometimes you have to restart AS to update that info rather than just a "refresh". Menu scripts are handled differently I think than tool scripts. I had that happen a few times.

Change anything like this:

Code: Select all

	return(MOHO.Localize(self.BASE_STR + 1, "Stereo Camera..."))

To something like this:

Code: Select all

	return "Stereo Camera..."

If that doesn't work with a refresh just restart AS.

This change applies to dialog boxes as well. If you don't strip that code no matter what you put in those dialog boxes as labels they will come up with whatever the "base string" is set for. Probably should change that "base string" variable. I have yet to figure out exactly what it is or if it should be changed. It's important for interface messages and such from pop up windows so that scripts don't conflict.

-vern

[heyvern]

I thought Mike encourages scripts to have a prefix as more than one team might produce a hand-held script or whatever.

Yes, you are correct. There should be some identifier.

I was going to suggest BB to identify the Boggit Brothers I have no preference at all to this. Chose the initials of your kid if you like.

Kid? What kid? Do you know something I don't? I've always been so careful!

Uh... you could use one of your kids initials?

-vern

[Lost Marble]

This is what is happening graphically.

I'm not sure if you meant this is what's happening in Anime Studio, or just in your rig. But just to confirm, yes, this is exactly what's happening in Anime Studio. The interocular distance is what you can set in the Project Settings dialog, and the two eyes are always parallel. Having the eyes converge can be very helpful in some situations, but can also be very tricky to get right.

The red/blue anaglyph is not part of any library - I lovingly hand-paint every pixel that you see. So I could render out other formats as well if that makes sense. What did you have in mind? Some possibilities would include rendering each eye as a separate movie/image sequence, or rendering both eyes side-by-side in full color in the same movie/image sequence. I'm not sure if there's a standard in say After Effects or Final Cut Pro for working with stereo footage. What would you guys like if you had your choice?