Instructions

Learn how to use Vimmerse's 3D immersive video technology.

All
Playback
iPhone
Capture
Uploading
Masking
Backgrounds
Re-Uploading
Kinect DK
Bullet Videos

Playback

Browser playback

Cheverons Icon

PC Browser playback

  • In a browser, locate a 3D video on the Vimmerse website. View featured content.
  • Select Web mode (which is the default)
  • Translate (x,y) position: click left mouse button and drag
  • Arc motion (x,y,z) position & orientation: hold down right mouse button and drag
  • Step-in/out: mouse scroll wheel up/down
  • Reset position: double click
  • Rotate angle: hold down center mouse button (scroll wheel) and drag
Cheverons Icon

Phone Browser playback

  • In a browser, locate a 3D video on the Vimmerse website. View featured content.
  • Select Web mode (which is the default)
  • Translate (x,y) position: two finger swipe
  • Arc motion (x,y,z) position & orientation: one finger swipe
  • Step-in/out: pinch
  • Reset position: double tap

App playback

Cheverons Icon

Windows PC app playback

  • Translate (x,y) position: click left mouse button and drag or use keyboard arrow keys (up, down, left, right)
  • Arc motion (x,y,z) position & orientation: hold down right mouse button and drag or use keyboard arrow keys (up, down, left, right)
  • Reset position: double click
  • Rotate angle: hold down center mouse button (scroll wheel) and drag or use W S A D Q E keyboard keys
  • Modify navigation sensitivity: select settings , adjust sensitivity slider
  • Modify focal point: select settings , adjust focus slider
  • Toggle GUI on or off: T keyboard key
  • Step-in/out: mouse scroll wheel up/down
Cheverons Icon

Android app playback

iPhone Capture

Cheverons Icon

Overview

3D video can be captured with the Vimmerse Capture app for iPhone Pro/Pro Max or iad Pro, or with the Kinect DK sensor. A video is captured which contains both regular color video (called texture) and depth, taking advantage of the built-in LiDAR sensor. 

Captured videos are submitted to the Vimmerse platform for content preparation. 

Creators can also create “bullet videos” in which the creator controls the 6DoF navigation and outputs a normal 2D video with 3D effects that is playable on any video player. 

Video quality can be improved by capturing a background image of the scene without any of the moving foreground objects/people. Video quality is also improved by providing a masked video that identifies the location of foreground objects/people for each video frame. Additional processing on a computer is required to create the masked video and the content is resubmitted to the Vimmerse platform for processing to create the improved 3D video. 

Cheverons Icon

1. Capture with the Vimmerse Capture app

3D video can be captured with the Vimmerse Capture app for iPhone Pro/Pro Max or iPad Pro, or with the Kinect DK sensor. A video is captured which contains both regular color video (called texture) and depth, taking advantage of the built-in LiDAR sensor. Captured videos are for content preparation.

  • Install the Vimmerse Capture app from the Apple App Store.
  • Mount phone on a tripod to ensure camera is stable. Either landscape or portrait orientation can be used.
  • If possible, first capture a background scene image without any foreground objects present (e.g. with any people or any objects that will move), by moving the image/video slider to image, and then capturing an image by selecting the record button.
  • Upon startup, the app will ask you to tilt the phone forwards and backwards to help determine its alignment. Be sure that the rotation lock is not enabled.
  • Optionally switch between RGB and Depth buttons to preview regular video (called texture) and depth (called geometry) video
  • Record a video by hitting the record button and stop button.
  • Optionally manually set the depth range in the settings tab by selecting the check box for Depth Ranges(m), then in the Record tab while previewing Depth, adjust the sliders to set the min depth and max depth in meters. Near objects appear brighter and far objects appear darker. Adjust the range to so that the full range of brightness is occupied, with the nearest objects white and furthest objects black, and avoid having the entire image be at a single brightness level.
Cheverons Icon

2. Upload your content

    • Optionally select the editor icon to trim the video.
    • Select the Gallery tab to preview your captured video as side-by-side regular video and depth
    • Optionally rename the capture and/or add a description by selecting the editing pencil and entering text
    • Select the cloud icon to upload the video to the Vimmerse website.
    • If you had captured a background image, select Yes to add it to the upload and select the appropriate image file from Photos.
    • Once the video has complete processing on the server, an example bullet video can be seen in the Vimmerse tab.
    Cheverons Icon

    3. Masking videos

    User content page screenshot
    • Select desired video
    My Content page example
    • Click the Download button and select “Download source views”. Example content can be found here.
    User content page screenshot

    Masking with RunwayML

    Screenshot from Runway ML

        Masking with After Effects

        • Use the Rotobrush tool on the texture video to cut out moving foreground objects and people in the scene. Go through every frame and make sure that the objects are fully and accurately within the pink outline. 
        • Import your texture video into a new project and create a new comp. Then drag the video in as a new layer.
        After Effects Screenshot
        • Create a new solid layer and set its color to white.
        • Expand the transfer control pane (the second icon in bottom left corner (see above)). Then under the track matte settings panel, set the solid layer’s track matte layer to the texture layer, then turn the texture layer’s visibility off. The resulting video will be an alpha matte that you can render out and upload. Render the comp in Adobe Media Encoder and name it v0_m.mp4.
        After effects with Rotobrushing example
          Cheverons Icon

          4. Background Generation

          • If you captured the scene without the foreground objects and people at the beginning of the scene, no action is necessary
          • Otherwise, optionally use After Effects or RunwayML to remove the foreground objects and people and create an inpainted background image or video. If the camera was on a tripod and is stable, a single background image should be able to be used, depending on the content. An instructional video for inpainting using RunwayML is available at https://www.youtube.com/watch?v=Afg9VlZOqaM.
          • For a single image, select a frame from the video with foreground objects (people, moving objects) that are as small a part of the scene as possible. Inpaint this frame, export and convert to .png format.
          • For a moving camera or for content with lighting changes, create a background video, by inpainting the entire video.
          • Alternatively, any video can be used as a background video, which may optionally contain depth.
          Cheverons Icon

          5. Re-Uploading

          • The content needs to be re-uploaded for processing to create the final 3D video, adding the mask video to create the final 3D video. The files listed below should be included. Make sure that all video files are of the same length and are synchronized.
          • Masked video: v0_m.mp4
          • Texture video: v0_t.mp4
          • Metadata file: any name with a .json extension
          • (Optional) Background texture image or video: b0_t.mp4 or b0_t.jpg or b0_t.png
          • (Optional) Background depth image or video: b0_d.mp4 or b0_d.jpg or b0_d.png
          • (Optional) audio file: any name with .aac or .mp3 extension
          • Depth/Geometry video: v0_d.mp4
          • On a browser, visit https://www.vimmerse.net/user-content and select Upload New Content. Optionally enter a name and/or description.
          • Use the default values for Capture Format, Downscale ratio, Mask source, and Rotation angle. Drag and drop all of the files listed above and select Upload.
          • After the files have been uploaded, select “Submit for Processing”.
          • You will be notified by email after the processing is complete or if there was an error.

          Kinect DK Capture Using Depthkit

          Capture using an Azure Kinect DK is supported with use of DepthKit.https://www.depthkit.tv

          Cheverons Icon

          1. Recording

          Cheverons Icon

          2. Export

          • Select the EDIT tab
          • Select REFINE
          • Select Enable refinement
          • Set the Filter Size and Fill Amount to the maximum values
          • Select ISOLATE
          • Optionally, for better depth representation, adjust the Depth Range sliders to cover the range of depths within the captured scene.
          • Preview the video and don’t be too aggressive when selecting the range, to avoid excluding objects outside the range.
          • Select EXPORT
          • Select Combined Per Pixel video
          • Export
          Preferred Depthkit Export Settings
          Cheverons Icon

          3. Upload

          • Login to the Vimmerse platform in a browser, select Upload content.
          • In the Capture format pulldown, select Auto, which is the default.
          • Locate the TAKE*.mp4 and TAKE*.txt files in the _Exports directory of the project.
          • On the Upload content page, upload the .mp4 and .txt files from your capture.
          • After uploading is complete, select Submit for Processing.

          Create bullet videos

          Bullet videos are 2D videos with creator-controlled 3D navigation, which can be posted to 3rd party video sharing services, including social media.

          The bullet video creation tool app is available for Windows.

          • Install the bullet video creation tool, Vimmerse Creator
          • Add the folder where Vimmerse Creator is installed to your path environment variable. By default it is installed to 'C:\Program Files\Vimmerse\Vimmerse Creator'. Tutorial information about how to edit the path environment variable is available at https://www.computerhope.com/issues/ch000549.htm.
          • Test with example content.
          • Open a Command Prompt in Windows (e.g. by typing “command prompt” in the search bar)
          • In the command prompt, launch the Vimmerse Creator with following command line:
            VimmerseCreator 3d_video.mp4 -opt posetrace.csv -ot bullet_t.mp4
            or
            VimmerseCreator 3d_video.mp4 -opt posetrace.csv -ot bullet_t.mp4 -og bullet_d.mp4
          • Create navigation path:
          • Create a key frame: Pause playback and select a pose (navigate to a particular viewing position and angle, as described in the PC player instructions.) Click on the key frame (bookmark) icon to set a key frame. Resume playback and repeat to create additional key frames.
          • Install the bullet video creation tool, the Vimmerse Creator Tool
          • Create freeze frame motion: Pause playback and select a pose. Set a non-zero time duration (in seconds) and click on the key frame (bookmark) icon to set a key frame. Repeat with different duration values and different poses.
          • When finished, click the export icon. The output bullet video will be stored in bullet_t.mp4 and the pose trace file in posetrace.csv.
            • Optionally edit posetrace.csv with text editor or spreadsheet and regenerate the bullet video with the following command line:
            • VimmerseCreator 3d_video.mp4 -ipt posetrace.csv -ot bullet_t.mp4 -og bullet_d.mp4

              Playback

              Browser playback

              Cheverons Icon

              PC Browser playback

              • In a browser, locate a 3D video on the Vimmerse website. View featured content.
              • Select Web mode (which is the default)
              • Translate (x,y) position: click left mouse button and drag
              • Arc motion (x,y,z) position & orientation: hold down right mouse button and drag
              • Step-in/out: mouse scroll wheel up/down
              • Reset position: double click
              • Rotate angle: hold down center mouse button (scroll wheel) and drag
              Cheverons Icon

              Phone Browser playback

              • In a browser, locate a 3D video on the Vimmerse website. View featured content.
              • Select Web mode (which is the default)
              • Translate (x,y) position: two finger swipe
              • Arc motion (x,y,z) position & orientation: one finger swipe
              • Step-in/out: pinch
              • Reset position: double tap

              App playback

              Cheverons Icon

              Windows PC app playback

              • Translate (x,y) position: click left mouse button and drag or use keyboard arrow keys (up, down, left, right)
              • Arc motion (x,y,z) position & orientation: hold down right mouse button and drag or use keyboard arrow keys (up, down, left, right)
              • Step-in/out: mouse scroll wheel up/down
              • Reset position: double click
              • Rotate angle: hold down center mouse button (scroll wheel) and drag or use W S A D Q E keyboard keys
              • Modify navigation sensitivity: select settings , adjust sensitivity slider
              • Modify focal point: select settings , adjust focus slider
              • Toggle GUI on or off: T keyboard key
              Cheverons Icon

              Android app playback

              iPhone Capture

              Cheverons Icon

              Overview

              3D video can be captured with the Vimmerse Capture app for iPhone Pro/Pro Max or iad Pro, or with the Kinect DK sensor. A video is captured which contains both regular color video (called texture) and depth, taking advantage of the built-in LiDAR sensor. 

              Captured videos are submitted to the Vimmerse platform for content preparation. 

              Creators can also create “bullet videos” in which the creator controls the 6DoF navigation and outputs a normal 2D video with 3D effects that is playable on any video player. 

              Video quality can be improved by capturing a background image of the scene without any of the moving foreground objects/people. Video quality is also improved by providing a masked video that identifies the location of foreground objects/people for each video frame. Additional processing on a computer is required to create the masked video and the content is resubmitted to the Vimmerse platform for processing to create the improved 3D video. 

              iPhone Capture

              Cheverons Icon

              1. Capture with the Vimmerse Capture app

              • Install the Vimmerse Capture app from the Apple App Store.
              • Mount phone on a tripod to ensure camera is stable. Either landscape or portrait orientation can be used.
              • If possible, first capture a background scene image without any foreground objects present (e.g. with any people or any objects that will move), by moving the image/video slider to image, and then capturing an image by selecting the record button.
              • Upon startup, the app will ask you to tilt the phone forwards and backwards to help determine its alignment. Be sure that the rotation lock is not enabled.
              • Optionally switch between RGB and Depth buttons to preview regular video (called texture) and depth (called geometry) video
              • Record a video by hitting the record button and stop button.
              • Optionally manually set the depth range in the settings tab by selecting the check box for Depth Ranges(m), then in the Record tab while previewing Depth, adjust the sliders to set the min depth and max depth in meters. Near objects appear brighter and far objects appear darker. Adjust the range to so that the full range of brightness is occupied, with the nearest objects white and furthest objects black, and avoid having the entire image be at a single brightness level.

              iPhone Capture

              Cheverons Icon

              2. Upload your content

                • Optionally select the editor icon to trim the video.
                • Select the Gallery tab to preview your captured video as side-by-side regular video and depth
                • Optionally rename the capture and/or add a description by selecting the editing pencil and entering text
                • Select the cloud icon to upload the video to the Vimmerse website.
                • If you had captured a background image, select Yes to add it to the upload and select the appropriate image file from Photos.
                • Once the video has complete processing on the server, an example bullet video can be seen in the Vimmerse tab.

                iPhone Capture

                Cheverons Icon

                3. Masking videos

                User content page screenshot
                • Select desired video
                My Content page example
                • Click the Download button and select “Download source views”. Example content can be found here.
                User content page screenshot

                Masking with RunwayML

                Screenshot from Runway ML

                    Masking with After Effects

                    • Use the Rotobrush tool on the texture video to cut out moving foreground objects and people in the scene. Go through every frame and make sure that the objects are fully and accurately within the pink outline. 
                    • Import your texture video into a new project and create a new comp. Then drag the video in as a new layer.
                    After Effects Screenshot
                    • Create a new solid layer and set its color to white.
                    • Expand the transfer control pane (the second icon in bottom left corner (see above)). Then under the track matte settings panel, set the solid layer’s track matte layer to the texture layer, then turn the texture layer’s visibility off. The resulting video will be an alpha matte that you can render out and upload. Render the comp in Adobe Media Encoder and name it v0_m.mp4.
                    After effects with Rotobrushing example

                      iPhone Capture

                      Cheverons Icon

                      4. Background Generation

                      Video quality can be improved by capturing a background image of the scene without any of the moving foreground objects/people. Video quality is also improved by providing a maksing video that identifies the location of foreground objects/people for each video frame. Additional processing on a computer is required to create the masked video and the content is resubmitted to the Vimmerse platform for processing to create the improved 3D video.

                      • If you captured the scene without the foreground objects and people at the beginning of the scene, no action is necessary
                      • Otherwise, optionally use After Effects or RunwayML to remove the foreground objects and people and create an inpainted background image or video. If the camera was on a tripod and is stable, a single background image should be able to be used, depending on the content. An instructional video for inpainting using RunwayML is available at https://www.youtube.com/watch?v=Afg9VlZOqaM.
                      • For a single image, select a frame from the video with foreground objects (people, moving objects) that are as small a part of the scene as possible. Inpaint this frame, export and convert to .png format.
                      • For a moving camera or for content with lighting changes, create a background video, by inpainting the entire video.
                      • Alternatively, any video can be used as a background video, which may optionally contain depth.

                      iPhone Capture

                      Cheverons Icon

                      5. Re-Uploading

                      • The content needs to be re-uploaded for processing to create the final 3D video, adding the mask video to create the final 3D video. The files listed below should be included. Make sure that all video files are of the same length and are synchronized.
                      • Masked video: v0_m.mp4
                      • Texture video: v0_t.mp4
                      • Metadata file: any name with a .json extension
                      • (Optional) Background texture image or video: b0_t.mp4 or b0_t.jpg or b0_t.png
                      • (Optional) Background depth image or video: b0_d.mp4 or b0_d.jpg or b0_d.png
                      • (Optional) audio file: any name with .aac or .mp3 extension
                      • Depth/Geometry video: v0_d.mp4
                      • On a browser, visit https://www.vimmerse.net/user-content and select Upload New Content. Optionally enter a name and/or description.
                      • Use the default values for Capture Format, Downscale ratio, Mask source, and Rotation angle. Drag and drop all of the files listed above and select Upload.
                      • After the files have been uploaded, select “Submit for Processing”.
                      • You will be notified by email after the processing is complete or if there was an error.

                      Kinect DK Capture Using Depthkit

                      Capture using an Azure Kinect DK is supported with use of DepthKit.https://www.depthkit.tv

                      Cheverons Icon

                      1. Recording

                      Cheverons Icon

                      2. Export

                      • Select the EDIT tab
                      • Select REFINE
                      • Select Enable refinement
                      • Set the Filter Size and Fill Amount to the maximum values
                      • Select ISOLATE
                      • Optionally, for better depth representation, adjust the Depth Range sliders to cover the range of depths within the captured scene.
                      • Preview the video and don’t be too aggressive when selecting the range, to avoid excluding objects outside the range.
                      • Select EXPORT
                      • Select Combined Per Pixel video
                      • Export
                      Preferred Depthkit Export Settings
                      Cheverons Icon

                      3. Upload

                      • Login to the Vimmerse platform in a browser, select Upload content.
                      • In the Capture format pulldown, select Auto, which is the default.
                      • Locate the TAKE*.mp4 and TAKE*.txt files in the _Exports directory of the project.
                      • On the Upload content page, upload the .mp4 and .txt files from your capture.
                      • After uploading is complete, select Submit for Processing.

                      Create bullet videos

                      Bullet videos are 2D videos with creator-controlled 3D navigation, which can be posted to 3rd party video sharing services, including social media.

                      The bullet video creation tool app is available for Windows.

                      • Install the bullet video creation tool, Vimmerse Creator
                      • Add the folder where Vimmerse Creator is installed to your path environment variable. By default it is installed to 'C:\Program Files\Vimmerse\Vimmerse Creator'. Tutorial information about how to edit the path environment variable is available at https://www.computerhope.com/issues/ch000549.htm.
                      • Download source views for your content, as described above in the Masking section, or test with example content.
                      • Open a Command Prompt in Windows (e.g. by typing “command prompt” in the search bar)
                      • In the command prompt, launch the Vimmerse Creator with following command line:
                        VimmerseCreator 3d_video.mp4 -opt posetrace.csv -ot bullet_t.mp4
                        or
                        VimmerseCreator 3d_video.mp4 -opt posetrace.csv -ot bullet_t.mp4 -og bullet_d.mp4
                      • Create navigation path:
                      • Create a key frame: Pause playback and select a pose (navigate to a particular viewing position and angle, as described in the PC player instructions.) Click on the key frame (bookmark) icon to set a key frame. Resume playback and repeat to create additional key frames.
                      • Install the bullet video creation tool, the Vimmerse Creator Tool
                      • Create freeze frame motion: Pause playback and select a pose. Set a non-zero time duration (in seconds) and click on the key frame (bookmark) icon to set a key frame. Repeat with different duration values and different poses.
                      • When finished, click the export icon. The output bullet video will be stored in bullet_t.mp4 and the pose trace file in posetrace.csv.
                        • Optionally edit posetrace.csv with text editor or spreadsheet and regenerate the bullet video with the following command line:
                        • VimmerseCreator 3d_video.mp4 -ipt posetrace.csv -ot bullet_t.mp4 -og bullet_d.mp4