Facebook has designed and developed a 3D-360 video capture system. Some important points as below:
- It has capacity to capture high resolution video
- The system also known asFacebook surround 360
- It is also well known for production-ready 3D-360, foe hardware and software video capturing
- It will be available on GitHub this summer.
- The design for camera hardware and software code will be available on Git repository for developers, which will help to promote the product
- It has been designed and developed based on algorithm
- It will produce maximum throughput
- Optical flow concept has been used to left-right eye stereo disparity
- It has ability to consolidate and generate stereoscope 360 panoramas
- It has capable to reduce the development time(weeks to overnights) due to stitching code based algorithm
- The system can exports 4K, 6k or 8K videos for each eye.
- The 8K videos maximise the industry standard output
- The output can be played with Gear VR by using Dynamic Streaming technology developed by facebook
More About 3D-360 system
It will helps to create professional grade for end-toend system. It will enable to edit, capture and render high resolution video. It has aims to provide high quality video capturing with providing maximum level of satisaction for artist.
Defining the challenges of VR capture
At inial level of developement was very critical time. Initially it was not available for everyone without special request. The cameras in the 3D system found as overheat so can't go with the production environment. The stiching takes a lot of time bacause everything must be done by the hand. These challenges and analysed properly to provide capture, edit and rendering features in easy manner.
To capture 3D video from 2 cameras in stereoscopic 360 and Monoscopic 360 was very chalenging. It allowed to look arround the whole scene rather than flat like a still photo.
Initially every and every shoots has been analysed properly. Facebook analysed the shots for each location which were taken with two cameras. It has found as same from left to right or right to left. It means need to shoots in streoscopic 360 location with 10 to 20 cameras which can support 30 to 60 frames per second simultaneously. Each shoots must be stitch from cameras into single videos. It should be happened from both eye (left to right or right to left directions.)
The analysis for both side capturing was very difficult but it found as feasible because already a lot of experiments happened since long time. By previous research helped a lot to make it as successful due to decrement of memory cost.
The VR capture System
Facebook did a lot experiment to make it reliable, high-quality and end-to-end capture system. They designed and developed a prototype which they had faced earlier. The below point has been taken care in the mind:
- The hardware (the camera and control computer)
- The camera control software (for synchronised capture)
- The stitching and rendering software
Above points are very important to achieve the goals. Each component is dependent on each others. It can affect the quality of product if any one have lack of synchronisation.
We tried to build which can support others working model. It will be help in terms of non-dependent on actual architure. So that anyone can build and deploy the product with the help of new technology.
The camera hardware
A system can be combination of hardware and software. Any one have lack of synchronsatio that might be affect the quality of the product. The below point is required for camers:
- Camera must be able to capture each frames simultaneously within leass than 1 ms of other, synchonisation is required
- All pixel can be viewed simultaneously, all camera must have a global shutter will capture all the objects
- Camera must have on-off features
- Camera must be able to capture the object continuously without heating.
- Rig and Camera must be rigid to improve quality.
Each and every points has been analysed and working model has been designed accordingly. The Point Grey has been used which have global shutter. It will not overheat if contnue in working. The cameras has been bolted into aluminium classic so that it must be stay in one place. To prevent internal component it has been coded from steel from outside.
The camera control and capture software and storage
Once design has completed then next step was to control the camera movement. It has been designed in such a way who can run on linux based system. It will help to transfered stream from cameras to desk easily.
Camera should be capture all the frames without dropped. For that Low level buffer is required. 30Gb/s transfer rate is required for 30Hz frames. Facebook used at initially 8-way level -5 RAID SSD disk to keep all the camera capture rates. Camera can be controlled over remote through a web brwoser in the devices that supports an HTML browsers.
System should enable exposures, shutter speed, analogue sensor gain and frame rate that can be controlled by capture software. All cameras are synced with remote system.
The Stitching Software
Capturing the frames and storing was very challenging task, a lot of computational algorithm has required. Already experiments has been done since last 20 years in case of computational photography, which gives idea to continue the implemetation.
Stitching process helps to collect the images part of capture screen into single video images. The visual images quality was good as expected. They tried to maintained the quality of images so that they had process the data in several steps:
- Converts raw Bayer input image to gamma corrected RGB.
- Mutual camera color correction
- Anti-vignetting
- Gamma and tone curve
- Sharpening
- Pixel Quality
- Correct the lens distortion and reproject images again
- Consolidate the all screen captures into single video screen
- Apply optical flow
- Consolidate final pixal of left and right flows
The key concept is optical flow. It will help to build code which is stronger for creating stitching than other. It delivers better solution in terms of results. Optical flow will compute all the object form left eye to right eye viceversa. It will follow top and bottom cameras into the side cameras. So that it can match the objects. It will automatically generates seamless steroscopic 360 panaromas.
Playback
Facebook 3D-360 stores output as 4K, 6K and 8K for each eye. It uses Gear VR for streaming high quality video. The output file can viewed by VR headsets such as the Oculus Rift and Gear VR. It can be shared on facebook as feed.
Build and create
Facebook has tested 3D-360 camera and promised that it has stabled. Every day technology changes remain. Design changes in enginering in terms of optical field of view, sensor resolution, camera arrangement, will iniate to modiy the design for 3d-360 also. Some other camera also have features of stitching, but they will increase the volume of captured data and bandwidth to process. Increasing rendering quality can be costly.
Optical flow change will help to reduce the number of cameras or ease the amount of required stitching.
Facebook has planned for open sourcing for 3D-360 in upcoming summer. It will help for others to come and join for better improvement.
A final note- making a personal connection
When the facebook member has started their work towards 3D-360. It was very exiting movement for all. They were very excited for working a leading-edge camera as a "VR camera".
Facebook team has started working hard. All the team was excited about the VR camera. They are just trying to connect the people through the technology.
View post on fbchandra.com
No comments:
Post a Comment