Direct from the desk of the CEO: Epic Games acquires RealityCapture – This is HUGE!
HUGE NEWS in the 3D world! Capturing Reality, the Bratislava-based company behind RealityCapture, a leading photogrammetry tool, has been acquired by Epic Games. It’s among the largest video game companies, and the maker of Unreal Engine, a leading 3D engine for video games.
What does this mean, and what’s going to happen? That’s always the question. Who are these companies, and what is this software all about, anyway? Let’s dive in a little bit.
What is Photogrammetry Software?
Before we step into the worlds of RealityCapture and Epic Games, let’s look at the software involved to better understand.
There are a few competing solutions in the market for photogrammetry software. But before we get ahead of ourselves, what is photogrammetry software, anyway? Basically, it is software that allows you to take a bunch of photos (and/or laser scans) of an object or place, and process those photos into an actual 3D shape: either a dense point cloud, or a solid mesh made of triangles, colored (textured) by the original photos.
As you might guess, this is a pretty computationally-intensive process. In terms of input data, the sky’s the limit: You might use thousands or tens of thousands of source photos to create, say, an aerial 3d model of a city. Over the years, photogrammetry has gone from a niche affair requiring a huge workstation to make even a humble model from thousands of photos, to a fairly fast process, allowing a regular person with a gaming-grade GPU (more on that later) to process a thousand photos into an extremely high-poly (lots of polygons), very detailed, 3D file of an object or environment.
You have quite a few photogrammetry software options if you want to get started yourself:
Meshroom / Alicevision
Meshroom is an open-source, free project that is fully functional and very powerful. This was the first 3d software that I used. Without knowing what I was doing, I dropped 120 photos into it and clicked “start.” It produced a nicely textured 3d mesh of a sandstone sculpture.
Meshroom is from a very talented team in France at MIKROS, including Benoit Manujean, Fabien Castan, among others. The main complaint about Meshroom is that it is slower than other solutions. It is not optimized for a single-node workstation the way some of the other software is.
It is, however, completely free, so instead of complaining, you can, of course, fix it yourself. I have high hopes for this product that it will continue to evolve and improve. Just recently, Meshroom announced support for spherical panoramas, for example. So it is in active development, and it is getting better all the time.
Agisoft Metashape
Formerly known as Agisoft Photoscan, Agisoft Metashape is a very popular photogrammetry software that has been around for over ten years. It is one of the few photogrammetry products that has a Mac version. Additionally, the software has a reasonable price of $179 for the standard version and $3499 for the pro version. Both allow an unlimited number of images as input.
3DF Zephyr
3DF Zephyr is also quite a popular photogrammetry software. It has a free version, limited to 50 photos input (which can be enough in some cases to create some simple 3d models of objects or rooms), with the licenses costing anywhere from $149 to $3900 (similar to Metashape). 3DF Zephyr works well, and there are lots of people who enjoy using it.
RealityCapture
RealityCapture, made by a company called Capturing Reality (yes, a bit weird), has been available since around 2015. RC is the product of a small team of mostly Slovak engineers. It is led by Michal Jancosek, Tomáš Bujnak, and Martin Bujnak, who studied at CVUT (the Czech Technical University). CVUT is one of the world’s leading institutions for computer vision and related fields. This team wrote many well-known papers with their professor, Tomáš Pajdla, as well as Zuzana Kukelova, an esteemed researcher and now also an assistant Professor at CVUT. She helped to write some of the underlying solvers that form a basis for the RealityCapture software.
Martin Bujnak, one of the cofounders of Capturing Reality, is clearly an accomplished and respected researcher in this field. As you can see, Bujnak, along with Kukelova and Pajdla, have published a number of papers, spanning many years, on the topics of 3D reconstruction, algebraic geometry, pose estimation, lens correction, and other topics, all relating in some way to photogrammetry / 3D reconstruction.
How Reality Capture Differs
RealityCapture has been a bit different from the other products in a couple of ways. First of all, it has typically been faster than the others – much faster, in some cases. Compared to Meshroom, for example, RealityCapture might be over 100x faster if you have a large dataset of thousands of photos. You can count the difference in days or weeks!
While Agisoft Metashape has significantly reduced processing time in its most recent version, RealityCapture is still faster, especially when it comes to large datasets. While the final result is slightly different for all of these products, and the character of the 3D mesh is somewhat different, the actual result of RealityCapture is excellent. Although the experts might argue about which software offers better quality in high-frequency vs. mid-frequency details. In the end, this doesn’t matter too much for most people. And if you are doing this stuff for your job, where time is money, well, it makes sense to use the fastest product and spend less time waiting for your computer to finish rendering.
The Capturing Reality team couldn’t resist capturing this car covered in snow in Bratislava. See the original here.
Reality Capture’s Pricing
The second way RealityCapture has been different is in its pricing. It has had weird pricing:
- It has a “PPI” (pay per input) version where you pay for each model, based on the number of pixels in the input data. So if you have 50 x 50 megapixel images as input, you pay a certain amount, but if you crop those photos in just the right way, eliminating unnecessary pixels from your model, you might pay a lot less.
- There is a $40/month version that lets you use up to 1000 images per project. But those images have to be 24 megapixels or less.
- Or you can pay $750 monthly for unlimited usage (that’s a lot!)
- or $15,000 per year (holy smokes!).
So, unless you’re a professional user, you might be stuck with the very limiting $40/month version. One thousand photos only get you so far. You can’t do truly large areas or objects with that many photos.
RealityCapture’s Positive Reviews
Some of the reviews of RealityCapture on Steam give some more detail:
“The undeniable advantage RC holds over all other [Photogrammetry] software is it’s speed, there’s really no question that this software is a lot faster at practically every step of the process than others. For material scans it does offer one advantage also, the orthophoto mode allows exporting of a height and color map with no baking, this makes it possible to output extremely high detail height maps from 1M+ polys that would otherwise be impossible on low to mid range PCs.”
“I compared this product to Agisoft Photoscan and calculation speed of RealityCapture is unbelievable. RealityCapture can align standard 100-200 photos datasets in minutes and in Photoscan can take it hours. For me, this is top product on photogrammetry market. When i am doing simple statue, i can have preview on the spot with notebook with incredible speed. Inspection tool for analyze quality of my shooting is very very helpful. RealityCapture can handle huge datasets i never seen align/matching of thousands of photos calculated so quick on my desktop. And quality of high-resolution outputs, like textures and meshes is unbelievable. And in-app simplifying ability of high-res meshes is excellent. I can not believe what I see and i can do in this piece of software.”
RealityCapture’s Mixed Reviews
But also, the rather strange pricing/licensing model obviously rubbed a lot of people the wrong way:
“An example of a loss of business acumen and, potentially, a leadership position, or maybe just greed. Best software, some time before. Now is buggy DRM and have hard limits in new license. Also, still no new features, no user-friendly interface, no documentation and support.”
“Excellent photogrammetry software, trapped within one of the most consumer-hostile business models I’ve ever seen outside of an MMO.”
“Let me start by saying that RealityCapture is the best photogrammetry software on the market. Full stop. I’ve tried plenty of other solutions (Zephyr, Metashape, COLMAP, meshroom), and none of them match the quality that RealityCapture can produce. However, that praise is overshadowed by how actively hostile their business model is to subscribers.”
You’ve made a 3D model, now what?
You can use one of the above software products to create a 3D model. It’s actually pretty easy. Take a bunch of photos of an object from all angles, drop them in your software of choice (it doesn’t matter which one – they all work fine), click a few buttons, and out comes an amazing, detailed 3-dimensional thing – it really is like magic. But then what? What do people do with this 3D data? Lots of stuff, such as:
3D printing
You can make a 3d print of that thing you scanned. Whether it’s your child’s face, or some statue scaled down so that it fits on your table, or a 1:1 reproduction of a car component that needs to be reverse-engineered, there are a range of reasons why people make 3d prints of 3d scans, from the hobbyist to the engineer. Here is a very popular video from Prusa, the 3D Printer company, showing how you can shoot photos of a statue, make a 3d model, and then print it.
Who wouldn’t want this printed and placed on their front stoop? See the original by Capturing Reality here.
Video Games
Obviously, many video games take place in 3D worlds. Grand Theft Auto, for example, is an entire metropolitan area full of streets, houses, mountains, and everything in between. It doesn’t quite look like real life yet, but it’s getting closer and closer.
Imagine this in Grand Theft Auto. See the original 3D model here. Image credit: Jeffrey Martin
Movies / VFX
More and more often, what you see in a movie never existed in real life, and the actors may or may not even see the virtual objects they’re sharing the frame with on their virtual sets. In fact, the production of video games and movies is starting to merge into something very similar (more on this later). VFX studios are also huge users of photogrammetry to gather environmental data and accurately recover the camera track.
Check out this video too! It’s worth 19 minutes if you’ve got it.
Visualization
A factory, architectural site, crime scene, construction site… the list is growing longer every year: all of these places benefit from having a 3D scan, which can be viewed in extreme detail on a computer screen rather than visiting it in person. 3D scanning and visualization help crimes get solved faster and buildings get built within budget.
Street and Aerial mapping
Not only Google Maps, Apple Maps, and most other mapping companies are driving and flying all over the world to make a 3d model of the whole planet. Hundreds of smaller companies are doing this as well, for the purposes of straight mapping as well as infrastructure inspection, mobile mapping surveying, and so on.
3D Scanning Challenges
Here’s the thing, though: in all of these cases, displaying a 3D scan is not easy. This part of the toolchain is still difficult. It requires training, knowledge, and practice. It’s possible to make a 3d model without really knowing anything about how the process works, but the result will be a model that’s made of 50 million triangles and 10 gigabytes in size, which is too complex to show to anyone.
This model needs to be simplified down to maybe 100,000 triangles or maybe even 10,000 triangles, which are proper UV maps and Normal Maps. This process is hard, and there are university courses devoted to it for people who strive to work in the VFX or video game industry.
Shakeup in the Photogrammetry Industry
I’ve thought for a while that some company might acquire one of these 3D Photogrammetry software companies, because, well, that’s what happens eventually when a small company gets really, really good at what it does, and it starts to become relevant to the larger industry it’s a part of. And so it has happened for RealityCapture. Most likely, everyone who uses RealityCapture is breathing a sigh of relief that they were not acquired by Autodesk, which famously buys companies that are potential competitors, and then kills those products. And as it turns out, the acquirer, Epic Games, is very interesting indeed.
Who is Epic Games?
Epic Games, along with its larger competitors EA, Valve, Sony, and others, is among the ten biggest video game companies in the world. Their most famous product is Fortnite, which has helped propel them into this top ten list. What is more interesting is the 3D engine that Epic Games makes, which is the software that allows them to build these games, which take place in 3d worlds. This is Unreal Engine.
What is Unreal Engine?
When you make a video game that has humans, buildings, land, water, flying objects, gravity, and interactions between all of these things, you need to build software which can show these things in 3D, which can figure out how these objects are lit, and even has some knowledge of physics, so that objects can interact with each other in a plausible way. This is generally called a 3d engine. Unreal Engine is one of many – there are lots of other ones available (another very popular one is called Unity, for example). Unreal Engine is the most popular and widely used 3d engine, and it is technically very impressive. The next version of Unreal Engine, UE5, is pretty amazing. Here, watch this:
And maybe you can watch this also.
If you don’t watch the second video, just the title is enough: “Unreal Engine 5 and reality – can you spot the difference?”
Unreal Engine is not only used to make video games. It’s also becoming a valuable tool for VFX and filmmaking. A great example of this is The Mandalorian, and how it was made. Most of The Mandalorian was shot not on location but rather on a stage, in front of a huge LED screen. Using cameras with motion control, the picture on the LED screen, containing a 3D world built in Unreal Engine, could move with the camera to create a seamless background.
If that doesn’t fully make sense, then watch this fascinating video.
So to answer the question of “what is Unreal Engine” – It is the software behind creating a simulation that can’t be distinguished from reality for the video game and movie industries, and more and more other industries as time goes on.
Who else has Epic Games acquired lately?
Epic Games acquired Quixel at the end of 2019. Quixel builds a library called “Metascans”, which is a huge library of photorealistic 3d photogrammetry scans. When Epic Games acquired Quixel, they made this whole library free to use for everyone using Unreal Engine.
Are you connecting the dots yet?
Photogrammetry + Unreal Engine
So what’s next in the video game world? What’s the next Portal, or Half Life, or GTA going to look like? How real will it look?
It’s clear that the people who make video games would like it to be easier to build a 3d world, using actual real-world objects, without modelling them by hand. And in fact, for ten years now, people have been scanning real-world objects, and many of them are simplified and imported into Unreal Engine or other 3D engines to be used inside one of your favorite video games. But this process has been convoluted at best.
We are moving into a time where it will soon be possible to scan a room, or even a town, and then click “publish” to send it to Unreal Engine, where it will undergo sophisticated processing – simplification in a way that can’t be detected by human eyes – something that is 1/1000 of the file size, but still looks good – and can be used as the basis of a video game. In many cases, there will no longer be the need for teams of 3D modelers, painstakingly sculpting the components of a whole world.
But what will actually be the result of this acquisition?
The four fates of a Big company + Small company acquisition
In the context of “Huge company A acquiring tiny company B”, there are four general scenarios:
Scenario 1: Same company, different owner
In Scenario 1, nothing much will change. Small Company continues doing what it has always been doing, and makes money for Big Company.
Scenario 2: extermination of a potential threat
In Scenario 2, Small Company, posing a long-term existential threat to Big Company, will be slowly suffocated and eventually killed off. Its product will no longer be available, and there will be no replacement. The engineers will be subsumed into the guts of the larger company, never to be seen again.
See: most acquisitions by Autodesk
Scenario 3: Acquihire
The Product made by Small Company, while interesting, was never that interesting anyway, but the people who built it, being obviously very talented people, are the valuable part. It, while exciting, is not nearly as exciting as having these engineers work on some other big project that Big Company is working on. The product will continue to exist, but slowly become neglected and eventually abandoned as the key engineers are pulled into Bigger Projects within the company.
One example of Scenario 3 is Apple’s acquisition of Beats. Here’s a video about that. TLDW: Beats was dismantled for its engineers and software product; the software was turned into the Apple Music streaming service, and the hardware / audio engineers built Apple’s earpods and other audio equipment. Beats is still there but it’s a walking corpse.)
Another example is GoPro’s acquisition of Kolor, which allowed GoPro to build a 360º camera product, but ultimately killed the much-loved software products made by Kolor, affecting the 360º photography industry as a whole
Scenario 4: The sum is more than the parts
In Scenario 4, Small Company, while not a present or future competitor, is strategically relevant to the larger company in the grand scheme of things, and the technology of the Small Company will help the Big Company continue whatever vertical integration it has been planning, for increased domination of its industry. The product from the Small Company will continue to evolve, but with deeper integration into the main products of Big Company.
What is the fate of RealityCapture?
Scenario #1 is unlikely
Epic Games does not care about the relatively tiny revenue of RealityCapture – In 2019, Epic Games reported $4.2 billion in revenue and $730 million in earnings. While RC is an industry leader in photogrammetry software, its earnings are probably in the low tens of millions of dollars, which is a rounding error for a company like Epic Games.
Scenario #2 is also quite unlikely…
If the acquiring company were Autodesk, we would definitely have a #2 scenario, because that seems to be what Autodesk does. Thankfully, for many people who use this software on a daily basis for their jobs, this is probably not the case.
That leaves us with Scenarios #3 and #4.
In scenario #3, RC slowly dies or slides into irrelevance as its key talent is pulled into other, tangentially related projects in Epic Games. In scenario #4, RC is given a blank check to improve its product, and is given a team of engineers from Epic Games to work on bridging the gap between what RC does and what Unreal Engine does.
My money is on scenario #4.
Without having any inside knowledge of either company or knowing any of the people involved, I would venture to say that the people involved in RealityCapture are not in it just for the money (of course, they like money, and of course, when someone offers you a great deal, you take it). I don’t expect that the cofounders of Capturing Reality, who are esteemed scholars and engineers themselves, would be interested in the idea of shutting down their product, which is currently an industry standard, in favor of “working for the man”. Of course, it is possible that they agreed to terms in this deal that no longer allow them the ability to control this; however, I find this idea unlikely.
So, the end game that I, and others in the industry, are hoping for is that Capturing Reality is now free from the mundane day-to-day stress of running a business and making money. Now owned by infinitely wealthy overlords with bigger fish to fry, RealityCapture will continue to evolve and thrive. This is of extreme relevance to Epic Games’ core business, particularly when ordinary users create 3D assets that can be useful for video games.
Eventually, the RealityCapture software will be free to use for hobbyists, who now have a more direct path to making money from their 3d scans than ever before (shoot, scan, publish on Quixel, profit!)
Conclusion: Unreal Engine + Quixel Metascans + RealityCapture = the Ultimate Simulation
Looking at all the pieces, it seems like an obvious outcome. Unreal Engine 5 is so good that it’s no longer possible to hand-sculpt 3d models– they have to be scanned. How do we scan them? We need maximum quality, and time is money. RealityCapture is most likely already being used extensively by Epic Games to create assets for Unreal Engine. People will always need new assets for games, so it makes sense to acquire Quixel and offer free models. Epic Games charges a 5% royalty when you use Unreal Engine, so they’ll make the money back eventually. For the next 3d models of objects, rooms, places, and even entire cities, it makes sense to own RealityCapture. Then, integrate it as seamlessly as possible with Unreal Engine. The easier it is to create 3D worlds in Unreal Engine, the better the video games and films will be.
Special thanks to Alexandre Jenny and Arie Kai-Browne for providing their thoughts and feedback while I was writing this article. Thanks to Lynn Puzzo for editing.
