The project was released to the world at SIGGRAPH 2019 in the Immersive Pavilion and it really struck a chord with the conference attendees. We were told that is was one of the highlights of the show and media outlets like VentureBeat, said it was the ‘Biggest wow moment’ of the show!
Even with four pods of indexes running around the clock, there was always at least a 45 minute line. This was just amazing, we were actually worried before the show that four stations might mean that some were sitting unused!
Thanks to everyone for the kind words and feedback, it was great for the team to see all the hard work and passion pay off! Our goal was to get people interested Michelangelo’s work, and with the average SIGGRAPH attendee spending at least fifteen minutes in the experience, it was obvious that this was more than just a graphics demo, but something where people were learning about the work through exploration!
Next up was relying on modern photogrammetry (PG) to better align the hand authored meshes to real-world data. This is essentially measuring distances between features in photos and generating a rough model. The ceiling is plastered by hand and very organic –a form that can’t be described by Letarouilly’s drawings above, nor did he really try.
I was on the first team that used PG in games (Crytek PhotoBump, 2006), and shipped the first commercial PG software for that purpose. I have used PG techniques for over a decade across many projects, and I like to joke that it “works great for rocks” because it works best when every centimeter of an object is unique and it’s surface is relatively matte.
I was presenting the Il Gigante at Oculus Research and someone in the audience asked why we relied on sub millimeter laser scans from Stanford, when we could have used photogrammetry. I discussed how the surface is polished marble, and that the CyberWare data from the 1990s is still a billion points -really great! Nothing PG could touch for a sculpture like that. But later it hit me, the Sistine Ceiling is essentially the photogrammetric equivalent of a rock, six thousand feet of painted fresco (matte) -every inch completely unique.
Over the next days, I wrote a python script to trawl publicly available photos and use their measurements in Capturing Reality, a modern photogrammetry software. The results were amazing, I was able to reconstruct the vault down to about 5cm accuracy:
Over the years I had spent a lot of time on acquisition of high res images of Michelangelo’s work. I have amassed quite a collection of books, and have access to many more through friendships with historians who study the ceiling. I have also traveled to rare document collections, like the Prints and Drawings Study Room of the Victoria and Albert Museum in London. Because taking a photo of a public domain work doesn’t extend it’s copyright, you are allowed to scan or photograph these prints and elephant folios. Don’t be fooled by a museum telling you that it owns the copyright to a 500 year old painting –that’s not possible.
The Conundrum: Reprojection
My goal was to reproduce the ceiling down to teaching quality. Where possible I wanted people to see brush strokes and cracks in the plaster.
So.. every photo taken of the Sistine ceiling vault has some slight perspective distortion, they’ve all been captured from some distance away. Every photo has been taken with a different film or sensor, a different lens, under different lighting conditions –even the best images, that slavishly attempt to reproduce the frescoes, is very inconsistent and distorted.
This presents a massive challenge if you wanted to say.. reproject images onto a curved vault across 100 UDIMs.
As you can see above, I decided to layout the UVs in a way that would make reprojection easier. Hide and mask the borders in the fuax architecture.
I am not an environment artist by trade, though I do battle Maya on a daily basis. I decided to try and do as much as I could in Maya, but the software’s baking tools are abysmal. So much time was spent on this reprojection that I ended up writing a Maya tool to batch reproject my maps. Towards the end of the project, I had a machine baking almost every night, this allowed me to really tweak projections and UVs without losing time.
The Next Issue: Color
All images of the Sistine ceiling widely vary in color temperature. Over the years, they have changed the lights from tungsten with open windows and natural sunlight, to yellow cast halogen, to some new LED technology that boasts “Not even the master himself could have imagined that [his work] could look like this!” (Which, is just a terrible quote btw)
I had to choose what general feel I wanted the chapel to have, I settled on the color temperature before the LED extravaganza, which was slightly warm yellow.
Years ago I had written something in Matlab to cast one eye of the Fuji FinePix 3D camera into the avg colorspace of the other, because each sensor white-balanced differently. I built on this to try and cast all images I pieced together into a consistent-feeling color space.
I used a coarse interpolation to cast all scanned images into a consistent colorspace, the new Photoshop tool ‘Match Color…‘ was also very helpful for some final touches.
Above is an example, a 3+GB photoshop file of the Michelangelo’s Last Judgment. This was pieced together from many elephant folios and books, but notice how the exposure and color temperature is widely different across all the images? So I would find a single photo that I liked for an area, or this entire wall, and then use it as I stated above, to interplolate and cast all the stitched together images into one consistent whole.
I am pretty happy with how the piece turned out, we hit almost all of my stretch goals, including a reprojection of the ceiling as it existed before the controversial cleaning.
Pre-cleaned Work The above said, I would like to use a better dataset, there are a few hard-to-find testaments left, but they’re not the easiest to get ahold of, those books, the final photographic survey of the ceiling cost between two thousand and four thousand dollars!
Giornata I discuss ‘giornata‘ in the experience and blend in all 16 days of work Michelangelo did on the Creation of Man. I would like to convert the entire ceiling into individual giornata, so that a visitor can watch the entire ceiling, from entrance to lunettes and then altar wall be painted day by day in order.
Marks in the Plaster Michelangelo used ‘cartoons‘ for most of the ceiling, I discuss this in the experience. But towards the end he started drawing with a knife, excising directly into the plaster. There’s been a survey of these markings, here’s what they look like lit at a raking angle:
It would be great to incorporate these into the normal map and allow the visitor to inspect the surface with a hand held light.
The goal of this project is to leave a lasting gift that can be used to spur interest in Michelangelo’s work, democratize the ceiling and allow anyone in the world to view and learn about it. It was only possible with the help of friends and coworkers, I am always on the lookout for other CG artists interested in Michelangelo’s work to help.
Help Research the Damage Caused by the Restoration Attempt At the moment, the pre-restored art assets are derived from some 1960s elephant folios. If you have access to high resolution, pre-restoration images, those would be a big help. Or, if you would like to donate, the book “The Vatican Frescoes of Michelangelo” is the final modern book showing detailed reproductions of the pre-restored ceiling. It’s ISBN is 0896591581 and it’s about $2,000.
The goal is to piece together the highest fidelity pre-restored texture set possible, warped into the same UV space, so that a user can seamlessly blend between the pre and post restoration work and more easily identify damage.
Help with Art Assets I have spent a long time reprojecting images into a cohesive unwrapped UV space. The ceiling is over 100 4k texture maps. It’s a challenge to make things feel consistent, since texture inputs come from different time periods, different lighting, different cameras, etc. Now having the final images, it should be easier to run image processing on them to make them more consistent.
Also, there are some environment art assets that would still be great to have, like the Papal throne, which tourists never get to see, it’s referenced in the project because Jeremiah looks down upon it. It would be great to have it in there. Or the Rovere coat of arms, which I phoned in, it’s a challenging model and I just didn’t have the time.