That is the question.

Whether ‘tis nobler in the mind to wrangle the proprietary formats of your camera manufacturer, or to take arms against a sea of sidecar files, and by opposing, end them…

But enough pseudo-Shakespeare for one post.

There has been some chatter on the Interwebs lately concerning the DNG format: there are quality and archival concerns, whether it’s worth converting one’s entire library to the format, what the benefits might be, and whether one ought to care. Today, I weigh in. For what it’s worth.

Coincidentally, I’ve used the same post title as Matt Kloskowski did in his take on the subject on Lightroom Killer Tips. Matt didn’t add a pseudo-Hamlet line, though, so I feel like I’ve done the headline proud.

After the break, a complete rundown on DNG; trust me, it’s going to be technical and editorial.

What Is DNG?

Skip this section if you already know what DNG is, or read it if you want to know what I think DNG is.

DNG is the “Digital Negative” format, spearheaded by Adobe. It is an “open” format, and a “standard” in some sense, though not an official ISO standard (yet). When we say that the format is “open,” we mean that the precise contents of a DNG file, its byte offsets, containers, methods of generating and reading the files, and so forth, are publicly available. This is not the case with any other RAW file format out there. The ability to read and write manufacturers’ RAW files (e.g. Canon, Nikon, et al.) is either licensed from the manufacturer or reverse-engineered.

Adobe created the standard out of, apparently, a desire to “universalize” (if you will) the ubiquitous RAW file formats that all manufacturers have created for themselves, and to introduce a truly interoperable format that any software or hardware maker could employ without 1) wrangling many different formats at once, or 2) singling out particular users of particular products.

The DNG format is based on many existing standards owned by prestigious “standards development organizations” (or SDOs), some of which you’ve probably heard of; they go by the names TIFF, JPEG, XMP, IPTC, ICC, CIE, and ZIP, to name a few. So, although the DNG format itself hasn’t been embraced by an SDO such as the International Standards Organization (ISO), it comprises previously standardized formats and is only a snippet of red tape away from being SDO-backed.

You can read a lot of nitty gritty on the format itself explained by Barry Pearson on his DNG Specification page.

Why Would You Want to Use DNG?

There are three main reasons that are generally cited:

• Archivability (future-proof-ness, if you want)
• Interoperability (openness, we like to say)
• Efficiency (YMMV, more on this later)

Archivability (which I’m pretty sure isn’t even a word) means that fifty years from now, when Canon owns the world and Nikon is completely extinct (har har), flying cars are a reality, and everyone has an army of robotic servants, your DNG files will still be supportable by manufacturers of hardware and software alike, because no secrets about it have been kept from the public.

“[I]f Nikon decides to not support my raw files one day, there’s some 15 year old in his room that’ll code up a raw conversion program in his sleep.”
—Matt Kloskowski

Does this claim hold water? I don’t know, you can download free software right now that can read the Photoshop version 3.0 format, which is already decades old, so it seems to me that if the ability to parse a format is out there (even if it was obtained semi-legally or with much effort by third-parties), it will be difficult to lose it. Moreover, a format such as Canon DNG or Nikon NEF will likely remain supported by anyone you care about, or, as Matt Kloskowski put it, “…there’s some 15 year old in his room that’ll code up a raw conversion program in his sleep.” So I am not worried about losing access to my RAW image data.

Interoperability means the ability for you to take the same file and use it in many different places. Of course whether this is an advantage to you or not depends on the places where you want to be able to use your RAW files. For me, Lightroom is the only program on my entire computer that ever sees a RAW file. I suppose if I round-trip through Photoshop, Lightroom is going to pass the CR2 through Camera Raw, but we’re talking about two closely-integrated Adobe applications there; whatever formats one supports, the other is sure to. Photomatix is going to see TIFFs, on the web you’ll see JPEGs, etc.

So why does interoperability matter? Adobe’s point in creating DNG is that it may matter in the future. If Great New Software X decides they can’t support your camera’s RAW format, and if you’ve converted it to DNG already, well, problem solved. Because Great New Software X will certainly support DNG given that the format is completely open and drop-dead simple (not to mention free) to implement, you have a much better chance of being able to drop your existing DNG images into any new, shiny tool.

“I actually tried ZIP compression in prototype versions of DNG, but the compression ratio was much better using lossless JPEG.”—Thomas Knoll

Okay, what about efficiency? This is where things get kind of cool. The Adobe DNG format stores the actual pixel image data in what is called Huffmann lossless JPEG format. What that means is that DNG files can sometimes be as much as 20% more efficient at storing image data on disk than a comparable RAW format, thus DNG files may be as much as 20% smaller. Lossless JPEG is completely pristine; there is no image-altering compression done, so the data is totally preserved, albeit compacted. The Huffmann algorithm for this compression happens to be more efficient than ZIP when there are more than 8 bits of data per channel (RAW is 12 or 16), so the Huffmann algorithm was used.

But here comes the kicker… Canon’s RAW (CR2) format already uses Huffmann lossless JPEG for its internal image data storage. So if you shoot Canon RAW, you will see no increase in data compression, since the data is stored in exactly the same way already. If you shoot in Nikon RAW (NEF), you will immediately see a 20% decrease in file size and no change in image quality.

For non-Canon shooters, that’s probably the most compelling reason to convert to DNG right now, which I should point out Lightroom can do automatically for you at import time. You’ve seen the option, right? There are a couple of ways to convert to DNG in Lightroom, and there are three of them explained on TheLightroomLab.com.

Sidecars Aren’t Just for Kids

No, seriously, most motorcycle sidecars could definitely seat an adult…

Joking aside, what we’re talking about are metadata “sidecar” files, typically named something like IMG_0195.xmp and so-called because they are saved alongside your original RAW image files, like the sidecar on a motorcycle. Sidecar files have been around since the invention of metadata and metadata libraries. The XMP format for storing image metadata in a sidecar file was developed by (guess who…) Adobe. XMP is necessary because the metadata support within RAW file formats and other imaging formats may comprise only a subset of what, for example, Lightroom is able to save and search.

Certain formats such as PSD, TIFF, and JPEG have pretty flexible metadata support in them already. DNG is no different. The “sidecar bonus” of the DNG format is that when you’re using DNG, you don’t need sidecars.

But who does, anyway? The fact is, Lightroom stores all of your image metadata in its own catalog. This is done 1) to make it quickly searchable and editable, and 2) to centralize it. If you want to write metadata to disk for certain files (or the whole catalog), then Lightroom will decide, based on the source format of each image, whether to save it directly into the file or to create an XMP sidecar file.

The purpose of XMP is to give your image metadata a place to live when the image file itself can’t accommodate it. So why would you need this ability? Two reasons. (Assuming your RAW files are in NEF, CR2, or similar; non-DNG).

1. If you share images with other people who use Lightroom and you want them to be able to see what you have done in the Develop module, as well as the IPTC tags, you will need to send them XMP sidecar files. This is preferable to sending your entire Lightroom catalog or creating a new one to house only the images you are sending.
2. In the event of a catastrophic disaster where your Lightroom catalog backups are corrupted, you can restore 95% of your data using the source RAW image files and associated XMP sidecar files. What you would lose is virtual copies, your history, and any collections. All edits and other settings are stored in the XMP files.

There are some major caveats here, though.

1. When you’re sharing images, you can always export them to DNG, which makes XMP unnecessary.
2. In the event of a catastrophic disaster, you’ll only have XMP files if you’ve previously selected batches of images within Lightroom and triggered the “Export Metadata to Files” function. Presumably you would carry out this procedure on images you have finished editing as part of your backup strategy. For the record, I do not do this.

Conclusions

So where does this leave us as far as DNG goes? Should we all run back to our computers right now and convert our whole libraries to DNG format? Once again, here are the major advertised benefits of the DNG format:

1. Archival (future-proof)
2. Interoperable (widely compatible)
3. Efficient (storage-wise, at least)
4. All-inclusive (obsoletes XMP sidecars)

It seems to me that the “archival” and “interoperable” characteristics of the format are bonuses, and as a supporter of free and open standards, I tip my hat to Adobe simply on principle. These characteristics don’t, however, increase the immediate convenience or reliability of my workflow, nor do I think the archival nature of the DNG format will have a measurable impact on my workflow in the coming years.

Since I am a Canon shooter, my Canon RAW (CR2) files already compress image data using the same Huffmann lossless JPEG system that DNG does, so I get no benefit there. If you you Nikon, et al., you may see an immediate storage benefit from converting your library to DNG.

Perhaps the most compelling reason to convert your library to DNG, or to start using DNG for your imports going forward, is the fact that DNG files are capable of storing all Lightroom-specific metadata directly within themselves, without the use of XMP sidecar files. If you are diligent in writing metadata to the files when you are through editing them, those files then fully encapsulate all the work that you’ve done, in one place, suitable for backup.

Even in the event that you lost your Lightroom catalog, you could still restore the final, edited versions of the images (which is where all of your hard work goes, after all), as well as tags and other helpful metadata. The only things you’d lose are virtual copies, history (of limited long-term utility, anyway), and catalogs (which can be rebuilt without nearly as much work as re-developing all your images).

To DNG or not to DNG? I think I will.

When you come on a workshop with Chris and I, you get to stand shoulder-to-shoulder with a total of 20 years of photography experience, heckle us with questions, meet people, and generally have a good time. Two instructors, awesome locations, and, here’s the sweet part… I’m giving you a discount.

Because you are still reading my humble blog after months of silence and because I value your loyalty and participation, I am offering you a great deal on my workshops this year. If you register for any workshop before March 1st, I’ll give you 10% off the price, just by mentioning this blog article when you sign up.

I’ll tell you what else. Just because I’m in such a good mood and excited to teach this year, if you want to register for more than one, I’ll give you 20% off of each of them. All you have to do is mention that I said so in the comments when you’re signing up on my website.

Look at this great lineup of workshops we have coming up this year!

I hope you have a couple of days to spend with us, I promise that our workshops will push your skills to the next level!

This time, though, a piece of “news” hit my screen that I had to share, and I hope that by reading it here you will receive the whole story and not just 1/3 of it as some people did when they got the information from digg, reddit, or similar.

We’re talking about megapixels, and we’re talking about marketing, and we’re talking about lies. Lies perpetrated by the manufacturers of point-and-shoot cameras with tiny little CCD sensors who keep increasing the megapixel count and marketing it like it’s the end-all be-all of imaging performance.

Not so! Read on…

This issue surfaced when a nameless blogger (literally, I have no idea who this person is) posted an article (you might call it a “diatribe”) about the “suicidal march” of point-and-shoot cameras toward more and more megapixels (spurned by the Consumer Electronics Conference, or CES, which was earlier this month in Las Vegas), and the results of cramming lots of pixels into a small area. You can read that article here.

In order to understand some of the background, though, you’ll want to read his or her other article about optical diffraction and Airy disks, which is very interesting, and then read his or her follow-up article where he (or she) addresses some of the anonymous Internet’s criticisms.

Now, I know it’s a gamble to spread anonymously written blog posts around as though they’re primary sources and laud the nameless, faceless author for their courage in uncovering what seems to be a pervasive scam, but to be honest I was blown away by the apparent truth of the whole thing.

I find it somewhat amusing that they chose to throw on the EF 50mm f/1.4 for their promo shoot; a lens that, for me, spontaneously stopped auto-focusing after only a couple years of use and that exhibits pretty significant vignetting. Nevertheless, the EOS-1D Mark IV is a fairly intense camera body, with its 10 frame per second burst mode, and ISO settings up to 102,400. You should be able to lay your hands on one here in the States for a mere $4,999.

Personally, I don’t think I’ll be purchasing a 1D Mark IV, if only because it has an APS-H sensor with a 1.3x crop factor and I am utterly spoiled by my 5D’s full-frame sensor. It is worth noting, also, that for the $4,999 that a brand new 1D Mark IV is going to cost you, you can pretty readily find a pre-owned 1Ds Mark III, which is 21 megapixels of full-frame goodness.

Over on the right you see a photograph taken by Michael “Nick” Nichols (and his team) for National Geographic, which is on the cover (well, part of it is on the cover, it’s a huge photograph) of the October issue of the magazine.

The image was made by jigsawing 83 separate photographs together (which you can probably tell by the jagged edges), each of which was taken by one of three DSLR cameras mounted on a gyroscope-leveled, pulley-lowered rig that Nichols and his team designed for the purpose. You can check out the photo on Hack a Day of Nichols with his rig; it looks like they’ve got six Pocket Wizards on there (I don’t know what the other three are for) and maybe a couple of bicycle wheels. All in all, a very righteous hack.

This particular redwood is allegedly the “most architecturally interesting” tree in the world, with several forks and bends stretching 300 feet into the sky. It’s only been standing there for over 1,500 years(!!), but now it has been recorded in the annals of photographic history forever.

Via (one of my favorite blogs) Hack a Day, via National Geographic, via NPR

Buchstabensuppe, by tillwe

For those of you Nikon shooters out there who ritualistically browse the B&H catalog and wonder What on Earth is a f/4.5-5.6G ED-IF AF-S VR DX lens, anyway?, now there is an answer for you, straight from our favorite bringer-of-technological-clarity, Bob Johnson at Earthbound Light.

In his post Nikon Lens Designation Alphabet Soup, Bob explains the meaning of all of those little acronyms that Nikon seems to throw around like confetti. At last, an understandable answer!

As the sun got lower, the sky got even more interesting. Here’s one of the keepers from that group that I can show off because it’s my blog so I get to subject you to whatever I want:

Coming up, we’ve got a really cool workshop in Acadia National Park in Maine (that’s October 16th through 18th), and then in November we’ll be doing yet another one-day Boston workshop if anyone wants to walk around the freezing streets of Beantown and pick up some tips.

For your daily dose of photography-related slow motion, I bring you a video of the Canon 5D shutter shot at 2,000 frames per second using (allegedly) a Phantom HD high speed camera.

Chris Benton is a professor of architecture at the University of California, Berkeley. He also straps his digital SLR onto a kite string and takes some of the most amazing aerial photographs I have ever seen. Chris combines a truly gifted eye for composition with a an engineer’s savvy for mechanical problem solving.

Using kites and remote-controlled camera rigs built by hand in his basement, Chris captures the world top-down, photographing everything from people and buildings to the patterns of nature. Watch this video from Make Magazine and be stunned!

Kite Aerial Photography on MAKE: television from make magazine on Vimeo.

Chris is quick to humbly share his experience and tips for aerial kite photography on his website (graciously hosted by Berkeley!)

I was blown away by the evolution of Chris’s camera rigs, which grew from fixed harnesses with rudimentary mechanical timers based on elastic bands, Silly Putty, and disposable cameras, to what he now uses, a remote-controlled, servo motor-driven, three-axis robot.

It’s definitely not commonplace for someone to possess both a grasp of electronic and mechanical engineering and a hawk-eye for artistic composition. Chris Benton has both.

Part Two: Cars

Aside from being an avid photographer, I admit to a streak of BMW fanboyism. When it comes to sports cars that are still solid daily drivers that make you feel like you’ve personally discovered the center of the universe and you’re sitting directly within it, nobody does it better than BMW.

To hype the release of their latest creation, the Z4 Roadster, BMW hired artist Robin Rhode to dip the Z4’s tires in multi-colored paints and drive it around like a fingerpainting on a warehouse scale. This was one of the most indulgent marketing campaigns I could think of.

I realize this isn’t strictly photography-related, but it is certainly art-related. If you feel gypped, go look at these photographs from the event

And here is a pretty cool video of how they put this thing together:

The next time you’re out shooting, or sitting around the house thinking about how next to use that studio space you set up in the basement, or the garage, or the attic… Think about breaking out of the box and doing something completely different. Robin Rhode did, and I think it came out pretty well.

Anyway, on with the show!

First things first, JPG Magazine is back! After being effectively shut down due to budget and business problems, JPG has resuscitated itself mostly thanks to the outpouring of support from its community and highly visible demonstrations such as savejpg.com which presumably gave investors the confidence they needed to pump more necessary capital into the parent company of JPG Magazine, 8020 Media.

Now that JPG has risen from the dead, maybe I’ll actually contribute something! You should, too.

Second, my little hands-on instruction business, ArtPhotoWorkshops.com is going to be doing a series of low-cost, short “photo-walk” workshops in the New England area. Starting with Cape Cod in a week and then Boston after that, I plan to take us out to Newport, Rhode Island, possibly Northampton, Massachusetts, maybe even out to the Quabbin Reservoir for the nature lovers.

If there are places you’d like to explore and learn technique, composition, and mechanics of photography, leave a comment and I’ll see if we can visit them!

I just returned from Las Vegas and Death Valley on a workshop, no photos to show yet, but I think it was an extremely successful trip. Death Valley is by far one of America’s most impressive sights, I recommend seeing it once in your life (or if you’re crazy like me, twice). Las Vegas is a very challenging location to photograph, but I think I was able to snag at least a small number of cool images in between games of craps and tall beers!

There are a few other cool things I want to share, so stay tuned for kite photography, painting with a car, and my opinions on both.