NetCDF Introduction and Resources – Storing Scientific Data

Network Common Data Form, or NetCDF, is a set of software interfaces and libraries designed for machine-independent representing of array-oriented scientific data. The NetCDF data format is an open standard. It’s self-describing, portable, and scalable. The NetCDF Application Programming Interface (API) itself allows for the creation, access, and sharing of scientific data.

These facts have conspired to make NetCDF very popular in the geosciences community over the past decades.

Brief History

The NetCDF project was started in 1989 by the University Corporation for Atmospheric Research (UCAR), a non-profit consortium of more than 100 universities and colleges. UCAR was established in 1959 to support atmospheric research and promote science.

The NetCDF software was developed by Glenn Davis, Russ Rew, Ed Hartnett, John Caron, Dennis Heimbigner, Steve Emmerson, Harvey Davies, and Ward Fisher at the Unidata Program Center in Boulder, Colorado. Unidata is a community of education and research institutions with the common goal of sharing geoscience data and the tools to access and visualize that data. Unidata is one of UCAR’s Community Programs (UCP), funded primarily by the National Science Foundation. Unidata is also the source of NetCDF software, standards development, and updates.

NetCDF has been updated and changed over its lifetime, from version 3.x dating back to 1997, through version 4.0 from 2008, to version 4.1 released in 2010. All of these different versions of NetCDF are maintained and updated by Unidata. The good news is that NetCDF is backwards compatible when it comes to its data files, meaning that all NetCDF files ever created can be read and used by the latest release of NetCDF.

NetCDF Libraries

Software libraries are an integral part of the NetCDF package, providing read/write access to NetCDF data files. The core NetCDF library is written in C and provides APIs for C, C++, Fortran 77, and Fortran 90. Besides these programming languages, NetCDF APIs based on C are also available for Perl, Ruby, Python, Haskell, Mathematica, MATLAB, IDL, and R.

A separate implementation of NetCDF is also available for Java, it is also developed and maintained by Unidata and called NetCDF Java.

NetCDF File Formats

The latest release of NetCDF supports three different binary formats for NetCDF data files. These formats evolved over time and were introduced in different NetCDF reelases.

The classic format was originally used in the first NetCDF release, and it is still the default format for creating NetCDF data files. The classic NetCDF format has strict limitations regarding the structure of data files larger than two gigabytes. Be sure to check these limitations if you plan on using and creating large files in the classic format.

The 64-bit offset format was introduced in NetCDF version 3.6.0 in 2004, greatly expanding the sizes of variables and files supported in NetCDF. The 64-bit offset format is very useful for managing large data files (over two gigabytes), however, only users running NetCDF version 3.6.0 or higher can use this format.

The NetCDF-4/HDF5 format was introduced in NetCDF version 4.0 released in 2008. This format allows the use of the Hierarchical Data Format 5 (HDF5), with full use of the new HDF5 dimension scales, creation ordering, and other advanced features of HDF5 version 1.8.0.

All of these file formats are platform independent and self-describing, which means that the data file contains a header section which describes the layout of the data contained in the rest of the file.

Regardless of the version of NetCDF you choose, the default format for creating data files is the classic format. Using another format has to be specified and declared during the creation of the file.

NetCDF Common Use and Applications

Since NetCDF was specifically designed for manipulating vast amounts of scientific array-oriented data, it is commonly used in meteorology, climatology, and oceanography applications. NetCDF is used as a common data format in many Geographic System Information (GIS) applications, and for exchange of scientific data.

Many scientific applications use NetCDF, like Ferret, GrADS, ArcGIS, NetCDF Explorer. Unidata has a good list of software using NetCDF.

NetCDF Resources

If you need to find out more about NetCDF, we suggest taking a look at some of the following online resources:

• The NetCDF Official Documentation is the first place to visit. You can find many useful resurces, as well as an official NetCDF User Guide, tutorial, programming notes and examples.
• The Official NetCDF Support Page can help in solving problems with NetCDF, with links to FAQ, archives, mailing lists, and email support.
• The Imperial College London Website has a nice section on NetCDF, with short instructions on getting and building NetCDF, a user guide, and a user tutorial with practical examples.

NetCDF Books

Books that specifically cover NetCDF are hard to find, and you will usually find chapters about NetCDF in books about GIS and geosciences. A few Fortran books also mention NetCDF. This does not pose a problem, as there are many quality NetCDF resources available online.

Conclusion

NetCDF has been around for decades, and it’s not going anywhere soon. While it’s a solution for niche applications, NetCDF is still alive and kicking, and it’s widely used to store scientific data around the globe.

Since climate science can’t wait for developers to ensure compatibility with every new software release, NetCDF conveniently offers backwards compatibility. If a lab has a data set from the late nineties, it can still reload it and use it as a historic model using the latest NetCDF release, which is vital in scientific use-cases.

On top of hardcore science applications, NetCDF also plays a prominent role in several industries, namely those that rely on GIS for their day to day operation.

NetCDF is an open standard, free to use and port to different platforms, which is great news for people interested in using it, but there is a downside to having an open, yet niche software solution — there aren’t that many resources around and the community is limited.

Further Reading and Resources

We have more guides, tutorials, and infographics related to mathematical and scientific computing:

• LabVIEW Programming Introduction and Resources: a scientific programming system, used to interface with hardware systems.
• MATLAB Resources: the industry standard in high level mathematical computation.
• Forth Programming Resources: scientific programming designed especially to interface with hardware.

What Code Should You Learn?

Confused about what programming language you should learn to code in? Check out our infographic, What Code Should You Learn? It not only discusses different aspects of the languages, it answers important questions such as, “How much money will I make programming Java for a living?”

What Code Should You Learn?