11 Project Management

This topic deserves a book in its own right. It’s easy to think of an Open Source Project as source code. It’s not. The source code is a major part of the project, however probably only 50% of the effort goes into the code. We have many stakeholders in a project including: contributors, users, security, distros, and competitors. The project needs documentation, build, test, bug reporting and many other elements.

You may have seen the sketch in The Life of Brian which begins with John Cleese asking the question “What have the Romans Ever Done for Us?” and somebody replies The Aqueduct. Within one minute they list Water Supply, Roads, Schools, Sanitation, Police, Laws, Medicine, Wine and Public Health. It’s much the same with Open Source. Of course we have source code, however we also have Build, Test, Platform Support, Documentation, User Support, Security, Release Engineering, Localisation and other matters that require time and attention.

You are probably not surprised to learn that most stakeholders consider their concern should be the top priority for the project. The challenge is that there are many stakeholders and therefore many top priorities. When dealing with a stakeholder’s issue, they frequently say All you have to do is bla bla bla. In my head, I hear the words in a slightly different order. I hear You have to do it all.

The difficulties of maintaining an open-source project are well explained in this article: https://steemit.com/opensource/@crell/open-source-is-awful from which I have copied this cartoon:

I will quote the following as this matches my experience.

If most businesses are using Open Source code for free, how are the developers compensated for that real time and effort? In the majority of cases the answer is with verbal abuse and emotional blackmail.

The very large projects (the Linux kernel, the Firefox web browser, etc.) end up with a few smart companies realizing it’s in their self-interest to fund full time development, and most of their work ends up being non-volunteer. They’re not the problem. The problem is the mid-range to small projects, maintained by volunteers, that get short-shrifted. People don’t value free.

Not a month goes by without some maintainer of an Open Source project throwing up their hands in frustration and walking away because of burnout. Burnout caused invariably by the demands that people make of their free time. The code was free, so why isn’t free support and personalised help available for life???

I am astonished at the verbal abuse I have received. About every three years I receive an email from somebody I have never met thanking me for my efforts. I get daily emails of criticism and complaint. I will not name a French Engineer on whose behalf I have spent hundreds of hours. Not once has he expressed appreciation. His emails and public posts of criticism are brutal.

When somebody provides a patch, they seldom provide test code, update the documentation or modify the build scripts. The feature is often incomplete. For example, in adding a new platform, nobody has ever provided platform specific code in src/version.cpp and src/futils.cpp. Sometimes they break all the sample applications. When I ask them to finish the job, they say: “oh you can do that.”. Nobody ever maintains or supports their patch. Contributors frequently refuse to modify a patch when asked to do so in a review.

I have found recruiting contributors to be very challenging and difficult. I appreciate the work done by everybody who has contributed. The future of Exiv2 is a matter for the Community. Perhaps this book will inspire somebody to maintain Exiv2 or write a replacement.

How to be an AB

I’ll leave you to figure out what an AB is. It might be Annoying Bikeshedder. They come in different versions.

The first is a DAB who deliberately blocks progress. Before we used GitHub, the only way to release a new version of Exiv2 was via the web-site. Sure we could tag a version in SVN, however this was passive. I’m not aware of any central organisation which tracks the versions of projects. So, publishing requires the password to enable the site to be updated. And there’s only one person who knows the password. There are other versions of this kind of obstruction. For example, you can own the domain registration or hosting contract and allow them to expire.

I think the need for a web-site for the project has been mostly replaced by GitHub. We can publish new releases (and pre-releases). However the effort to transfer all project resources to GitHub is considerable. We have had complaints about the repository being too big, so we have an SVN repository for team resources such as old releases, this book, the project logo, minutes of team meetings and so on. And while I understand the team’s hostility to SVN, no sensible alternative has been proposed.

The first couple of releases I published on GitHub were not instantly tagged. Guess what? Within hours, somebody complained. People have complained about the version numbering scheme. GitHub automatically generates bundles when you create a release. Somebody complained about them having different checksums from the bundles on exiv2.org Somebody complained that the pre-release web-site was too similar to the release web-site (although every page is labelled) so I put “Pre-Release” on the background of every page. You might expect that complaints would include words of appreciation for the effort to make the release. You’d be wrong. Complaints are normally abrupt. Words such as Please and Thank You are seldom used by members of the Community.

On the subject of web-sites, I must mention DOS attacks. DOS is Denial of Service. Occasionally, exiv2.org will deliver a message about being severely busy. This is because the web-site is being subjected to about 100 HTTP get requests per second. Every request is from a different IP address. These attacks can continue for several days. Why does somebody attack exiv2.org in this way? How can you defend exiv2.org from such an attack? We’ll call this MAB for Malicious.

How about TAB which is to change the project tools. Git came close to killing me. I know many people love Git and think it’s the greatest thing ever invented. I don’t. I worked on Acrobat at Adobe. A big project. When I retired in 2014, there were about 200 developers who had been working for 20 years on 25 million lines of code. To build it, you need 100GBytes of free space for the generated binaries. How can git handle such a monster when every repos has 100% of the code and 100% of the project history? Nobody has given me an answer.

When we adopted Git, it took me 2 years to figure out how to submit a PR. I purchased the book Pro Git. It doesn’t cover PRs. So, the only way to submit code is undocumented. I am very grateful to Luis and Andreas S for helping me with Git. I eventually wrote this on a card:

The funny brown marks were added by our cat Lizzie. I’d just written this card when she arrived in my office fresh from a hunt in the garden. I don’t know what she did to the card. She’s never done anything like this before or since. She expressed her opinion of git.

Another flavour is the AAB which I reserve for the fuzzing police. The A stands for Aggressive. I’ve discussed my dislike of these people here: 10.2 The Fuzzing Police

Or there’s the review mechanism which I’ll dub RAB You insist:

1. All code changes must be approved.
2. No contributor can approve their own change.
3. Nobody reviews or approves any code change.

There are many other forms of AB. For example, there is legal LAB. This involves a legal challenge. You say “We might be infringing somebody’s patent!”. This is particularly effective as you don’t need to provide evidence. Even if there is a written legal opinion you can refute that with the words: “The legal opinion has not been tested in court.”. The case has not been tested in court for the obvious reason that it is not illegal. This show-stopper was used by two contributors to block BMFF support in Exiv2 v0.27.3. I received more than 100 emails from users asking “What is the legal problem?”, so I called a meeting on Zoom and users on 5 continents attended. The two contributors who raised the show-stopper did not bother to turn up, although one emailed later to say “Apologies. I fell asleep on the couch and missed the meeting.”.

How about this method? You complain about a font being used. We’ll call this FAB. This is very effective because you’re only asking for a 100% reformat of the book and all the graphics. That’s not much to ask. When I designed the Exiv2 Logo, a contributor asked for the font to be changed. I proposed alternatives and received no response.

As for myself, I am a CAB where C stands for clever. However I am an AB and that’s why I’ve found it difficult to recruit and retain contributors.

There are so many ways to incur the outrage of stakeholders. And so many ways in which people can and do complain. Working with such people is very unpleasant. However, working with solid contributors such as Alex, Christoph, Dan, Kev, Luis and Milos is a lot of fun and intellectually rewarding. When I released v0.25, a contributor in Peru said on Facebook Robin should get a medal for his work. Exiv2 would have died years ago without his commitment. So I asked my family to write to the UK Government to propose that I be given an honour. The family silently refused. Alison comforted me by saying Nobody is ever going to thank you for working on Exiv2.

Solutions to the issue of AB Users

You’ve probably now realised that an AB User is an abuser. I do not know why members of the Community indulge in abusive behaviour. Feelings of superiority? I don’t know. I assure you that it is very unpleasant to be the subject of abuse.

There are ways to fix on-line abuse. We could do what the Fuzzing Police do. They do not negotiate with any project. They arrive unexpectedly and deliver their message. And they track your response and performance.

The Community would be well served by a similar task force to investigate complaints about the behaviour of stakeholders and maintainers. There are undoubtably stakeholders who would report me. However I would be able to complain about a stakeholder if I felt they had behaved inappropriately.

For sure, I would not welcome my performance being reviewed. However a fair, honest and independant review of an issue would be helpful. The stakeholder and I would shake hands and learn from the situation. If the stakeholder or maintainer do not attend the review, their github account should be suspended.

Another possible solution would be to register a complaint about an individual. In the same way as a bug report can be opened on a project, it would be helpful to open a complaint about an individual. And that complaint can only be closed by the person who opened it. I suspect, we’d quickly discover two things:

1. A few people raise many complaints about other people.
2. There are many open complaints about a few people.

While there is no sanction for a stakeholder being unreasonable, bad behaviour will continue. A solution is possible. If you are thinking I never contribute, therefore this does not affect me!, I ask you to think again if you use open-source. When maintainers are abused and leave their project, everybody looses out.

Will the Community do something about on-line abuse?

The Future of Open Source

Historically, open source was a collection of one-man projects of limited scope. I believe many of the early projects were rewrites of Unix utilities that ran on a few operating systems. Users were expected to download the code and make the most of it. Before the web, contact with the author was limited.

Things have changed. A library such as Exiv2 is installed on a billion machines and most users are unaware of the existence of the library on their platform. However the view from the Exiv2 Captain’s Bridge is of many more developers demanding help with an ever increasing number of platforms. And they want shorter release cycles. One thing remains the same. The code is being maintained by one person. This model is broken. The pressure on the maintainer is growing exponentially. When combined with on-line abuse and criticism, the maintainer’s plate is toxic. Something must be done about on-line abuse.

Some projects have been able to build a business based on an open-source project. Examples are Consulting and Premium Support. I’ve never thought of any way to generate revenue for Exiv2. Occasionally, I have responded to feature requests from a business with a proposal to undertake paid consultancy. Such proposals were silently rejected without discussion. So, businesses are happy to request a 500 hour project, however they are unwilling to pay for the effort.

Being retired, I can spend time almost every day dealing with Exiv2. However, contributors in employment have higher priorities in their life. When contributors are between jobs, they are even less likely to work on Exiv2. If the project could apply for a funding grant, contributors between jobs could be paid to develop features.

Something has to change and here are some ideas about funding and recognition.

1. GSoC give students a bounty. That program could be duplicated by Adobe, Facebook, Microsoft, Apple and other Tech Giants.
2. Large organizations could make donations to a foundation from which projects could apply for grants.
3. Github could carry ads and share the revenue with open source projects.
4. A federal program like Darpa.
5. Community recognition like the oscar or grammy awards.

What’s going to happen? I don’t know.

TOC

11.1 C++ Code

Exiv2 is written in C++. The code has taken a great deal of inspiration from the book Design Patterns: Elements of Reusable Object-Oriented Software. The code makes considerable use of STL containers such as vector, map, set, string and many others. The Exiv2 code started life as a 32-bit library on Unix and today builds on 32 and 64 bit systems running Linux, Unix, macOS and Windows (Cygwin, MinGW, and 7 editions of Visual Studio). It can be built by GCC, Clang and Visual Studio. Although the Exiv2 project has never supported Mobile Platforms or Embedded Systems, it should be possible to build for other platforms with modest effort.

The code on the 0.27-maintenance branch and earlier, was written to the C++ 1998 Standard. The Exiv2 main branch requires a C++11 Compiler. The Exiv2 main branch is a major refactoring of the code and provides a new API. The project maintains a series of v0.27 “dot” releases on branch 0.27-maintenance for security updates. These releases are intended to ease the transition of existing applications in adapting to the new ‘main’ API. As of v0.27.3, the “dots” can be compiled for C++11 and C++14 using the following cmake options:

$ cmake .. -DCMAKE_CXX_STANDARD=11 -DCMAKE_CXX_FLAGS=-Wno-deprecated

However, compiling “the dots” in C++11 or C++14 uses compiler legacy compatibility to compile std::auto_ptr. The main branch code replaces std::auto_ptr with std::unique_ptr and therefore cannot be compiled with C++98. ‘main’ can be compiled with C++11 and later including C++14, C++17 and C++20.

TOC

11.2 Build

The build code in Exiv2 is implemented using CMake: cross platform make. This system enables the code to be built on many different platforms in a consistant manner. CMake recursively reads the files CMakeLists.txt in the source tree and generates build environments for different build systems. For Exiv2, we actively support using CMake to build on Unix type plaforms (Linux, macOS, Cygwin, MinGW, NetBSD, Solaris and FreeBSD), and several editions of Visual Studio. CMake can generate project files for Xcode and other popular IDEs.

Exiv2 has dependencies on the following libraries. All are optional, however it’s unusual to build without zlib and libexpat.

Name	Purpose
zlib	Compression library. Necessary to support PNG files
expat	XML Library. Necessary to for XMP and samples/geotag.cpp
xmpsdk	Adobe Library for xmp. Source is embedded in the Exiv2 code base
libcurl	http, https, ftp, ftps support
libssh	ssh support
libiconv	charset transformations
libintl	localisation support

Conan

Starting with Exiv2 v0.27, we can use Conan to build dependencies. I very much appreciate Luis bringing this technology to Exiv2 as it has hugely simplified building with Visual Studio. In the CI builders on GitHub, Conan is also used to build on Linux and macOS. The CI/Windows builds are performed by appveyor. The Visual Studio builds are performed using Conan, the Cygwin64 and MinGW/msys2 CI script installs the dependencies directly.

Prior to using Conan, the build environment for Visual Studio was hand built for Visual Studio 2005 and relied on Visual Studio to convert to the edition in use. Additionally, source trees for dependencies were required in specific locations on storage. We did support CMake on Visual Studio, however it required a 500 line cmd file cmakeBuild.cmd to download and build the dependencies. The effort to create and maintain that apparatus was considerable.

Conan brings a fresh approach to building dependencies. Every dependancy has a “recipe” which tells Conan how to build a dependency. The recipes are held on servers and are fetched from remote servers on demand. Exiv2 has a file conanfile.py (written in python) which tells Conan which dependencies are required. Conan fetches the recipe, builds the dependency and caches it for future use. When dealing with very large libraries such as openssl, the recipe might pull prebuilt binaries for your requested configuration. For more modest libraries (such as expat and zlib), the recipe will fetch the source and build. Conan caches binary dependencies in your ~/.conan directory. This arrangement means that you seldom build dependencies as they are usually in the cache.

I have always supported the plan to use CMake as our common build platform and invested a lot of effort in cmakeBuild.cmd. Using Conan with Visual Studio is much superior to our prior methods of working with CMake and Visual Studio.

Luis has made other very useful contributions to Exiv2. He rewrote most of the CMake code. It’s shorter, more robust, more flexible and easier to understand. He also introduced CPack which packages both our Source Bundle and binary builds for public release on https://exiv2.org

The final element of CMake which we have not yet deployed is CTest. Perhaps one day this will be investigated and integrated into the Exiv2 build environment.

Thank You to Luis for introducing Conan to Exiv2 and all your outstanding work with CMake.

The documentation about using Conan with Exiv2 is in README-CONAN.md

Build Options

There are numerous build options provided for Exiv2. The documentation for this is in README.md. Most of the options concern dependencies, the configuration:

Description	Choices
build_type	debug or release
kind	static or shared
configuration	32 or 64
C-runtime	shared or static
compiler	GCC or Clang or 2008 … 2019
language standard	98 or 11 or 14 or 17

And we have not considered the selection of build dependencies required by the user. For example, support for PNG, XMP, Localisation, Web Protocols and Character Conversions.

There are a number of convenience options to build packages for release, on-line documentation, unit_tests and ASAN support. ASAN is the “Address Sanitiser”. When this option is selected, the code is built and instrumented with address checking. Every memory access is tested before use. This has a considerable performance penalty and is only intended for test and development purposes. It shouldn’t be used in production.

An interesting option is BUILD_WITH_CCACHE. This option can dramatically speed up rebuilding by caching compiled code. If CCache determines that there are no code or configuration changes, the compiled object from the cache is used. This can accelerate build times by 100x.

While lots of effort has been invested in the CMakeLists.txt and *.cmake files, some users may want something that has never been investigated by Team Exiv2. For example, we do not support building for ARM processors. It’s highly likely that Exiv2 can be successfully built for those machines and the recommended way is to use options such as -DCMAKE_CXX_FLAGS to introduce the necessary compiler and linker options. Other examples of “possible, yet not supported” are to request Visual Studio to use Clang, or its own CMake support, or its own Package Manager.

Regrettably there are users who look to Team Exiv2 to support every possible configuration. This is impossible. The essential thing is that we have built and tested our code on many platforms. Users will always think of novel ways in which to build and deploy. I worked with a user to build Exiv2 on OS/2. I had no idea that OS/2 is alive and well.

TOC

11.3 Security

This is discussed in detail here: 10 Security.

11.4 Documentation

The following types of documents used in Exiv2. They are:

	Documents	Creator	Comment
1	The exiv2 man page exiv2.1	man, man2html	Unix mark up syntax (troff)
2	User Manuals	Markdown	User Documentation
3	API Documentation	Doxygen	From .cpp and .hpp files
4	Web site pages	Scripted (mostly)
5	Release Notes	Markdown	GitHub PRs
6	GitHub Wiki Pages	Markdown	https://github.com/Exiv2/exiv2.wiki.git

Life would be simpler with a single way to define documents and scripts to propagate changes to their destination. In some ways, this has been done. However the nature and format of the document classes are different, and the current arrangements will not yield to much more simplification.

Markdown

I’m very pleased by Markdown. Perhaps one day, the utility man will support this format as that would simplify the maintenance of exiv2.1.

This book has been written in markdown, and the User Documentation in the release. Markdown is also used extensively on GitHub for discussion and the project Wiki pages.

Doxygen generated API Documentation

The API documents are generated from comments in the C++ code. Doxygen generates UML diagrams of the class hierarchy, table of contents, navigation links and more. It does a very nice job with modest effort from Team Exiv2.

Release Documentation

Creating release notes takes quite a lot of time and effort. When Exiv2 v1.00 is released, the GitHub tools will probably do an adequate job. I feel manually creating the release notes is a better approach. The Release procedure is discussed here: 11.9 Releases

Exiv2 man page

I don’t like the man page because it’s in Unix man page troff syntax which is arcane and unfamiliar. However man pages are very useful and valuable for users. The man page is stored in man/man1/exiv2.1 When editing the man pages, I inspect the changes with commands such as:

$ cd \<exiv2dir\>
$ env MANPATH=$PWD/man:$MANPATH man exiv2 | grep EXIV2
EXIV2(1)                                                              EXIV2(1)
                                  Nov 6, 2020                         EXIV2(1)
$ 

The man pages are converted with man2html for display on https://exiv2.org/manpage.html. In the release scripts, man page is converted to PostScript and then to PDF as follows:

$ env MANPATH=$EXIV2HOME/man:$MANPATH man -t p | ps2pdf

TOC

11.5 Testing.

This is discussed in detail here: 8 Test Suite.

11.6 Samples

Exiv2 has sample applications which have their own documentation: README-SAMPLES.md. In Exiv2 v0.27.3, there are 17 samples applications and 19 test programs. The test programs are intended for use by the test suite and are not installed on the user’s computer.

The following programs are built and installed in /usr/local/bin.

Name	Purpose
addmoddel	Demonstrates Exiv2 library APIs to add, modify or delete metadata
exifcomment	Set Exif.Photo.UserComment in an image
exifdata	Prints Exif metadata in different formats in an image
exifprint	Print Exif metadata in images Miscelleous other features
exifvalue	Prints the value of a single Exif tag in a file
exiv2	Command line utility to read, write, delete and modify Exif, IPTC, XMP and ICC image metadata. This is the primary test tool used by Team Exiv2 and can exercise almost all code in the library. Due to the extensive capability of this utility, the APIs used are usually less obvious for casual code inspection.
exiv2json	Extracts data from image in JSON format. This program also contains a parser to recursively parse Xmp metadata into vectors and objects.
geotag	Reads GPX data and updates images with GPS Tags
iptceasy	Demonstrates read, set or modify IPTC metadata
iptcprint	Demonstrates Exiv2 library APIs to print Iptc data
metacopy	Demonstrates copying metadata from one image to another
mrwthumb	Sample program to extract a Minolta thumbnail from the makernote
taglist	Print a simple comma separated list of tags defined in Exiv2
xmpdump	Sample program to dump the XMP packet of an image
xmpparse	Read an XMP packet from a file, parse it and print all (known) properties.
xmpprint	Read an XMP from a file, parse it and print all (known) properties..
xmpsample	Demonstrates Exiv2 library high level XMP classes

Most of the programs are about 100 lines of C++ and do simple tasks to demonstrate how to use the library API. Three of the programs are substantial. They are: exiv2, geotag and exiv2json

The Exiv2 command-line application enables users to manipulate metadata in images using most of the features of the library. Being a general utility, it has about 4000 lines of code. The length of the program proves the point that it is full featured, however the quantity of code rather obscures the use of the library APIs.

Exiv2 has always resisted the temptation to provide a GUI version of the program as that would involve considerable cross-platform development and user interface skills. As Andreas Huggel summarised: Exiv2 does depth, not breadth. Providing a GUI would lead the project away from metadata into the world of the User Experience.

TOC

11.7 Users

User Support is very time consuming. I prioritise working with users as the most important aspect of the project. Occasionally, in the run-up to a release, I will ask a user to wait. However, my default is to deal with users as quickly as possible. I try to acknowledge and confirm their report within 24 hours and to fix/close issues in one week.

The reason to give them priority is the importance of users to the project. Without users, the project is dead. Without support, users will not use the code. I know this is true because I have taken some sabbaticals to deal with other matters in my life. When I am not active, the number of user reports and requests falls quickly. When I return from my break, the number of user reports immediately increases.

I have been very disappointed by the appreciation shown by users to my attention to their questions. Very few people have the courtesy to use words like “Please” and “Thank You”. Why is this? I don’t know. Moreover, I am astonished by the abuse I have encountered. The on-line behaviour of some users is unacceptable. I have encountered this behaviour from our OEM Engineers when I worked on Adobe PostScript. However, I could refer that to management at Adobe and at the OEM and the matter would be settled. With open-source, I have to accept and ignore this awful behaviour.

I have wondered if the users who behave this way believe that I am a business and have let them down in some way. Open source is a Community. In reporting a bug, they are participating in the development process. I always thank users for reporting issues. It’s sad that they seldom have the courtesy to thank me for fixing the issue.

A member of my family is the Principal of a College. We were discussing the behaviour of parents of students. She said the one word you must never use with a parent is No. It’s the same with open source users. It’s pointless to say No because they will not accept this. A good example is Lens Recognition. The configuration file was added in 0.26 to enable users to fix lens recognition issues by updating an ascii file. Users frequently demand that I fix their lens in C++ to save them a minute to update ~/.exiv2. Saying No is pointless.

On a more positive note about dealing with users, I have enjoyed many on-line discussions with frequent visitors to exiv2.org. For sure, I include Arnold, Mikayel, Alan and Steve in this group and there are many more. If you are courteous, I am always pleased to hear from you. We are a Community with a shared vision of working together. Thank You for participating.

TOC

11.8 Bugs

Exiv2 has used three bug tracking systems during its 17 year life. In the early days, issues were stored on a forum hosted by Yahoo. (Who?). About the time that I joined the project in 2008, Redmine was installed to track issues on exiv2.org. I really like Redmine. It has a nice UI with good search, cross referencing, and reporting tools. I very much appreciated the Rest API to query and download data in JSON format. I wrote a python script to generate various reports to monitor release progress.

We moved the code to GitHub when Exiv2 v0.26 was released in April 2017. I didn’t know that GitHub provided issue tracking and many other project management tools. We could consider closing exiv2.org in future and providing all project resources from GitHub.

I believe the GitHub Rest API provides a mechanism with which we could collect data. I could use that to generate report similar to my Remine/python/JSON code.

I’m pleased with GitHub. For sure, it’s a “one stop shop” for a project. They provide good tools. The best aspect of GitHub is that I met Luis and Dan on GitHub. And numerous other frequent contributors (acknowledged on page 2 of this book). For sure GitHub has brought more order to the world of open-source.

TOC

11.9 Releases

Releases (both RCs and GMs) are published on GitHub. Users can receive notifications by subscribing an RSS reader to: https://github.com/exiv2/exiv2/releases.atom. There is a summary of releases here: https://github.com/Exiv2/exiv2/releases. All releases (both RCs and GMs) are available from exiv2.org at: https://exiv2.org/archive.html

Making a new release is very time-consuming. The business of performing the builds and updating the web-site is straightforward. It is totally scripted and easy to perform.

However the time involved in determining the contents of the release, updating the release notes, submitting all the PRs, testing and documenting is considerable.

Moreover, I like to publish release candidates. I never make code changes between the final release candidate the Golden Master. Let me define the terminology and the version numbering scheme.

Version	Name	Status	Purpose
0.27.7.3	Exiv2 v0.27.3	GM	Golden Master. This is the final and official release.
0.27.3.2	Exiv2 v0.27.3.2	RC2	Release Candidate 2.
v0.27.3.20	Exiv2 v0.27.3.2	RC2 Preview	Dry-run for release candidate. For team review.
v0.27.3.81	Exiv2 v0.27.3	Security Fix	Security Release
v0.27.3.29	Exiv2 v0.27.3.29	Development	Should never be installed for production.
v0.27.4.9	Exiv2 v0.27.4.9	Development	Should never be installed for production.
v0.27.99	Exiv2 v1.00.0.9	Development	Should never be installed for production.

The release procedure is documented here: svn://dev.exiv2.org/svn/team/website/Checklist.txt

It typically takes about 3 months to make a release and consumes 100-400 hours.

In month 1, the release and release notes are developed. Depending on the complexity of the features being added for release, this can be 40 to 200 hours of work. In month 2, we respond to matters arising from RC1. As with month 1, it’s usually 40-200 hours of work to reach RC2. In month 3, we do nothing. It’s an afternoon’s work to publish GM and tag the release.

If an issue arrives between RC2 and GM and it is decided to change code, I always accept a schedule delay and publish RC3.

It’s only fair to say that others will say “Oh, it shouldn’t be so complicated.”. And I agree. It shouldn’t. I’ve been the Release Engineer for at least 6 releases and have not discovered any tricks to eliminate the work involved. You could just tag the current development branch, bump the version number and hope for the best. Don’t.

Once we release Exiv2 v1.00, I hope that a further dot release from Exiv2 v0.27-maintenance will never be required. I suspect we will see Exiv2 v0.27.5 in 2022 with security fixes which will need to be ported from main.

The Macro EXIV2_TEST_VERSION

This enables application code to easily test for a minimum version of exiv2.

#define EXIV2_TEST_VERSION(major,minor,patch) \
    ( EXIV2_VERSION >= EXIV2_MAKE_VERSION(major,minor,patch) )

For example, to safely call image->setIccProfile(), this is compile time safe and will not link Exiv2::Image::setIccProfile() if you are using any version of Exiv2 prior to 0.27.0 when this API was added.

#ifdef EXIV2_TEST_VERSION(0,27,0)
    image->setIccProfile(....);
#endif

TOC

11.10 Platforms

There are several parts of Exiv2 which are platform specific. Additionally the platform dependent function getopt() in the C-runtime library is never used.

src/getopt.cpp

The command-line handler getopt() is used by the Exiv2 command-line application and by samples/metacopy.cpp, samples/getopt-test.cpp and samples/toexv.cpp. In the early days of Exiv2, getopt() was provided by the C-runtime library. When support for msvc was added, the code in src/getopt.cpp was added. Relying on the C-runtime library revealed differences between platforms and between platforms and src/getopt.cpp. It was decided to ensure consistent behaviour to use src/getopt.cpp on all platforms.

src/version.cpp

I’m rather proud of the code in src/version.cpp. In addition to reporting the version number, version.cpp reports the build settings used to compile the code. It also inspects the run-time environment to determine shared objects which have been loaded. I added this to the test harness because I was suspicious that we were not testing the correct shared object.

I’ll write more later about how this is achieved.

src/futils.cpp

This file has utility code for dealing with files and paths. For example, there is a base64 encoder/decode which is used to manage paths of the form data:abc…. There is also a URL parser for decomposing URLs to determine protocol, serverer, user, password and other URL paraphernalia.

.../exiv2/0.27-maintenance $ grep -e ') {' -e getProcessPath src/futils.cpp | grep -v if | grep -v for | grep -v while
    char to_hex(char code) {
    char from_hex(char ch) {
    std::string urlencode(const char* str) {
    char* urldecode(const char* str) {
    void urldecode(std::string& str) {
    int base64encode(const void* data_buf, size_t dataLength, char* result, size_t resultSize) {
    long base64decode(const char *in, char *out, size_t out_size) {
    Protocol fileProtocol(const std::string& path) {
    Protocol fileProtocol(const std::wstring& path) {
    std::string pathOfFileUrl(const std::string& url) {
    std::wstring pathOfFileUrl(const std::wstring& wurl) {
    std::string getProcessPath()
.../exiv2/0.27-maintenance $ 

The function std::string getProcessPath() determines the process path and is similar to code in src/version.cpp.

UNICODE path support

There is a build option EXIV2_ENABLE_WIN_UNICODE which may be used on Windows when building with Visual Studio. This is useful for applications using wchar_t path strings. I believe this is the default for most applications using the Qt libraries. This version of the library can be used by other applications such as command-line utilities which link wmain(). When the library is build with UNICODE path support, the char versions of the API are also built.

Please be aware that this feature only applies to paths. Using UNICODE in UserComments and other Tags is explained in the Exiv2 man page and discussed in more detail below under the title Character Set Encoding.

Here is a typical build sequence to build with UNICODE path support for Visual Studio:

>cd    <exiv2dir>
>mkdir build && cd build
build>conan install .. --profile msvc2019Release --build missing
build>cmake .. -G "Visual Studio 16 2019" -DEXIV2_ENABLE_WIN_UNICODE=On
build>cmake --build . --config Release

When the build finishes, you can inspect the setting and read the export table of ImageFactory::open() functions as follows. The output presented here has been simplified for presentation in this book.

>cd <exiv2dir>
>mkdir build && cd build
build>bin\exiv2 -vVg unicode
exiv2 0.27.3
processpath=C:\Users\rmills\gnu\github\exiv2\0.27-maintenance\build\bin
executable=C:\Users\rmills\gnu\github\exiv2\0.27-maintenance\build\bin\exiv2.exe
library=C:\Users\rmills\gnu\github\exiv2\0.27-maintenance\build\bin\exiv2.dll
have_unicode_path=1
build>dumpbin/exports bin\exiv2.dll | grep ImageFactory | grep open > foo.txt
Exiv2::Image::auto_ptr Exiv2::ImageFactory::open
... (class std::basic_string<char,struct std::char_traits<char>,class std::allocator<char> >
...  const & __ptr64,bool)
Exiv2::Image::auto_ptr Exiv2::ImageFactory::open
...(class std::basic_string<wchar_t,struct std::char_traits<wchar_t>,class std::allocator<wchar_t> >
... const & __ptr64,bool)
Exiv2::Image::auto_ptr Exiv2::ImageFactory::open(unsigned char const * __ptr64,long)
Exiv2::Image::auto_ptr Exiv2::ImageFactory::open(class std::auto_ptr<class Exiv2::BasicIo>)build>

Caution: You should use the “Developer Command Prompt” in Visual Studio Studio to ensure you have the utilities dumpbin and undname on your path. To filter the output with grep, you will need to ensure grep is on your PATH.

Performing the same tests on a default build (without UNICODE path support), shows one less entry point because the library does not provide the UNICODE entry-point Exiv2::ImageFactory::open(std::wstring,const bool& ).

The UNICODE library has been build with both wstring and string entry points. All samples (except exifprint.cpp) are built to use the char entry points, so you can the test suite runs.

build> cmake --build . --config Release --target tests

The test suite passes because our test image paths do not need UNICODE path support. See README.md for more information about running the test suite.

When you build the library with UNICODE path support, the sample program samples/exifprint.cpp is built as a UNICODE application. You can observe the UNICODE path support in the following way:

build>copy %USERPROFILE%\Stonehenge.jpg %USERPROFILE%\√.jpg
build>bin\exifprint.exe %USERPROFILE%\√.jpg | grep Date
Exif.Image.DateTime                          0x0132 Ascii      20  2015:02:18 21:42:57
Exif.Photo.DateTimeOriginal                  0x9003 Ascii      20  2015:02:15 12:29:49
Exif.Photo.DateTimeDigitized                 0x9004 Ascii      20  2015:02:15 12:29:49
Exif.NikonWt.DateDisplayFormat               0x0003 Byte        1  0
Exif.GPSInfo.GPSDateStamp                    0x001d Ascii      11  2015:02:15

build>cd ..\build_no_unicode
build_no_unicode>bin\exifprint.exe %USERPROFILE%\√.jpg | grep Date
build_no_unicode>

When you build the library with UNICODE path support, wchar_t versions of the following APIs are built:

Exiv2::BasicError
Exiv2::FileIo::FileIo
Exiv2::HttpIo::HttpIo
Exiv2::XPathIo::XPathIo
Exiv2::ImageFactory::create
Exiv2::ImageFactory::createIo
Exiv2::fileExists
Exiv2::fileProtocol
Exiv2::ImageFactory::getType
Exiv2::ImageFactory::open
Exiv2::pathOfFileUrl
Exiv2::readFile
Exiv2::ExifThumb::setJpegThumbnail
Exiv2::FileIo::setPath
Exiv2::XPathIo::writeDataToFile
Exiv2::ExifThumbC::writeFile
Exiv2::writeFile
Exiv2::PreviewImage::writeFile

Character Set Encoding

This is discussed here: https://github.com/Exiv2/exiv2/issues/1258

Being a native English speaker, I find it difficult to understand the use of other character sets. I understand the importance of this to users, however I have no experience of typing anything other than 7-bit ascii.

The Exif specification for ASCII states:

2 = ASCII An 8-bit byte containing one 7-bit ASCII code. The final byte is terminated with NULL.

Exiv2 does not enforce the 7-bit condition. You can read/write any 8-bit value, including NUL. This is discussed: https://github.com/Exiv2/exiv2/issues/1279#issuecomment-689053734

You can use charset= on the ‘Comment’ tags which are:

714 rmills@rmillsmbp:~/gnu/github/exiv2/0.27-maintenance/build $ taglist ALL | grep '\tComment,' | cut -d, -f -6
Photo.UserComment,  37510,  0x9286, Photo,  Exif.Photo.UserComment, Comment
GPSInfo.GPSProcessingMethod,    27, 0x001b, GPSInfo,    Exif.GPSInfo.GPSProcessingMethod,   Comment
GPSInfo.GPSAreaInformation, 28, 0x001c, GPSInfo,    Exif.GPSInfo.GPSAreaInformation,    Comment
715 rmills@rmillsmbp:~/gnu/github/exiv2/0.27-maintenance/build $ 

The format of Exif Comment values include an optional charset specification at the beginning. Comments are stored as Undefined tags with an 8-byte encoding definition follow by the encoded data. The charset is specified as follows:

[charset=Ascii|Jis|Unicode|Undefined] comment
charset=Undefined is the default

$ exiv2 -M'set Exif.Photo.UserComment charset=Ascii My photo' x.jpg
$ exiv2 -pa --grep UserComment x.jpg
Exif.Photo.UserComment       Undefined  16  My photo
$ exiv2 -pv --grep UserComment x.jpg
0x9286 Photo     UserComment Undefined  16  charset=Ascii My photo

$ exiv2 -M'set Exif.Photo.UserComment charset=Unicode \u0052\u006f\u0062\u0069\u006e' x.jpg
$ exiv2 -pa --grep UserComment x.jpg
Exif.Photo.UserComment             Undefined  18  Robin
$ exiv2 -pv --grep UserComment x.jpg
0x9286 Photo      UserComment             Undefined  18  charset=Unicode Robin

$ exiv2 -M'set Exif.GPSInfo.GPSProcessingMethod HYBRID-FIX' x.jpg
$ exiv2 -pa --grep ProcessingMethod  x.jpg
Exif.GPSInfo.GPSProcessingMethod           Undefined  18  HYBRID-FIX
$ exiv2 -pv --grep ProcessingMethod  x.jpg
0x001b GPSInfo   GPSProcessingMethod       Undefined  18  HYBRID-FIX

Exif Comments and characters outside the Basic Multilingual Plane

See: https://github.com/Exiv2/exiv2/issues/1279#issuecomment-689053734

IPTC and CharacterSet

IPTC Data Section 1 (Envelope) may include a Record 90 (CharacterSet). I know nothing about this record. It was briefly discussed here: https://github.com/Exiv2/exiv2/issues/1203

JSON Support

I really admire the code in samples/JZon.cpp. The Swedish Engineer who created this made a super-human effort to fix a bug for me. However, the latest public version of that code has lost his fix. I have spoken with him about this and we agreed that I would maintain the copy in the samples/. It isn’t in the library. It is compiled and linked with samples/exiv2json.cpp

TOC

11.11 Localisation

Localisation is documented in README.md.

11.12 Build Servers

At different times, we have used different build server technologies.

1. Appveyor, Travis, GitLab and CodeCov
2. My build script ./build.sh
3. Jenkins and buildbot

Appveyor, Travis, GitLab and CodeCov

Those build systems are provided by GitHub and work very well. To use them, you have to check the appropriate box in the GitHub Branch Settings and add a configuration file to the branch. Exiv2 uses:

CI	Configuration	Comment
GitLib	.gitlab-ci.yml	Linux and Unix
Travis	.travis.yml	Linux, macOS and Unix
Appveyor	appveyor.yml	Visual Studio, Cygwin64 and MinGW64/msys2
Code Cov	codecov.yml	Linux Code Coverage

I wrote the following Appveyor Configuration (YMP) to build Cygwin and MinGW.

init:
  - echo %PYTHON%

environment:
  PYTHON: "C:/Python37-x64"

  matrix:
    - APPVEYOR_BUILD_WORKER_IMAGE: Visual Studio 2019
      BUILD: MINGW64
      INTEGRATION_TESTS: 1
      ARCHITECTURE: x86_64
      UNIT_TESTS: 1
      WEBREADY: False
      WARNINGS_AS_ERRORS: ON
    - APPVEYOR_BUILD_WORKER_IMAGE: Visual Studio 2019
      BUILD: CYGWIN64
      INTEGRATION_TESTS: 1
      ARCHITECTURE: x86_64
      UNIT_TESTS: 1
      WEBREADY: False
      WARNINGS_AS_ERRORS: ON

shallow_clone: true

install:
    - echo %APPVEYOR_BUILD_FOLDER%
    - if "%BUILD%"=="MINGW64" set "PATH=c:\msys64\mingw64\bin;c:\msys64\usr\bin;c:\msys64\usr\local\bin;"
    - if "%BUILD%"=="MINGW64" C:\msys64\usr\bin\bash -c "python -m pip install --upgrade pip;pip3.exe install lxml ; for i in base-devel git coreutils dos2unix tar diffutils make  \
         mingw-w64-x86_64-toolchain mingw-w64-x86_64-gcc      mingw-w64-x86_64-gdb     \
         mingw-w64-x86_64-cmake     mingw-w64-x86_64-gettext  mingw-w64-x86_64-python3 \
         mingw-w64-x86_64-libexpat  mingw-w64-x86_64-libiconv mingw-w64-x86_64-zlib    \
         mingw-w64-x86_64-gtest ; do (echo y | pacman -S $i) ; done
    - cd %APPVEYOR_BUILD_FOLDER%
    - if "%BUILD%"=="CYGWIN64" set "PATH=c:\cygwin64\usr\local\bin;c:\cygwin64\bin;c:\cygwin64\usr\bin;c:\cygwin64\usr\sbin;"
    - if "%BUILD%"=="CYGWIN64" C:\cygwin64\bin\bash -c "wget https://raw.githubusercontent.com/transcode-open/apt-cyg/master/apt-cyg ; chmod +x apt-cyg; mv apt-cyg /usr/local/bin"
    - if "%BUILD%"=="CYGWIN64" C:\cygwin64\bin\bash -c "apt-cyg install cmake zlib-devel libexpat-devel libxml2-devel libxslt-devel python38 python38-pip python38-libxml2"
    - if "%BUILD%"=="CYGWIN64" C:\cygwin64\bin\bash -c "/usr/bin/python3.8.exe -m pip install --upgrade pip"

build_script:
    - cmd: set   CMD=mkdir -p build
    - cmd: set   CMD=%CMD%; cd build    
    - cmd: set   CMD=%CMD%; cmake .. -G 'Unix Makefiles' -DCMAKE_CXX_STANDARD=98 -DCMAKE_CXX_FLAGS=-Wno-deprecated
    - cmd: set   CMD=%CMD%; cmake --build . --config Release
    - cmd: rem echo %CMD%
    - cd %APPVEYOR_BUILD_FOLDER%
    - cmd: if "%BUILD%"=="MINGW64" C:\msys64\usr\bin\bash -c "%CMD%"
    - cmd: set   CMD=which python3 python
    - cmd: set   CMD=%CMD%; python --version
    - cmd: set   CMD=%CMD%; build/bin/exiv2 --verbose --version; pwd ; ls -l
    - cmd: set   CMD=%CMD%; cd build ; cmake --build . --config Release --target python_tests
    - cmd: echo %CMD%
    - cd %APPVEYOR_BUILD_FOLDER%
    - cmd: if "%BUILD%"=="MINGW64" C:\msys64\usr\bin\bash -c "%CMD%"
    - cmd: set "PATH=c:\cygwin64\usr\local\bin;c:\cygwin64\bin;c:\cygwin64\usr\bin;c:\cygwin64\usr\sbin;"
    - cmd: set  CMD=rm -rf build
    - cmd: set  CMD=%CMD%; mkdir -p build
    - cmd: set  CMD=%CMD%; cd build
    - cmd: set  CMD=%CMD%;cmake .. -DCMAKE_CXX_STANDARD=98 -DCMAKE_CXX_FLAGS=-Wno-deprecated
    - cmd: set  CMD=%CMD%; make
    - cmd: set  CMD=%CMD%; make python_tests
    - cmd: echo %CMD%
    - cd %APPVEYOR_BUILD_FOLDER%
    - cmd: if "%BUILD%"=="CYGWIN64" C:\cygwin64\bin\bash -c "%CMD%"

My build script ./build.sh

503 rmills@rmillsmm-local:~/gnu/exiv2/team/contrib/buildserver $ svn info build.sh
URL: svn://dev.exiv2.org/svn/team/contrib/buildserver/build.sh
...
504 rmills@rmillsmm-local:~/gnu/exiv2/team/contrib/buildserver $ ./build.sh 
usage: ./build.sh  { --help | -? | -h | platform  | switch | option     | location value }+ 

platform:    all[32] | msvc[32] | linux[32]  | macos      | cygwin | mingw | unix | freebsd | netbsd | solaris
switch:     --source | --debug  | --static   | --clang    | --background
options:   --[no]nls | --video  | --asan     | --status   | --[no]unit     | --[no]publish | --[no]webready
msvc:         --2019 | --2017   | --2015     | --2013     | --2012         | --2010  | --2008
location: --server B | --user C | --builds D | --cpp  {98 | 11 | 14 | 17}  | --stamp stamp 
           --github {rmillsmm,github,E}      | {--tag tag | --branch branch}
505 rmills@rmillsmm-local:~/gnu/exiv2/team/contrib/buildserver $ 

I abandoned Jenkins for several reasons:

1. It was insecure on the MacMini and could be hacked.
2. As configured by me, it was invoking a complicated ssh script.
3. I didn’t understand how to trigger Jenkins from GitHub.
4. I didn’t like the Cygwin ssh server which runs the bash shell.

I decided to forget about Jenkins and focus on the ssh script. So build.sh, parses its command arguments, writes the build script and transfers it by ssh to the appropriate VM. On the machine, rmillsmm, I have VM such as rmillsmm-w10, rmillsmm-ubuntu, rmillsmm-solaris and so on. On Windows, I use the excellant bitvise ssh server and the native server on other platforms.

Bitvise is a very solid server. https://www.bitvise.com/ssh-server. It can be configured for a variety of shells. I use cmd.exe to build on Visual Studio or Cygwin/64 or MinGW/msys2/mingw64. ./build.sh invokes the batch files cmd64.bat, cygwin64.bat or msys64.bat to configure the environment on the build machine. These scripts are discussed and documented in README.md.

Jenkins and buildbot

There are “out of the box” build servers. I evaluated both in 2013.

I didn’t like Google’s buildbot.

I used Jenkins for several years before deciding that it was not working well for me. Please understand that I have no criticism of Jenkins, my unhappiness was caused by my complicated Cygwin bash script. That script has to invoke cmd.exe to build Visual Studio, then launch bash to run the test scripts.

I am glad to say that in Exiv2 v0.27.3, I studied the test environment and documented how to execute the bash scripts from cmd.exe on Windows. The 0.27-maintenance branch now has python code which makes it possible to remove bash from the test. So much of my unhappiness with Jenkins involved running the test suite and perhaps it’s easier to configure Jenkins.

However, as GitHub provides good CI support, we don’t need Jenkins.

TOC

11.13 Source Code

The source code for Exiv2 resides on GitHub https://github.com/exiv2/exiv2

There are team resources stored on subversion: svn://dev.exiv2.org/svn/team.

Here are most (but not all) of the team directories:

731 rmills@rmillsmbp:~/gnu/exiv2/team $ ls -l
drwxr-xr-x+ 73 rmills  staff  2336 25 Oct 16:52 book               this book
drwxr-xr-x+  6 rmills  staff   192  4 Apr  2020 contrib            contributions (including build.sh)
drwxr-xr-x+ 11 rmills  staff   352  8 Jun 12:02 drawings           miscellaneous artwork
drwxr-xr-x+  7 rmills  staff   224  3 Dec  2018 license            license discussions
drwxr-xr-x+ 24 rmills  staff   768 23 Jul 19:11 logo_files         artwork and fonts
drwxr-xr-x+  7 rmills  staff   224 21 Mar  2019 meetings           meeting minutes
drwxr-xr-x+ 98 rmills  staff  3136  8 Jul 11:26 releases           released source and binary builds
drwxr-xr-x+  4 rmills  staff   128 19 May 12:03 rmills             my home-made scripts
drwxr-xr-x+ 15 rmills  staff   480  8 Jul 11:31 website            source for web site and release scripts
732 rmills@rmillsmbp:~/gnu/exiv2/team $ 

TOC

11.14 Web Site

The website source and release procedures are store in subversion. svn://dev.exiv2.org/svn/team. The release process is discussed in detail here: 11.9 Releases

Exiv2 v0.27.5 will be the last release to be published on exiv2.org. The plan is to migrate the web-site to the GitHub Wiki. We will no longer provide binary builds of the code. If folks wish to use exiv2 without building from source, their platform package manager is available on all supported platforms.

TOC

11.15 Servers

We use several servers:

1. Apache (https://exiv2.org and https://pre-release.exiv2.org)
2. Subversion (svn://dev.exiv2.org/svn/)
3. GitHub (https://github.com/exiv2/exiv2)
4. Redmine (https://redmine.exiv2.org)
5. SSH on the buildserver

I am pleased to say that the management of exiv2.org is undertaken by Nehal. I don’t think it’s an onerous task, however I appreciate having this taken off my back.

TOC

11.16 API

This is discussed in detail here: 9. API/ABI Compatibility

11.17 Contributors

It’s very difficult to recruit people to work on open source. In fact, it’s so difficult that I wonder if open source can survive in future. Lots of people have made small contributions to Exiv2, however only a hand-full have made a sustained effort. Furthermore, contributors can disappear for months with no indication of their intention. I’m not criticising anybody for how they behave, however it’s simply impossible to plan or schedule. When folks are paid in the office, you can reasonably expect that they will turn up regularly and can be assigned tasks. This model is invalid in open source.

I believe the large open source projects such as Apache, Clang and Mozilla employ engineers to undertake the work. I don’t know how they are funded. However, when pay-checks are offered, recruitment is possible in the market.

A modest project such as Exiv2 has no money. In fact, I pay for the modest expenses such as hosting the web-site and running a build server on a Mac Mini.

The major success I have had with recruitment is when Dan and Luis arrived in summer 2017. We adopted GitHub in April 2017 when Exiv2 v0.26 was released. I wondered if the move to GitHub had increased the project visibility and Dan and Luis would be the first of many contributors. 4 years later, we have enjoyed contributions from Alex, Kev, Leo, Miloš, Rosen and two Peters.

I find the data on OpenHub very interesting: https://www.openhub.net/p/exiv2.

You can spot trends and events in the history:

Period	Comment	Period	Comment	Period	Comment
2014‑2020	Growing	2013‑14	I retired and moved home	2018‑2019	Robin, Luis and Dan
2004‑2008	Andreas	2016‑2017	Robin and Ben	2019	I retired and contributors spiked
2009‑2012	Andreas and Robin	2017	Moved to GitHub	2020	I returned for Exiv2 v0.27.3
2012‑2015	Robin and Neils	2017	Dan and Luis Joined	2020	I wrote this book

The report provides interesting insight. Andreas was the top contributor until I overtook him in 2020. We don’t monitor the book or release scripts on OpenHub. My contributes to those parts of the project put me well ahead of Andreas.

A boss once asked me “Do you know the 80/20 rule? 80% of the project is done by 20% of the people!”. For sure, this is true in Open Source.

So, how are contributors recruited? The answer is I don’t know. For sure, I appreciate the work done by Andreas, Luis, Dan, Neils and about 20 other wonderful people. Curiously, I’m not aware of any lady contributors. I only recall one support question asked by a lady.

TOC

11.18 Scheduling

This is a major and important topic. Apart from writing code, I’ve spent more time thinking about project scheduling than any other aspect of Software Engineering.

There are two worlds. There is the perfect world inhabited by management. They live in a world which is quite different from mine. In their world, the specification is clear, the schedule is realistic, nobody makes mistakes and everything works. It’s a wonderful place. Sadly, I’ve never had the good fortune to live in that world.

I worked in a company which, to hide their identity, I will call “West Anchors”. A colleague was giving a board presentation in which they had a slide:

It is the Policy of West Anchors ® to get it right first time, every time.

There we have it. Nothing is ever late, nothing is more difficult than expected, all suppliers deliver on time. Every specification is faultless. All modules work together perfectly. Nobody is ever sick.

When I discussed the project schedule with my boss, I asked him why there was no time in the schedule for bugs and fixes. His response was “There better not be any”. Five years later, West Anchors were closed by their owners. Presumably the owners were bored by perfection.

I also had the misfortune to work at a company where the boss was an expert in planning. He explained to me that the only challenge in Software Engineering was getting the schedule right. Everything else was trivial.

So, if you live in the perfect world, you’ll not find anything interesting or useful in this part of the book, because I’m talking about the less than perfect world in which I live. I usually call it Reality.

Another challenge is that many users pretend to live in the mythical world where everything works. Those users are hostile to open-source projects populated by people who live in my depressing world of Reality.

Scheduling an open-source project is almost impossible. You are dealing with volunteers. You might think you know the volunteers, however you don’t. It’s unusual to have even met the people. How can you understand the pressure and stress in another person’s life when you know so little about their circumstances. And remember they are volunteers. They can walk off the job if they wish. In a business, management have tools such as reviews, salary, careers, vacations, bonuses, promotions and lay-offs to manipulate the employees. In open-source, you have none of those tools.

Here are my thought about how to solve the scheduling problem.

The Problem

The problem is really simple. How to plan large projects and deliver on time to budget.

The state of the game

Currently, planning is based mostly on PERT and descendant technology. Products such as Microsoft Project are designed to schedule resources and tasks. And indeed it works for some projects and fails hopelessly for others.

When the London 2012 Olympic Games were bid, the budget was $3 billion. The final cost has been stated as $20 billion. I have no data to say if there were other costs, such as policing, which are not included in the $20 billion.

This is rather common. The cost of construction of the aircraft carriers HMS Queen Elizabeth and HMS Prince of Wales are other high visibility projects in which the plan and reality are very different.

The reason for cost over-runs is that new work items are required that were not known early enough to be in the plan. We cannot know what we do not know. However there may be a way to calculate the size of the unknowns at the beginning of a project.

A project is recursive and requires recursive handling

When a project is simple - for example painting your house - it is possible to measure the size of the task and calculate the quantity of materials and labour required. This method works fine for a well defined project with quantifiable tasks.

However, if you want your house to be painted in an extra-ordinary way, this method totally fails. Think of Michael Angelo in the Sistine Chapel in Rome. Michael and the Pope came close to blows in a 20 year struggle to produce one of the wonders of man’s creativity. Nobody gives a hoot today about the cost. Nobody cares about how long it took. Nobody can understand why the customer and the contractor were divided over something as trivial as money.

The reason for the cost over-runs is because the project is recursive. In a simple project, you have a sequence (or connected graph) of tasks:

	Task	Task	Task
Begin	Remove furniture	Apply N litres of paint	Restore furniture	Done

If the project had many rooms (say 10-20), you have to schedule resources (people). You have a finite set of painters, and you may have more than 1 team of painters. However the basic linear model is not affected.

When you scale to painting something large (like an Aircraft Carrier), two items rapidly emerge to invalidate the simple model.

1) Requirements Change

The Aircraft Carrier requires stealth paint that hasn’t been invented.

2) The paint task is large

You require training and inspection services to manage quality.

And many other things arrive which were not in the simple model. In the worst case, new tasks can be larger than the original task. You have an exploding, complex challenge.

To deal with this, you have to start a project inside the project. Something like “Remove furniture” is obvious in a house. But what would that mean on an Aircraft Carrier?

So, we stay calm and add more items to the project plan. And that’s when and why everything goes wrong. The plan gets longer and more detailed. However it’s still the same old linear model.

My observation is that the project is an assembly of projects. As you develop the project, every line item in the simple model is a project. And then there are projects inside the projects. For example if special paint is required for the Aircraft Carrier, that task is probably a complex network of projects involving chemistry, special machines to apply the paint, and maintenance processes for the ship in service.

What does this have to do with Fractals?

Everything. A project is a recursive entity that must be handled with recursive techniques. Fractal Geometry deals with recursion.

Being a retired Software Engineer, I have often been told “The project is late because you (the engineer) did not itemise the project properly at the outset.”. Wrong. It’s the inflexible PERT model that cannot handle recursion.

The State of Project Planning Today

The software industry has a huge and sad collection of projects which have come off the rails. If the bosses had known at the outset, things would have been different. There are two things we care about passionately:

1. How long is this going to take?
2. How much is this going to cost?

Notice, we don’t get overly bothered about what “it” is. We care about time and money.

If we do not know about the special paint for the Aircraft Carrier, are we upset? No. However the cost and schedule damage is painful for everybody involved.

Now, we can’t know what we do not know. Is there are a way to calculate the size of the unknowns? There might be, as I will explain.

When you plan a project, you say “How long did it take to do the last one?”, take into account inflation and apply optimism “We won’t make the same mistakes again.”. This is very bad thinking. The United Kingdom has not built an Aircraft Carrier for almost 40 years. Most of the engineers working on HMS Queen Elizabeth were not born when HMS Invincible sailed to the Falklands.

A whole collection of project planning tools are now used in the software industry. Together they go under the banner: “Agile” or “Scrum”. Scrum imposes a regime of meetings and reviews on the project team. Several of these techniques are interesting.

Measure	Description	Observation
Story Points Task Size Task Poker	The size of a task is not 1,2,3,4,5 as difficulty increases. They use the Fibonacci series: 1,2,3,5,8,13,21 … So big tasks rapidly increase their allocation of resources and time.	Fibonacci series is recursive: X(n) = X(n-1) + X(n-2) where X(1),X(2)=1
The Sprint Step wise linear	Scrum says “we can’t plan everything at once, however we can complete well defined tasks on sprint schedules (typically 2 weeks).	I don’t know how scrum deals with tasks that are longer than 1 sprint.
Velocity	The team velocity (average story points completed over the last 3 sprints) are monitored and used to verify that the team is being neither optimistic nor pessimistic in their determination of story points for tasks.	Velocity is not predicted, it is measured by project performance. In a nutshell, it is recursive.

However scrum has a fatal weakness. Nobody knows the size of the total project. The model is fundamentally inadequate, because it is a monitoring tool and not predictive.

Can we have a single unified model for project planning?

I believe there’s a measure in Fractals called "Roughness” which measures some feature of the recursive item. If you measure the roughness of animal lungs (which are of course recursive), they are about the same in Elephants, Humans and Mice. A value of 1.0 implies that the object is perfectly smooth. Higher values represent the chaotic nature of the item.

I think it’s possible to measure roughness in past projects in addition to the historical performance. So, although we have never built HMS Queen Elizabeth, we could know from other Naval projects:

1. How much paint/painter/per time unit (the only measure in Microsoft Project)
2. The roughness of painting Naval Ships (projects hiding inside the project)

Both are required to estimate the size of the task. PERT models assume a roughness of 1.0 and that is why it fails on large projects. No large project has a roughness of 1.0.

So how can we use this?

We need to do three things:

1. Add roughness to every item in the project plan
2. Collect data to estimate roughness
3. We need a pot of time and money, which I call Contingencies

Contingencies are a % of the whole project that should be used to assign resources as sub-projects emerge. All items in the project should have contingencies from which additional resources can be allocated. This is non-confrontational and does not require blame and finger pointing. We knew about the roughness and must plan for it.

In the past, I have applied contingencies as big brush stokes to the complete project. If the project is similar to the last one, contingencies are 10%. If the project involves many unknowns, perhaps it is 300% of the project.

An ex-boss thought 414.159% Factor 3.14159 to walk round the object, then 1.0 for the task itself! The point is that when you do something for the first time, you will spend time doing work that is subsequently abandoned. Nothing new can be achieved without trial and error.

Research is required to measure task roughness in past projects to validate this approach.

Other serious limitations with PERT

There are serious limitations with PERT. I only intend to investigate the use of fractals in planning and to ignore other limitations of PERT such as:

Trouble	Observation
PERT assumes that you can itemise and quantify every task in the project.	If you are investigating something new, you can probably do neither of these things.
Many projects cannot be quantified.	Why isn’t PERT used: 1. In crime investigation. 2. In medical treatment. 3. In investment management.
You will do abortive work and encounter road blocks.	Program Managers never plan time for this. Every innovative project has abortive work.
People are not interchangeable.	People leave, or are assigned to other projects. New team members require time to come up to speed with the project.
Some tasks have a gestation period.	If you are having a baby, more women won’t reduce the 9 month wait. Adding people is often counter-productive.
Management, and other project stakeholders, change goals and objectives.	The circumstances surrounding the project can change and have major implications for the project.

Because of the recursive nature of projects, there are serious limitations hiding inside these limitations.

So what am I going to do about this?

When I retired, I was thinking about doing a PhD in this area and thought it might take 10,000 hours over 5 years. The only tasks that I could define in 2014 were:

1. Write an outline of the project
2. Find a University willing to mentor/supervise the effort
3. Learn all about Fractals
4. Research and publish a thesis
5. Graduate

What is the roughness of these tasks? Unknown. Graduate is simple. Or is it? Do I need a new kilt? Who’s going to attend? Where will everybody stay? Even little tasks can grow into projects.

One thing is certain, getting a better approach to project estimation is of enormous importance. We have to do better. I have tried to set out here an area of investigation that is worthy of attention.

Final words about this. I didn’t undertake a PhD. Instead I have spent 10,000 hours working on Exiv2. This book is my thesis. The presentation at LGM in Rennes would have been my defence. My reward is to know that I’ve done my best.

TOC

11.19 Enhancements

I’m not sure there is anything very interesting to be said about this. There are really different types of requests. For example, adding recognition for one lens may only require one line of C++, a test image and a 10-line python test script. This is straightforward and can be fixed within hours. At the other extreme is the request to support BMFF files. This project involves research, code, test, build and documentation changes. And to make it even more difficult, the Community challenged the legality of providing the feature. This feature took one year to complete. Probably 500 hours were spend on legal discussion.

In principle, anybody can develop a feature and submit a PR. In reality, this seldom happens. When this does happen, the effort required by me and the developer is often about the same. So, code offered in a PR often doubles my work-load.

TOC

11.20 Tools

Every year brings new tools. For example: cmake, git, MarkDown, Conan and C++20. One of the remarkable properties of tools you have never used is that they are perfect and solve all known issues, until you use them. Tools you have never used are bug free and perfect. Or so I am told.

I had an issue with the release bundles for Exiv2 v0.26. My primary development platform is macOS. Remarkably, the version of tar shipped by Apple puts hidden files in bundles to store file extended attributes. I didn’t know this until the bundles shipped and a bug report appeared. You cannot see those files on macOS, because tar on macOS recreates the extended attributes. However there were thousands of hidden files in the source bundle on Linux. I recreated the bundles as Exiv2 v0.27a and shipped them. There is an environment variable to suppress this. I believe it is: TAR_WRITER_OPTIONS=–no-mac-metadata.

Case closed. Except for very critical emails about changing bundle checksums.

For v0.27 we adopted CMake to do the packaging. Very nice. Works well. Guess what? CMake produces .tar.gz files which have these hidden files. Several people emailed to say “You wouldn’t have this problem if you used CPack.”. 100% wrong. It is a known documented issue in CPack. So, the issue resurfaced because we used CPack. Additionally, we had three release candidates for v0.27 which were published on 27 October, 15 November and 7 December 2018. v0.27 shipped on 20 December and the bug report arrived on the day after Christmas Day.

I rebuilt the bundles as Exiv2 v0.27.0a and shipped them on 2 January 2019. I updated the build script to ensure that source bundles are created on Linux.

Please understand that I have nothing against using new tools. However most of the hype surrounding new tools is nonsense. This has been studied. There are 5 stages in adopting new tools.

There is one recent tool which has surprised and pleased me. I have written this book using markdown and very pleased with the experience. As Americans say “your mileage may differ!”.

TOC

11.21 Licensing and Legal

Licensing is a legal minefield. Exiv2 is licensed under GPLv2. Until Exiv2 v0.26, Andreas offered a commercial license for Exiv2. The contract between Andreas and users is not the concern of the Exiv2 open-source project.

In the days of the Commercial license, I made no distinction between open-source and commercial license users when it came to dealing with support and other requests. I felt that the commercial license freed the user from the obligations of GPL. However, it did not provide priority support, enhancement requests or any other benefit.

The general subject of the legality of Exiv2 hasn’t been explored. There has been an enormous discussion about the legality of reading BMFF files. See https://github.com/Exiv2/exiv2/issues/1229.

The BMFF legal issue has caused me to wonder if Exiv2 is legal. I also wonder if any open source is legal! What makes something legal or illegal? Is everything legal until there is a law which declares it as illegal, or everything illegal until permitted by legislation? I suspect everything is legal until there is a legal precedent such as legislation or a court ruling to the contrary.

Dealing with legal matters is not like reporting a bug. Exiv2 is an open-source project and we get a regular stream of issues reported on https://github.com/exiv2/exiv2. I acknowledge, investigate, reply and close the issue. By design, the process is focused on resolution. Legal processes are very different. When you ask for legal advice, you are instigating an open-ended process which will expand endlessly.

Closely related to licensing is patents. There is a patent non-agression group called OIN which is the Open Invention Network. Members agree to give toll free use of their patents to other members. They have an impressive list of members include Facebook, Google, SAP, Microsoft, IBM and Canon. They also maintain the Linux System Definition. https://openinventionnetwork.com

It makes sense for Exiv2 to belong to OIN and be included in the Linux System Definition. The process to join is to fill in a form and sign it. And that’s a problem. What is the legal status of Exiv2? I don’t know. When you have a child, you register the birth and the child is recognised in law as a legal entity. If you register a company, it is a legal entity with a board of directors. Board members can sign documents on behalf of the company. However, open source is neither of these things. It’s a collection of source on a server. I think it belongs to the authors. According to openhub.net there are 85 contributors. Presumably I need the consent of all 85 contributors. Even if I could contact all 85, it’s possible that somebody would withhold their permission to join OIN.

There is a reason to believe its not possible to get everybody to agree that Exiv2 should join OIN. Andreas used to offer a commercial license for Exiv2. In return for a modest fee, companies could ship products using the library and be free of the obligations of GPLv2. I had the idea of asking Google to purchase the commercial rights from Andreas. At least one team member objected because they didn’t want anybody to “own” their work. With some thought and discussion in my dining room, the team reached the conclusion that the Exiv2 Open Source Project was under no obligation concerning the Commercial License and a “Position Paper” was written. I had the good fortune on vacation to meet a Legal Person who specialized in Intellectual Property. I sent him the “Position Paper” and he concurred with our reasoning. So, we were able to release the project from the Commercial License with no consequence for Team Exiv2.

I was surprised by the Team’s resistance to the idea of asking Google to buy the commercial rights. My conclusion is that unless you have the contributor’s support, you should assume that you do not have their support. I don’t intend to sign the OIN License. I give permission for a future maintainer to sign the OIN License on my behalf.

TOC

11.22 Back-porting

I believe there are some folks maintaining back-ports of Exiv2. Our friend Pascal works on darktable and has back-ported many features and fixes. Thank You, Pascal for undertaking that chore.

I have to say that the inertia of the Linux Distros is considerable. It can take several years for new releases to arrive on the platform. I don’t know anything about the distros and I’m not going to judge why it is so sluggish.

TOC

11.23 Partners

Without question, dealing with some other projects which use Exiv2 has been very difficult. Folks who have adopted Exiv2 in their product may feel they are entitled to make enhancement requests, demand fixes, superior support and other privileges. In a nutshell, they feel entitled. They are not. They are entitled to the same as all other stakeholders. No more. No less.

TOC

11.24 Development

As this is the first and last book I will ever write, I’d like to close the discussion of Project Management with some thoughts and opinions about how software is developed. Management have been searching for the silver bullet that will cause projects to deliver on time, to budget, with great performance, few bugs and low cost maintenance. This search has been proceeding for more than 50 years. We’ve made some progress. However system complexity out-strips our management and control tools. The challenges are immense.

I’ve seen different approaches used. In the IT world, people involved in system development adopted and modified the drawing office model. In the drawing office, you have draftsmen working on drawing boards and engineers working at desks. The engineers do the design and the draftsmen draw it. The systems analyst was the designer and the programmers created the code. They work in a strict regime of SSADM - the Standard Structure Analysis and Design Methodology. This is often called “The Waterfall Method”. It’s horrible. It’s inflexible, slow and very expensive. It’s amazing that anything can be delivered this way.

When I worked at West Anchors, the analysts promoted all the programmers to programmer/analyst. So the programmer had to do the programming and the work of the analyst. This enabled the analyst to concentrate on office politics. The QE team at West Anchors didn’t test anything. They approved the test plans written by the programmer/analyst and they inspected the test logs required to prove that the programmer/analyst had done all the work. The parrot phrase of everybody who wasn’t a programmer/analyst was “I’m not technical” which meant “I’m not going to help you, so don’t ask. And, by the way, I am superior to you and you will do exactly what I tell you to do.”.

Before I retired, the circus started adopting Scrum. Loads of meetings. The project is divided into two-week sprints. There were two days of review meetings at the end of every sprint. Two days of planning meetings at the start of every sprint. Daily stand-up meetings which were usually about 1 hour. And I’m sure I’ve forgotten other pointless meetings. Sometimes people say they are agile. I haven’t figured out what that is. I think it’s some kind of “Let’s not bother looking ahead. It’ll be great if or when it’s delivered.”. And of course, all software development engineers (except me) are geniuses who create perfect code and therefore have no reason to document or help lesser co-workers.

In the last 10 years we have seen AI move out of the lab and into our homes, cars and phones. Probably 50% of code development time is spent on test related activity. Perhaps in future, AI will undertake more of that work. Remember it works 7x24, never takes a vacation and works very quickly. I have high hopes that AI can be used to automate testing in future. However, all coins have two sides and the AI may drown the engineer with very obscure bugs. In some way, we see this with CVEs discovered by automatic fuzzing libraries.

There is a method of developing code that works for me and that’s to do everything myself. This model doesn’t scale. However it is effective. Do I create bugs? Of course, I do. However I find and fix them. Many of the best people with whom I worked in Silicon Valley use this approach. And when I think about it, that’s exactly how Andreas created Exiv2.

Another method that I believe is very effective is prototyping. Working in a sand-box with a small amount of code can be very effective to explore and learn. I can say with certainty that I have learned more about metadata in 12 weeks by writing this book than I discovered by working on the Exiv2 code for 12 years. Program Management people hate prototyping because it doesn’t have a specification, milestones, deliverables or schedule.

If you have good folks on the team, the development will be enjoyable and the results will be good. However, Software Development in large chaotic company such as West Anchors is Russian Roulette with a bullet in every chamber. Good Luck. I’m happy to be retired.

TOC

12 Code discussed in this book

The latest version of this book and the programs discussed are available for download from:

svn://dev.exiv2.org/svn/team/book

To download and build these programs:

$ svn export svn://dev.exiv2.org/svn/team/book
$ mkdir book/build
$ cd book/build
$ cmake ..
$ make

I strongly encourage you to download, build and install Exiv2. The current (and all earlier releases) are available from: https://exiv2.org.

There is substantial documentation provided with the Exiv2 project. This book does not duplicate the project documentation, but compliments it by explaining how and why the code works.

TOC

tvisitor.cpp

The syntax for tvisitor is:

$ tvisitor
usage: tvisitor -{ U | S | R | X | C | I }+ path+
$

The options are:

Option	Exiv2 Equivalent	Meaning	Description
U	–unknown	Unknown	Reports unknown tags
S	-pS	Structure	This is the default
R	-pR	Recursive	Descend as deeply as possible
X	-pX	XMP	Report XMP
C	-pC	Icc	Report ICC Profiles
I	-pi	Iptc	Report IPTC Data

The options can be in any order and undefined characters are ignored. The most common option that I use is -pR which is equivalent to exiv2’s -pR. It could be simply stated as tvisitor -R foo. In the test harness, I use -pRU.

The option U prints Unknown (and known) tags. An unknown tag is an item of metadata for which tvisitor does not know the name. A tiff ‘tag’ is identified by a 16 bit integer and these are defined in the TIFF-EP and Exif Specifications. The MakeNote tags are not standardised and unknown tags are reported in the following style:

   address |    tag                    |      type |    count |    offset | value
       382 | 0x003b Exif.Nikon.0x3b    |  RATIONAL |        4 |      1519 | 256/256 256/256 256/256 256/256

The option S prints the Structure of the file. For example, use the following file: https://clanmills.com/Stonehenge.jpg, we see the structure of the JPEG file:

$ tvisitor -pS ~/Stonehenge.jpg 
STRUCTURE OF JPEG FILE (II): /Users/rmills/Stonehenge.jpg
 address | marker       |  length | signature
       0 | 0xffd8 SOI  
       2 | 0xffe1 APP1  |   15288 | Exif__II*_.___._..._.___.___..._.___.___
   15292 | 0xffe1 APP1  |    2610 | http://ns.adobe.com/xap/1.0/_<?xpacket b
   17904 | 0xffed APP13 |      96 | Photoshop 3.0_8BIM.._____'..__._...Z_..%
   18002 | 0xffe2 APP2  |    4094 | MPF_II*_.___.__.._.___0100..._.___.___..
   22098 | 0xffdb DQT   |     132 | _.......................................
   22232 | 0xffc0 SOF0  |      17 | ....p..!_........
   22251 | 0xffc4 DHT   |     418 | __........________............_.........
   22671 | 0xffda SOS   |      12 | .._...._?_..
END: /Users/rmills/Stonehenge.jpg
$ 

The option R performs a Recursive descent of the file and dumps embedded structures such as the TIFF which contains the Exif Metadata. It also descends into IPTC data and ICC Profiles.

$ tvisitor -pR ~/Stonehenge.jpg 
STRUCTURE OF JPEG FILE (II): /Users/rmills/Stonehenge.jpg
 address | marker       |  length | signature
       0 | 0xffd8 SOI  
       2 | 0xffe1 APP1  |   15288 | Exif__II*_.___._..._.___.___..._.___.___
  STRUCTURE OF TIFF FILE (II): /Users/rmills/Stonehenge.jpg:12->15280
   address |    tag                              |      type |    count |    offset | value
        10 | 0x010f Exif.Image.Make              |     ASCII |       18 |       146 | NIKON CORPORATION
        22 | 0x0110 Exif.Image.Model             |     ASCII |       12 |       164 | NIKON D5300
  ....
  END: /Users/rmills/Stonehenge.jpg:12->15280
   15292 | 0xffe1 APP1  |    2610 | http://ns.adobe.com/xap/1.0/_<?xpacket b
   17904 | 0xffed APP13 |      96 | Photoshop 3.0_8BIM.._____'..__._...Z_..%
  STRUCTURE OF 8BIM FILE (MM): /Users/rmills/Stonehenge.jpg:17922->78
       offset |   kind | tagName                      |  len | data | 
            0 | 0x0404 | PSD.8BIM.IPTCNAA             |   30 | 12+1 | ..__._...Z_..%G..__._...x_.___
    STRUCTURE OF IPTC FILE (MM): /Users/rmills/Stonehenge.jpg:17922->78:12->39
        Record | DataSet | Name                           | Length | Data
             1 |       0 | Iptc.Envelope.ModelVersion     |      2 | _.
             1 |      90 | Iptc.Envelope.CharacterSet     |      3 | .%G
             2 |       0 | Iptc.Application.RecordVersion |      2 | _.
             2 |     120 | Iptc.Application.Caption       |     12 | Classic View
    END: /Users/rmills/Stonehenge.jpg:17922->78:12->39
  END: /Users/rmills/Stonehenge.jpg:17922->78
   18002 | 0xffe2 APP2  |    4094 | MPF_II*_.___.__.._.___0100..._.___.___..
   22098 | 0xffdb DQT   |     132 | _.......................................
   22232 | 0xffc0 SOF0  |      17 | ....p..!_........
   22251 | 0xffc4 DHT   |     418 | __........________............_.........
   22671 | 0xffda SOS   |      12 | .._...._?_..
END: /Users/rmills/Stonehenge.jpg
$

There is no plan to have a man page for tvisitor because it has a 200 page book! tvisitor isn’t intended for any production use and has been written to explain how Exiv2 works. In less than 4000 lines of code it decodes the metadata in all formats supported by Exiv2 plus BMFF formats .CR3, .HEIC and .AVIF. Additionally, it supports BigTiff, extended JPEG, dumping ICC profiles and many other features which are not supported in Exiv2.

tvisitor is currently being tested using more than 10,000 images harvested from ExifTool, raw.Pixls.us, RawSamples.ch and images collected from issues reported to Exiv2. My aim is to successfully read 9990 which is 99.9% reliability. I fully expect the Community to attack me concerning the 0.1% that are not successfully decoded. On-line abuse from the Community is the reason that I am retiring.

I have written the book for two purposes:

1. To Explain how Exiv2 works.
2. To Explain how to parse metadata.

Exiv2 provides a unique capability to the Community and its long term maintenance is of importance to Linux. To my knowledge, no book as been written about Metadata. The tvisitor code would provide a good resource from which to develop a new Metadata Library.

TOC

dmpf.cpp

The purpose of this program is to inspect files. It’s od on steroids combined with parts of dd.

$ dmpf -h
usage: ./dmpf [-]+[key=value]+ path+
options: bs=1 count=0 dryrun=0 endian=0 hex=1 skip=0 start=0 verbose=0 width=32
$ 

There are numerous examples of this utility in the book. The options are:

Option	Description	Default	Comment
bs=	block size	1	1,2 or 4
count=	number of bytes to dump	rest of file	Accumulative
dryrun=	report options and quite	0
endian=	endian 0=little, 1=big	0	toggles native endian
hex=	0=int or 1=hex	0
skip=	number of bytes to skip	0	Accumulative
start=	number of bytes to start	0	Set by path->start:length+
verbose=	echo the settings	0
width=	width of output	32

path can be - or path or “path:offset->length”

The term Accumulative means you may use the option more that once and they will be added.
Non Accumulative settings are set when encountered. So, the last setting will prevail.

Here are some examples, using files/Stonehenge.jpg for which the structure is:

$ tvisitor -pS files/Stonehenge.jpg
STRUCTURE OF JPEG FILE (II): files/Stonehenge.jpg
 address | marker       |  length | signature
       0 | 0xffd8 SOI  
       2 | 0xffe1 APP1  |   15272 | Exif__II*_.___._..._.___.___..._.___.___
   15276 | 0xffe1 APP1  |    2786 | http://ns.adobe.com/xap/1.0/_<?xpacket b
   18064 | 0xffed APP13 |      96 | Photoshop 3.0_8BIM.._____'..__._...Z_..%
   18162 | 0xffe2 APP2  |    4094 | MPF_II*_.___.__.._.___0100..._.___.___..
   22258 | 0xffdb DQT   |     132 | _.......................................
   22392 | 0xffc0 SOF0  |      17 | ....p..!_........
   22411 | 0xffc4 DHT   |     418 | __........________............_.........
   22831 | 0xffda SOS   |      12 | .._...._?_..
END: files/Stonehenge.jpg
$ 

Purpose	Command	Output
Dump first 16 bytes	$ dmpf count=16 files/Stonehenge.jpg	0 0: ....;.Exif__II*_ -> ff d8 ff e1 3b a8 45 78 69 66 00 00 49 49 2a 00
Dump 12 bytes of Exif data	$ dmpf count=12 skip=2 skip=4 skip=6 files/Stonehenge.jpg	0xc 12: II*_.___._.. -> 49 49 2a 00 08 00 00 00 0b 00 0f 01
Dump 12 bytes of Exif data	$ dmpf ‘files/Stonehenge.jpg:12‑>24’	0xc 12: II*_.___._.. -> 49 49 2a 00 08 00 00 00 0b 00 0f 01

TOC

csv.cpp

The purpose of this program is to “pretty print” csv files. The only use of this program is to “pretty-print” csv output from taglist for presentation in this book. For example:

$ taglist ALL | csv - | head -3
[Image.ProcessingSoftware]  [11]    [0x000b]    [Image] [Exif.Image.ProcessingSoftware] ...
[Image.NewSubfileType]  [254]   [0x00fe]    [Image] [Exif.Image.NewSubfileType] [Long]  ...
[Image.SubfileType] [255]   [0x00ff]    [Image] [Exif.Image.SubfileType]    [Short] ...
$ 

TOC

CMakeLists.txt

Here is the CMakeList.txt for the code that accompanies the book. It’s similar and simpler version of the cmake code in Exiv2.

cmake_minimum_required(VERSION 3.8)
project(book VERSION 0.0.1 LANGUAGES CXX)
include(CheckCXXCompilerFlag)

set(CMAKE_CXX_STANDARD                11 )
set(CMAKE_CXX_EXTENSIONS              ON )
set(CMAKE_OSX_ARCHITECTURES     "x86_64" )
set(CMAKE_XCODE_ARCHS           "x86_64" )

# build for the current version of macOS
if ( APPLE ) 
    execute_process(COMMAND sw_vers -productVersion
                    OUTPUT_STRIP_TRAILING_WHITESPACE
                    OUTPUT_VARIABLE SW_VERS
                   )
    set(CMAKE_OSX_DEPLOYMENT_TARGET "${SW_VERS}")
endif()

# don't build for 32bit (size_t/sprintf unhappiness)
if("${CMAKE_SIZEOF_VOID_P}" STREQUAL "4")
    message(FATAL_ERROR "32 bit build is not supported")  
    error()
endif()

# the programs
add_executable(visitor     visitor.cpp    )
add_executable(tvisitor    tvisitor.cpp   )
add_executable(dmpf        dmpf.cpp       )
add_executable(csv         csv.cpp        )
add_executable(parse       parse.cpp      )

# options
option(    EXIV2_TEAM_USE_SANITIZERS  "Enable ASAN when available"                  OFF )
if ( MSVC ) 
   option( EXIV2_ENABLE_PNG        "Build compressed/png support (requires libz)"   OFF )
else()
   option( EXIV2_ENABLE_PNG        "Build compressed/png support (requires libz)"   ON  )
endif()

if( EXIV2_ENABLE_PNG )
    find_package( ZLIB REQUIRED )
endif( )
if ( EXIV2_ENABLE_PNG AND ZLIB_FOUND ) 
    target_link_libraries ( tvisitor PRIVATE ${ZLIB_LIBRARIES} )
    target_compile_options( tvisitor PUBLIC -DHAVE_LIBZ)
endif()

if(WIN32)
    find_library(WSOCK32_LIBRARY   wsock32)
    find_library(WS2_32_LIBRARY    ws2_32)
    target_link_libraries(parse    wsock32 ws2_32)
    target_link_libraries(tvisitor wsock32 ws2_32)
endif()

# ASAN (not on Windows)
if ( EXIV2_TEAM_USE_SANITIZERS AND NOT (CYGWIN OR MINGW OR MSYS OR MSVC) ) 
    check_cxx_compiler_flag(                                  -fno-omit-frame-pointer       HAS_NO_EMIT)
    if(HAS_NO_EMIT)
        add_compile_options(                                  -fno-omit-frame-pointer )
        set(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} -fno-omit-frame-pointer")
    endif()
    check_cxx_compiler_flag(                                  -fsanitize=address,undefined  HAS_FSAU)
    if(HAS_FSAU)
        add_compile_options(                                  -fsanitize=address,undefined )
        set(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} -fsanitize=address,undefined")
    endif()
    check_cxx_compiler_flag(                                  -fno-sanitize-recover=all     HAS_FSRA)
    if(HAS_FSRA)
        add_compile_options(                                  -fno-sanitize-recover=all )
        set(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} -fno-sanitize-recover=all")
    endif()
endif()

# Test harness (in ../test)
add_custom_target(test  COMMAND ../test/run.sh )
add_custom_target(tests COMMAND ../test/run.sh )

# This is intentionally commented off
# See Chapter 8 Testing for discussion about building libtiff
if ( 0 ) 
    include_directories(/usr/local/include)
    link_directories(/usr/local/lib)
    add_executable(create_tiff create_tiff.cpp)
    target_link_libraries(create_tiff z tiff jpeg)
endif()

# That's all Folks!
##

TOC

make test

The code in the book has a simple test harness in test/run.sh. When you build, you can run the tests with the command:

586 rmills@rmillsmbp:~/gnu/exiv2/team/book/build $ make tests
Scanning dependencies of target tests
20200717_221452.avif passed
args passed
avi.avi passed
avif.avif passed
Canon.cr2 passed
Canon.crw passed
Canon.jpg passed
cr3.cr3 passed
csv passed
dmpf passed
heic.heic passed
IMG1.HEIC passed
IMG_3578.HEIC passed
mrw.mrw passed
NEF.NEF passed
NikonD5300.dcp passed
ORF.ORF passed
Stonehenge.jpg passed
Stonehenge.tiff passed
webp.webp passed
-------------------
Passed 20 Failed 0
-------------------
Built target tests
587 rmills@rmillsmbp:~/gnu/exiv2/team/book/build $ 

The code to implement the tests is in test/run.sh

#!/usr/bin/env bash

pass=0
fail=0

# Create reference and tmp directories
if [ ! -e ../test/data ]; then mkdir ../test/data ; fi
if [ ! -e ../test/tmp  ]; then mkdir ../test/tmp  ; fi

report()
{
    stub=$1
    # if there's no reference file, create one
    # (make it easy to add tests or delete and rewrite all reference files)
    if [ ! -e "../test/data/$stub" ]; then
        cp "../test/tmp/$stub" ../test/data
    fi
    
    diff -q "../test/tmp/$stub" "../test/data/$stub" >/dev/null 
    if [ "$?" == "0" ]; then
        echo "$stub passed";
        pass=$((pass+1))
    else
        echo "$stub failed"
        fail=$((fail+1))
    fi
}

# test every file in ../files
for i in $( ls ../files/* | sort --ignore-case ) ; do
    stub=$(basename $i)
    # dmpf and csv are utility tests
    if [ $stub == dmpf -o $stub == csv -o ]; then
        ./$stub ../files/$stub 2>&1 > "../test/tmp/$stub"
    else 
        ./tvisitor -pRU "$i"   2&>1 > "../test/tmp/$stub"
    fi
    report $stub
done

echo -------------------
echo Passed $pass Failed $fail
echo -------------------

# That's all Folks
##

TOC

The Last Word

I hope you found this book interesting. More to the point, I hope you found the book useful. I hope Exiv2 will live into the future, or this book inspires somebody to write a new library.

I’m going off to cut the grass and to run in the beautiful countryside around my home in Camberley, England. And I’m going to play the Euphonium and the Piano.

TOC