Cost savings in The Netherlands: Now you see it, now you don't

The Open Source Observatory flashed an eye-popping headline last week: “Moving to open source would save [The Netherlands] government one to four billion [euro].”

I had hoped I could do the dirty work of going over the report in fine detail and give you the summary, but there are two problems: first, it’s only in Dutch (I guess the actual problem is I can’t read Dutch), and second, the government took it down.

In a matter of days, the Court of Audit surfaced another report on the same general topic of cost savings for open source. This one has a wildly different conclusion:

“We concluded...the potential savings the government could make by making more use of open source software were limited.”

Interestingly, a headline from the Court of Audit’s own news excerpt is: “Ministries already use a lot of open source.”

This report, too, doesn’t seem to be available, except for a summary in English.

Headlines from outfits covering the story have read “Netherlands open source report says no savings can be made.” Discussion boards have chimed in stoking a conspiracy to cook numbers, government succumbing to Microsoft lobbying, and so on.

My take: Much ado about nothing.

First, I think we can agree The Netherlands has an issue with transparency, at least when it comes to handling this issue.  But, beyond that I doubt there is any behind-the-scenes-lurking.

When the Open Source Observatory wrote about the first report, they noted it was written by one person in the Ministry of Interior, and largely meant for an internal audience. Only after pressure from representatives in Parliament did the Ministry send over the report.

Report, here, may even be the wrong word. Governments do a lot of internal prospecting via memos, briefs, etc. just like any other organization. Similar to the issues I brought up about the relationship between the US Congress and its Congressional Research Service, government needs some room to think on its own. Not every document amounts to an answer that should be translated into policy.

It could be that in this case the Parliament caught wind of someone’s quick calculations and wanted to see the analysis without any word on how “official” the document was, the background of the writer, or the vetting process, if any, the report was put through. In all reality, it may be a “bad” report.

But, still, let’s say hypothetically its numbers and methodology are sound. What do we make of the Court of Audit report?

A separate Audit organization is a common function of democratic governments. They can perform audits as we typically think, in the financial sense, but they also do performance or program audits.

Auditors work a lot like consultants having to take on projects quickly and turn them around just as quickly. They’re also outside the organization, which can be good for impartiality (both conscious and unconscious) but bad for data collection and organizational knowledge. Although they often work on teams around related topics to gain some domain expertise, the economics just don’t work out to keep an expert in every area on hand.

Even smart generalist evaluators stumble when it comes to something as arcane as open standards and open source in procurement. It’s likely this report was generated by people working on a tight timeline without much experience in the area or even inside-knowledge of the Ministries on which they were focusing.

This is where we get this decidedly cautious and wishy-washy writing from the report’s summary:

“Regarding the replacement potential, we concluded that it could not be said in advance what part of a ministries' software could be made 'open'.”

In addition to:

 “Whether certain advantages and disadvantages, opportunities and risks are applicable in a specific situation can only be determined by studying the conditions in that situation and through specific market research of the available software products and services.”

Because this appeared in the conclusion section, this to me translates as: ‘We didn’t actually calculate anything.’

Even though I said much ado about nothing, every time a report like this comes out, (or in the case of the first Netherlands' one, doesn’t come out) it is still important politically, even if the findings are not. MPs, senators, representatives, whatever you have, love the ease of pointing to simple headlines: savings, no savings.

Which is why it’s important to speak the language of the auditors and program evaluators. Doing so begins with a cost-benefit framework: guidelines that point out potential benefits, costs, and conventions for how to calculate them.

Good reports start out with a general review of the literature, who’s already done and found what. It’s imperative to cite and work from, or justify the departure from, existing frameworks.

That’s what we’re missing. There are frameworks for cost-benefit analysis of environmental issues, juvenile justice programs and so on, but we revert to ROI or TCO when it comes to software. Government procurement is and should be discussed in terms much bigger than cost savings.

The way to improve reports like these in the future is not necessarily to accuse of wrong-doing or file comments after the fact, but ensure that the people writing them have a helpful resource from which to work.

I’ll stop throwing ideas out and actually do some work next post, assessing the current state, while hopefully furthering the field a bit.