Digital preservation basics in four online seminars

If you are new to digital preservation, you may want to check out four ‘webinars’ organized by the California State Library and the California Preservation Program. The one-hour webinars promise to give you a basic understanding of what digital preservation is all about, of interest especially to librarians and archivists who are involved in developing digital projects.

The first webinar is scheduled for December 8, 12 PM Pacific time (which is 21.00 hrs in Holland). Topics include: ‘storing digital objects, choosing and understanding risks in file formats, planning for migration and emulation, and the roles of metadata in digital preservation.’ See http://infopeople.org/training/digital-preservation-fundamentals.

Disaster planning and enabling smaller institutions (iPRES2011, 7)

As this iPRES was moved from Tsukuba, Japan, to Singapore because of the earthquake and tsunami in Japan in March this year, it was only fitting that iPRES2011 should include a panel session on disaster planning. Neil Grindley (JISC) asked if digital preservation does not implicitly include disaster planning, but Angela Dappart (DPC) argued that with an entire infrastructure going down, the problems will be massively larger. Plus, as Arif Shaon (STFC) observed, ‘Grade A preservation should include it, but we have not reached that stage yet.’

Shigeo Sugimoto of Tsukuba, who would have been iPRES’s host in Japan, took a forward-looking view at disaster planning. Many physical artefacts were lost during the earthquake, and having lots of digital copies at different locations can certainly help rescue cultural heritage, provided the metadata are kept at different locations as well.

Shigeo Sugimoto (right) with José Barateiro of Portugal during the disaster planning session.

There is one catch, though: many smaller institutions do not have the means (money, staff) to build digital archives. Therefore, in Japan the idea has been put forward to design a robust and easy-to-use cloud-based service for small institutions:

In the Netherlands, I am involved in two Dutch Digital Preservation Coalition (NCDD) working groups who are looking at the same problems: how to enable smaller institutions to preserve their digital objects. Professor Sugimoto and I have agreed to stay in touch and exchange information and experiences.

On governance, trust and certification (iPRES 3)

I am going to blog about this parallel session in reverse order, because that way it makes more sense to me.

At the end of the session (but at the beginning of this post!) Devan Ray Donaldson of the University of Michigan reminded us what ‘trust’ (as in Trustworthy Digital Repositories, or TDR’s) is all about: end users (those that have had no involvement in either production or archiving of a document) need some assurance that the document they are getting from an archive is, in fact, authentic, that it is what it is supposed to be, and has not been tampered with or altered in any way. [BTW: that does not mean that the archive guarantees that the information in the document is reliable. The archive does not know that. The only thing an archive can do is assure that what the end user gets is the same thing that originally came into the archive.]

Archives know that end users care about trust, about authenticity. So Donaldson wants to study how we communicate with the end user about that authenticity. If we put some seal of approval on a document, will the end user trust it more than if we do not put any seal on it? That is an interesting question. Donaldson intends to use HathiTrust documents to test this, and, to me, that is the only ‘flaw’ in his plan – if such is the word, that is. HathiTrust contains digitized book pages, and that type of document is a lot easier to trust and be regarded as “authentic” than, e.g., e-mail. Donaldson agreed, but, as he said: you’ve got to start somewhere.

Next (in whichever order) came Olivier Rouchon of CINES, a large data centre in France (photo right, “Cannot I even have  lunch without being photographed?” J). CINES finds itself in a strange political situation: as an organization  CINES has a remit for only four years, but  it also has the express mandate to do long-term preservation and its clients ask for 30-year guarantees. This is a strange dichotomy and CINES has decided to seek certification as a trusted repository to a) lock the mission, and b) attract larger volumes of data to be preserved.

CINES went through various (self-) audits to attain ever higher levels of certification. That took a lot of work. Rouchon estimates that 1 fte of his 11 fte’s is constantly busy with audits. But, says Rouchon, ‘that should not stop you from doing it.’ First of all, it is mostly a lot of work the first time around. Once you have a good system in place, the next audits become business as usual. Secondly, CINES is using the audit system as an internal quality assessment instrument to keep improving the quality of the service. By comparing the outcome of audits over time the organization can measure its progress.

The EU is now building a three-tiered certification system: the first level is the relatively lightweight Data Seal of Approval, then comes a self-audit, and the highest level of certification is awarded by an external audit. The APARSEN project recently did a number of test audits, a.o. at CINES, and will publish the results shortly.

Steve Knight from the National Library of New Zealand enquired how we know that we can trust the auditors doing the auditing. Rouchon trusts his own (internal) auditors and part of the aim of the APARSEN test audits was to train auditors.

Having talked about trust, and about auditing trust, I now come to the last (first) presentation. Basically, it was about building all the capabilities you need to assure trust and prove trustworthiness into your system. It was also about not dealing with digital preservation as an issue (and a system!) that stands apart from the rest of your organization, but to build an information system for your organization that integrates digital preservation requirements, make them ‘ubiquitous’. Christoph Becker of TU Wien(photo right) told his audience that we have lots of models and concepts and frameworks (OAIS, TRAC, RAC, Drambora, Platter, etc. etc.), but ‘we still lack a holistic view.’ His team takes its cue from frameworks from the IT industry, such as ‘enterprise architecture’, and COBIT (goal-oriented, process-oriented, control-based) to build a Maturity Model based on CMM – you measure your maturity by a set of criteria to identify places for improvement … and then I lost the story. My mind tends to switch off when the discussion becomes abstract and high-level. It is a flaw, I know, but one I have to learn to live with. The basic idea, however, integrating digital preservation, is a good one, and so is using existing industry frameworks, so for those of you who are better at high-level discussions, do check out Becker’s paper in the proceedings which come online soon. The paper is called “A Capability Model for Digital Preservation: Analysing Concerns, Drivers, Constraints, Capabilities and Maturities”.

 Parallel session ‘Governance’

Emulation is becoming a viable option (#KEEP) (1)

Just about everybody in our community has heard of emulation as an alternative preservation strategy to migration. But emulation is mostly talked about -- and in terms that are not very encouraging: emulation is rumoured to be very difficult to do, to require lots of expert knowledge and to be really expensive. On 26-27 October the road show of the European KEEP project came to the KB in The Hague and some 45 attendees were given the opportunity to learn more about emulation and test the tools developed by KEEP.

The resulting workshop (led by Jeffrey van der Hoeven, KB, at right) was very much worthwhile and enjoyable, with almost the entire KEEP project team sharing its knowledge. For all those interested in the deep technical details, I refer you to the KEEP website where all the slides will appear soon. For those of you who would like a broader overview, here are my experiences from the last few days (with a selection of slides from the presenters; thanks, KEEP team, for allowing me to use those).

The audience included representatives from libraries, archives, a technical university, two The Hague-based criminal courts, a criminal information service, a film museum and a museum for media art.

What emulation is

In my general introduction to the workshop (what is the problem, etc.), I gave a definition for dummies that was not contested by the experts in the room, so I dare repeat it here. Our problem is that digital objects really only work on the hardware/software combination on which they were made. But these platforms change quickly. Migration is a strategy whereby you change the digital object to make it run on a newer platform. In the long term this is risky, because migrations are never 100%. There are always little changes. Until one day the object no longer is what it is supposed to be. Plus: it only works for a limited number of relatively simple file formats (text, images).

An alternative is emulation. You leave the digital object as it is, but you build software to make the new hardware/software combination pretend it is the old combination.

One can imagine this requires a lot of technical expertise about the old platform and the new platform and this for every possible file format around, of which there are thousands, plus all the dependencies within more complex objects such as websites.

Hence, it does not seem very practical. But when it works, you get a more authentic look and feel than with migration. Two participants in the project come from the computer game industry (Computer Spielemuseum and the European Games Developer Federation, EGDF), where look and feel is everything. Quite a few single emulator programmes have been developed already and new ones are developed all the time. But these are just pieces of the puzzle.

Emulation according to KEEP in four basic steps

Bram Lohman of Tessella summed up emulation according to KEEP in four basic steps, and I will let his slides speak for themselves:

 

The KEEP approach

The KEEP project wants to make emulation more practical by building a number of black boxes that take care of the most technical aspects of emulation; in this slide they are the green boxes.

 

• The Transfer Tools Framework is your assistant during the ingest phase. It helps you get content from different media (floppy disk, cd-rom, cassette, etc.) and store it into your digital archive.
• At the access stage, there is the Emulation Framework, that analyses the digital object, finds the proper emulator software and delivers everything you need to run the object in a new computer environment.
• The last building block is the KEEP Virtual Machine. This is the most daring goal of the project, and you can see in the slide that it is only pencilled in for now. Emulators themselves are pieces of software which become obsolete as hardware platforms change. The KEEP Virtual Machine is a piece of software that  is so basic and universal that it will ‘port’ into many hardware platforms and thus make emulator software much less prone to technology changes. This is the ‘portability’ part of the KEEP acronym.

A special component of the Emulation Framework is TOTEM, which helps users find technical information about the digital object to be emulated in various available databases such as PRONOM (see earlier post):

What emerged from the workshop

The attendees at the workshop were impressed by the work of the KEEP project. Jacob Taekema of the Amsterdam Municipal Archives (previously Rotterdam) concluded at the end of the two-day event that he could now see emulation developing into a really viable option. Aminata Kpewa and her colleagues of the Special Court of Sierra Leone were really happy to discover a network of people and a tool framework that could finally help them sort out the mess of obsolete databases in their archives.

The important accomplishment of the KEEP approach is that it takes care of a lot of the most technical, expert work (finding the best emulator, configuring the original environment), thus making emulation more accessible for less technically expert staff. The good news is also that the framework can be included into any existing preservation solution – Safety Deposit Box producer Tessella is one of the partners in the project. In addition, all of the work of KEEP is available in open source.

Representatives of the KEEP team discussing archival problems with representatives of the Special Court for Sierra Leone – and making plans for on-site help.

But the KEEP team itself readily admits that much, much work still needs to be done:

• The framework now works for a limited number of platforms and file formats (slide at right), and, obviously, many more should be added to really make the framework practicable. As yet the framework cannot yet handle complex objects, although the emulation strategy is considered the most promising for those. It is a matter of finding the resources to put in lots of more R&D work.
• At the ingest point (the tool transfer framework), the approach still depends on the availability of all sorts of old hardware to read obsolescent media carriers (floppy disks, tapes, etc.). Such devices are being collected by private individuals and sometimes by computer museums, but it was suggested that perhaps we need a more structured approach to safeguarding these essential machines and the knowledge about how to use them (nationally, perhaps, such as within the Dutch Digital Preservation Coalition?)
• Whoever wants to work with an emulated object will have to know how that programme worked, what the commands were, etc. Thus we need to collect and keep as many manuals and specifications as possible. KEEP is building a knowledge base.
• The KEEP project itself will end in February 2012. But much work still needs to be done. The KEEP team are doing everything they can to transfer the project to an organization such as the Open Planets Foundation (OPF), which can continue to develop the approach. Let us hope that this transfer is successful and more resources will become available to develop emulation as a strategy, e.g. from the European Community.

Digital preservation stumbling block: copyright law

A major stumbling block for KEEP project, for emulation, and, indeed, for all of digital preservation, is copyright legislation. David Anderson of the University of Portsmouth took upon himself the ungrateful task of finding out about possible legal constraints, and, unfortunately, he found (too) many. In fact, the deliverables of the KEEP project itself will be handed over to the European Commission in loose components because of copyright restrictions. Which is, when you come to think of it, pretty ridiculous. I will write a separate blog post about this important subject and David’s work soon.

More KEEP workshops planned

If this post has wetted your appetite for emulation, you are most welcome to attend one of the forthcoming KEEP national workshops: 10 November Zagreb, 29 November Rome, 24 January Cardiff. Details on the website.

Alexander Fernandez on behalf of the European Games Developer Federation EGDF: ‘Involve game developers in preserving their work and building the emulation knowledge base.’

(For those who question the need to preserve games: they are part of our cultural heritage and they play a growing role in education. Plus: there is a keen and creative community out there developing those games, and they may well help projects such as KEEP to develop means to preserve other complex objects.)

PS: Not to complicate matters too much, and also because it was not mentioned during the workshop, I did not speak, in this blog post, of a third preservation strategy emerging from the CASPAR project, the ‘representation information network’; everything about that in an earlier post.

Are we battling risks or adding value? (#DPS2011) (4)

Despite all our efforts, the digital preservation community is still having a hard time “selling” its activities to funders and to society at large. As Bohdana Stoklasova of the Czech National Library said at the Goportis Digital Preservation Summit: our work is unglamorous and undervalued.

Bohdana Stoklasova making sure that the funders of the Czech National Library’s digital projects get their credit’s worth – as specifically required by funding deals.

Recently, the argument has been raised that we should stop describing our work in terms of ‘risk’ and ‘loss’ and ‘threats’, but rather speak in terms of ‘adding value’. During the panel discussion and coffee breaks at the Goportis Digital Preservation Summit, however, the consensus was that bad news sells better than good news. By suggesting that we organize a little data disaster I may have gone a little too far, but to my mind risk management is still the best approach to thinking about digital preservation.

At the conference, Angela Dappert (previously British Library, now at the UK Digital Preservation Coalition) made the case for the risk management approach. In her view, risk management is central to all digital preservation activities. ‘Risk’, Angela began, ‘is uncertainty of outcome.’ It is something that might happen, and if it happens, it will become an issue’ (a problem). Risk management is about preventing risks from becoming issues. In that sense preservation is proactive whereas conservation is usually reactive (something has happened).

Below are some slides from Angela’s presentation (thanks for these, Angela); if you are interested in the subject be sure to check out the full presentation at http://www.digitalpreservationsummit.de/presentations.html.

It is important to note that not all risks need to be addressed immediately. It is a matter of balancing the risks against the available means to combat them. In some cases one may wish to accept a risk (e.g., when the collection is not essential for the organization, or when it concerns masters of digitized collections for which the analogue original is still available):

Angela gave an overview of available tools for risk management, including the most well-known framework, Drambora, and a new kid on the block in which Angela is involved, the Timbus project.

Just try and manage the risk of losing cultural heritage in a city like Hamburg …

An encyclopedia of file format information? (#DPS2011) (3)

Ingest being as complicated as it is (see previous post), let alone what comes after …, would it not be a terrific help if there was an encyclopedia where the best knowledge of the community was continually being assembled, structured and made accessible for all of us to work with? An encyclopedia telling us, e.g., what .pdf is, what hardware/software combination we need to render it, how the computer can recognize it, what versions and variations are known, what complications are known and how to deal with those, and, last but not least, what our colleagues’ experiences are with migrating or emulating this file format? Answers to the question: what is the best preservation strategy for this file format?

Would not that be heaven on digital preservation earth?

Format registry ‘ecosystem’ workshop at Goportis DPS2011.

Well, in fact, the community has been working on such an encyclopedia (more commonly known as a ‘file format registry’) for quite a while now. PRONOM of the UK National Archives was considered a great start, but the upkeep of the system was just too much work for one institution. Then came initiatives such as the Global Digital Format Registry (GDFR) and the Unified Digital Format Registry (UDFR) – which is, as yet, in the planning stage.

The Goportis Digital Preservation Summit organized a workshop on this issue, led by Bill Roberts of the Dutch National Archives. In his introduction he listed the issues involved in setting up and maintaining such a registry: if the work is too much for one or a couple of organizations and we go for a decentralized approach (Wikipedia-style): Whom Do We Trust? How do we know that the information in the registry is reliable?

Workshop leader Bill Roberts: ‘The most common preservation strategy I know of is “stick it in the archive for now”.’

The workshop discussions clearly demonstrated that there is a lot that is difficult to agree upon across the (broad) community. The question: ‘Do we even have an standard for a file format?’ prompted laughter from the workshop attendees.

But some undaunted souls can always be found to give a good idea a new chance. Bill Roberts reported on such an idea, arisen within the Open Planets Foundation. The idea is to set up a ‘Registry Ecosystem’. The objectives are:

And the approach is:

The workshop attendees supported the idea and agreed on a number of issues:

• if possible find and mobilize the talent of ‘lonely geeks’
• include as much prior knowledge as possible (although vendors such as Microsoft will be extremely frustrating partners to work with)
• there should be common core data models
• a test corpus is very much needed
• the scope of the ecosystem should be broad and include such things as preservation policy procedures and software tools.

On the spot a highly qualified working group was set up to work on the registry ecosystem (including Bill Roberts, Leo Konstantelos of Glasgow, David Anderson and Janet Delve from Portsmouth, Adrian Brown of the Parliamentary Archives and Michelle Lindlar of Goportis; some other names were mentioned, but have yet to be approached). The working group is to meet virtually and deliver results within three months.

And, to quote Bill’s last slide: ‘The most important thing: Lots of people out there have pieces of the puzzle; we must encourage and enable them to share.’

PS: Leo Konstantelos suggested dropping the term ‘file format registry’, because the scope of what is described here is much larger. He suggested ‘technical registry’.

See also Bill Roberts´own blog post at the Open Planets Foundation website.

On ingest, or: “receiving” is a complex word (#DPS2011) (2)

The second day of the Goportis Digital Preservation Summit was all about ingest, or 'receiving' content in your repository or archive. In his keynote, Seamus Ross, formerly of the digital preservation taskforce at Glasgow and now at the University of Toronto, was quick to dispell any illusions that 'receiving' is an easy thing to do.

Interoperability troubles – from Open Office to Powerpoint – causing stress before Ross’s presentation (Nina Stoffers, left, Ross right).

Ross’s presentation was a complete Ingest 101 course, and so, I will let his slides tell most of the story.

Ingest is about “receiving” content from producers:

Ideally, we would want to create a work flow that is consistent, error-free, well-documented, in accordance with our organization’s policies:

Preferably, you know who the producers of your content are and you start negotiating with them so that they deliver the best possible quality. However, keep in mind that whatever makes our lives easier, is most likely to make the producer’s lives more difficult. That is where the bargaining begins. Ideally, you get this:

But in practice, this is most likely what you get most of the time:

Seamus Ross: ‘Most of the work we do during ingest is about fixing all these errors, is about compensating for the communication failures between producers and archives.’ So, what do we do?

How do we do all this? Ross: “You are a craftsman. You must accept that your tools are blunt.” Present tools for identification and validation are far from perfect. They still require a lot of manual work and the people who work with them must be very knowledgeable. Also, “You may be sure that producers will deliver error-laden stuff, no matter how well you train them.”

Ross stressed that policies are an essential part of the equation:

But even policies cannot guarantee smooth sailing:

Having said that, Ross did have a list of useful reference material for the audience, including an instructive case study at http://artefactual.com/wiki/index.php. Check out his slides when the complete set comes available via the event website. He also mentioned the useful NDIIPP tools and services directory at http://www.digitalpreservation.gov/partners/resources/tools/index.html and the Cairo Tools Survey. But remember Ross’s warning that working with these tools requires quite a bit of prior knowledge.

Seamus Ross: “But do not worry too much – digital archaeology will play a role in the future.” 

During the Q&A Adam Farquhar of the British Library offered his more optimistic view of the state of digital preservation (see yesterday’s post). Ross’s reply: ‘But that concerns only a narrow range of object types.’ Databases, for instance, are still a very real problem to deal with.

Goportis co-organizer Yvonne Friese checking #DPS2011 tweets.

More good stuff from this densely packed conference in the next few days. About whether OAIS is still helpful, about tools, about file format registries. And about thinking before you act, the New Zealand version.

Digital Preservation Summit (1): how far have we come?

We have gathered in cold, rainy, windy Hamburg today and tomorrow for the Goportis Digital Preservation Summit (#DPS2011). So the sunny note on which Adam Farquhar of the British Library started off the conference was quite welcome - except that some of us (including the undersigned) saw a few more clouds in the sky than Adam. A matter of the cup being half full or half empty?



Adam Farquhar (left) with Angela Dappert (DPC)

Farquhar said our progress to date is 'pretty encouraging'. 'Digital preservation has become business as usual,' he said, 'for large memory institutions.' Now that I reread my notes, the addition about large memory institutions is probably crucial to Adam's argument, but what stuck in my mind, and also in the mind of Steve Knight from the National Library of New Zealand (both of us presented during the day), is that in our opinion, digital preservation is still quite a long way off from being business as usual for most of the stakeholders - including data producers, funders, and all but the very largest memory institutions.

 

Adam also said that we are doing digital preservation 'at a substantial scale', citing recent BL projects involving the migration of millions of objects - whereas in my community (the Netherlands Coalition for Digital Preservation) I hear much grumbling about the (lack of) scalability of the tools we have at our disposal at present.

 

Fortunately (I mean in terms of agreeing on issues), Adam also saw a number of challenges:

• changes in digital materials (flash, social media with short urls)
• content in context - when a publication is commented upon over the years, it changes
• dynamic content - complex objects such as 3D interactive views of crystals; html5 (which incorporates javascript elements)
• a lack of skills in memory institutions - which is getting worse because of the budget cuts.

And I agree with all those (and surmise that Steve Knight will too).

At the end of his keynote, Adam Farquhar said two things:

• Do not wait until we know everything to get it right, but do whatever you can now
• Within our community, we need to become more honest about what works and what does not. That is the only  path to true learning.

To which I can only say: hear! hear!

There is much more good stuff to report from this conference (including instructive disagreements between presenters), but this time live or even semi-live blogging is difficult because I am presenting and moderating myself - plus: this conference is very well organised and the audience does not get any (boring) time off for blogging. Also, it is a 9 to 6 programme, and your blogger needs time to eat and sleep. So, dear readers, I must ask for a little patience. But I assure you: ALL shall be revealed  ... and in a matter of days even, because then we have the KEEP workshop coming up, and iPRES ...

On a more practical note: powerpoint and laptops are wonderful inventions, but can somebody PLEASE come up with a solution whereby every presenter is visible to the audience?
(Thanks to Natalie Walters for allowing me to use this image)