Taking emulation another couple of steps further (iPRES11, 5)

It is Friday morning, 9 pm. In the other room they are talking about cost modelling. I have opted for the emulation session, continuing the thread from the KEEP workshop I blogged about last week. Judging by the number of participants, emulation is not yet a “hot” topic in the DP community. But perhaps it is only because it is Friday morning, the third conference day – people keep sneeking in with cups of black coffee in their hands.

The first paper conveys a really bright idea, which has also come up in the Netherlands (Maurice van den Dobbelsteen at the National Archives). If your job is archiving large amounts of data from a controlled environment (e.g., a government ministry), would it not be a great idea to simply make a virtual copy of the entire hardware/software environment in which the objects are produced? Then, when the data come into the archive, all you have to know is when they were produced and at which ministry, and your emulation environment is ready to go. That would save loads of work at the preservation stage.

A similar procedure could work if you want to harvest the archive of significant persons. For example, there was a project to emulate Salman Rushdie’s computer environment, to be able to access his files later.

Euan Cochrane (left) and Dirk von Suchodoletz

Euan Cochrane of the Archives of New Zealand and Dirk von Suchodoletz of the University of Freiburg present this approach at iPRES2011. They ran some tests and inevitably found some technical problems, but nothing that cannot be overcome. And I can imagine that, if it works, it can really be a time and money saver, especially in the long run. However, as always, there are challenges. Some are inherent to all emulation strategies: you need workable emulators and emulators themselves become obsolete – for which, of course, there is the KEEP approach which I blogged about earlier. Bram Lohman is presenting KEEP here in a minute.

One obstacle is unique to this particular approach: the data producers should include ‘emulatability’ in their calls for bids for computer systems. Technically this should not be too difficult, but in terms of licenses, there may be catches. I will blog about the copyright problems later. The KEEP project did a lot of work on that.

The next presentation raises the level of complexity a bit – at least for non-techies like me. It is entitled: “Using emulation as a tool for migration”, by Klaus Reichert et al.

Emulation developers: from the left Dirk von Suchodoletz, Klaus Reichert and Euan Cochrane

Because I did not understand it very well, I asked Klaus during coffee break. This is my layman’s version of how it works: almost every software programme comes with “little” migration tools, e.g., between Word 2003 and Word 2010. If you want a file that does not run on your present software/hardware combination and there is no direct migration tool, you can re-create the original environment (emulation), perform the “little” migration there, and then use the resulting file on your present system. The advantages are that you do not need to write new migration routines and you can use the file within your present context – without the old “handicaps”, if you will. See the slide below.

 The case studies in this session include Mark Guttenbrunner, on home computer software emulation, Roman Graf & Reinhold Huber-Mork on braille conversion, and Geoffrey Brown (Indiana University), on emulating some interactive Voyager (1989-1997) publications on cd-rom, classic Mac applications such as Robert Winter’s Interactive Beethoven’s 9th (you could play themes, play notes, play synthesizer versions without certain rhythms, etc.). Let me give you the screenshot as a reminder of what we do all our hard work for (you have to imagine the music). You can find the technical details in the proceedings.

With vintage American pragmatism, Brown said: ‘Our goal is to demonstrate that emulation is practical’ – and that is what we all want. But he also said: ‘At the moment, a lot of this is hobby stuff.’ And there seems to be lot of that in the emulation environment. To really make it work we need more sustainable initiatives such as, perhaps, the Open Planets Foundation (OPF Director Bram van der Werf is in the room  …).

Geoffrey Brown answering questions.

After this semi-live blogging session, we are off to lunch. This afternoon there is the workshop on International Alignment in Digital Preservation, in which I am involved myself. As it is Friday afternoon, the last conference day, and as we have competition from a session on web analytics and of organized visits to the National Library and National Archives, we are not expecting the greatest turnout. But quality can do a lot to make up for quantity ;-)

Expect a few more posts, though. Because of time constraints I had to forego some good stuff. I will report on that in the course of the next few days.

What to Wear is not only a ladies’ question here – air conditioning sometimes works too well for some of us.

 

Emulation is becoming a viable option (#KEEP) (1)

Just about everybody in our community has heard of emulation as an alternative preservation strategy to migration. But emulation is mostly talked about -- and in terms that are not very encouraging: emulation is rumoured to be very difficult to do, to require lots of expert knowledge and to be really expensive. On 26-27 October the road show of the European KEEP project came to the KB in The Hague and some 45 attendees were given the opportunity to learn more about emulation and test the tools developed by KEEP.

The resulting workshop (led by Jeffrey van der Hoeven, KB, at right) was very much worthwhile and enjoyable, with almost the entire KEEP project team sharing its knowledge. For all those interested in the deep technical details, I refer you to the KEEP website where all the slides will appear soon. For those of you who would like a broader overview, here are my experiences from the last few days (with a selection of slides from the presenters; thanks, KEEP team, for allowing me to use those).

The audience included representatives from libraries, archives, a technical university, two The Hague-based criminal courts, a criminal information service, a film museum and a museum for media art.

What emulation is

In my general introduction to the workshop (what is the problem, etc.), I gave a definition for dummies that was not contested by the experts in the room, so I dare repeat it here. Our problem is that digital objects really only work on the hardware/software combination on which they were made. But these platforms change quickly. Migration is a strategy whereby you change the digital object to make it run on a newer platform. In the long term this is risky, because migrations are never 100%. There are always little changes. Until one day the object no longer is what it is supposed to be. Plus: it only works for a limited number of relatively simple file formats (text, images).

An alternative is emulation. You leave the digital object as it is, but you build software to make the new hardware/software combination pretend it is the old combination.

One can imagine this requires a lot of technical expertise about the old platform and the new platform and this for every possible file format around, of which there are thousands, plus all the dependencies within more complex objects such as websites.

Hence, it does not seem very practical. But when it works, you get a more authentic look and feel than with migration. Two participants in the project come from the computer game industry (Computer Spielemuseum and the European Games Developer Federation, EGDF), where look and feel is everything. Quite a few single emulator programmes have been developed already and new ones are developed all the time. But these are just pieces of the puzzle.

Emulation according to KEEP in four basic steps

Bram Lohman of Tessella summed up emulation according to KEEP in four basic steps, and I will let his slides speak for themselves:

 

The KEEP approach

The KEEP project wants to make emulation more practical by building a number of black boxes that take care of the most technical aspects of emulation; in this slide they are the green boxes.

 

• The Transfer Tools Framework is your assistant during the ingest phase. It helps you get content from different media (floppy disk, cd-rom, cassette, etc.) and store it into your digital archive.
• At the access stage, there is the Emulation Framework, that analyses the digital object, finds the proper emulator software and delivers everything you need to run the object in a new computer environment.
• The last building block is the KEEP Virtual Machine. This is the most daring goal of the project, and you can see in the slide that it is only pencilled in for now. Emulators themselves are pieces of software which become obsolete as hardware platforms change. The KEEP Virtual Machine is a piece of software that  is so basic and universal that it will ‘port’ into many hardware platforms and thus make emulator software much less prone to technology changes. This is the ‘portability’ part of the KEEP acronym.

A special component of the Emulation Framework is TOTEM, which helps users find technical information about the digital object to be emulated in various available databases such as PRONOM (see earlier post):

What emerged from the workshop

The attendees at the workshop were impressed by the work of the KEEP project. Jacob Taekema of the Amsterdam Municipal Archives (previously Rotterdam) concluded at the end of the two-day event that he could now see emulation developing into a really viable option. Aminata Kpewa and her colleagues of the Special Court of Sierra Leone were really happy to discover a network of people and a tool framework that could finally help them sort out the mess of obsolete databases in their archives.

The important accomplishment of the KEEP approach is that it takes care of a lot of the most technical, expert work (finding the best emulator, configuring the original environment), thus making emulation more accessible for less technically expert staff. The good news is also that the framework can be included into any existing preservation solution – Safety Deposit Box producer Tessella is one of the partners in the project. In addition, all of the work of KEEP is available in open source.

Representatives of the KEEP team discussing archival problems with representatives of the Special Court for Sierra Leone – and making plans for on-site help.

But the KEEP team itself readily admits that much, much work still needs to be done:

• The framework now works for a limited number of platforms and file formats (slide at right), and, obviously, many more should be added to really make the framework practicable. As yet the framework cannot yet handle complex objects, although the emulation strategy is considered the most promising for those. It is a matter of finding the resources to put in lots of more R&D work.
• At the ingest point (the tool transfer framework), the approach still depends on the availability of all sorts of old hardware to read obsolescent media carriers (floppy disks, tapes, etc.). Such devices are being collected by private individuals and sometimes by computer museums, but it was suggested that perhaps we need a more structured approach to safeguarding these essential machines and the knowledge about how to use them (nationally, perhaps, such as within the Dutch Digital Preservation Coalition?)
• Whoever wants to work with an emulated object will have to know how that programme worked, what the commands were, etc. Thus we need to collect and keep as many manuals and specifications as possible. KEEP is building a knowledge base.
• The KEEP project itself will end in February 2012. But much work still needs to be done. The KEEP team are doing everything they can to transfer the project to an organization such as the Open Planets Foundation (OPF), which can continue to develop the approach. Let us hope that this transfer is successful and more resources will become available to develop emulation as a strategy, e.g. from the European Community.

Digital preservation stumbling block: copyright law

A major stumbling block for KEEP project, for emulation, and, indeed, for all of digital preservation, is copyright legislation. David Anderson of the University of Portsmouth took upon himself the ungrateful task of finding out about possible legal constraints, and, unfortunately, he found (too) many. In fact, the deliverables of the KEEP project itself will be handed over to the European Commission in loose components because of copyright restrictions. Which is, when you come to think of it, pretty ridiculous. I will write a separate blog post about this important subject and David’s work soon.

More KEEP workshops planned

If this post has wetted your appetite for emulation, you are most welcome to attend one of the forthcoming KEEP national workshops: 10 November Zagreb, 29 November Rome, 24 January Cardiff. Details on the website.

Alexander Fernandez on behalf of the European Games Developer Federation EGDF: ‘Involve game developers in preserving their work and building the emulation knowledge base.’

(For those who question the need to preserve games: they are part of our cultural heritage and they play a growing role in education. Plus: there is a keen and creative community out there developing those games, and they may well help projects such as KEEP to develop means to preserve other complex objects.)

PS: Not to complicate matters too much, and also because it was not mentioned during the workshop, I did not speak, in this blog post, of a third preservation strategy emerging from the CASPAR project, the ‘representation information network’; everything about that in an earlier post.

Is emulation something for you (2)

If you tried to register for the The Hague Keep workshop (26-27 October, see last post) but were told the workshop was sold out, you may want to try again. Twenty more seats have been made available.

Is emulation something for you?

In their efforts to come up with catchy acronyms, project managers sometimes think of  wonderfully sounding names that, however, tell you too little about what is really going on. The European KEEP project is a case in point to me: KEEP stands for ‘Keeping Emulation Environments Portable’. Perhaps it is just my slow brain and/or the often fuzzy official project language, but for some reason I kept getting visions of shopping bags and briefcases …

On the occasion of the KEEP road show, which is coming to The Hague on 26-27 October (and to Zagreb, 9-11 November, Rome: 29-30 November), I asked KB colleague and KEEP participant Jeffrey van der Hoeven (at left, emulating KEEP user satisfaction) to explain it to me. Here is my version of what he told me:

The most well-known method to deal with software and hardware obsolescence is migration: you change the bits and the bytes of a digital object to make them work on a new platform. However, migration turns out to be not at all as risk-free as we would hope. Plus: it does not work for complex objects such as video games, websites, etc.. An alternative is emulation: you do not change the bits, but write software to make a new computer function as if it were an (old) computer. This means writing emulators for every possible combination (which is a lot of expensive R&D work), but if it works, there are fewer risks involved than with migration.

However, working with emulators is not for dummies. It is technically challenging work for specialists. The KEEP project developed an ‘emulation framework’ that takes care of that. It automatically selects the right emulator and configures the software required to render the object. That sounds quite handy.

Now, what about the ‘portability’? Emulators themselves are pieces of software that become obsolete over time. Therefore, KEEP is developing a KEEP ‘virtual machine’ – that will allow for execution of any software on any platform at any time.

Does this sound too good to be true? Come to The Hague (or Zagreb or Rome) and find out for yourself. Be sure to bring some old obsolete floppies with you to test the systems hands-on.