Duurzame toegang (long-term access)

This blog has moved / Deze blog is verplaatst

2011-12-17T14:35:00.004+01:00

As of 15 December 2011, this weblog and its archive have been incorporated into the NCDD website, new url: http://www.ncdd.nl/en
or directly to the weblog, new url: http://www.ncdd.nl/blog/

Per 15 december 2011 is deze blog met archief en al verplaatst naar de NCDD website, nieuwe url: http://www.ncdd.nl
of direct near het weblog, nieuwe url: http://www.ncdd.nl/blog/

DISH2011 wrap up: the digital shift and us (DISH 4)

2011-12-08T18:48:00.133+01:00

One can always trust Clifford Lynch of the Coalition for Networked Information to bring the issues home, and DISH2011 was no exception. ‘The digital shift is disrupting our organizations in fundamental ways,’ he said, finally addressing the question that the other four keynotes had left open: what does it all mean for us, for memory institutions.

Here are some of Clifford's observations:

‘In many ways, digital surrogates are more useful, more accessible and more robust than physical objects. That is deeply upsetting for people who have dedicated their entire lives to collecting and maintaining physical objects.’

‘Our new, online, audiences are very difficult to get hold of.’

‘We used to know where our collections began and ended. Our new users have no patience for those, often historical, accidents. To remedy this situation, organizations band together, but this raises new questions of ownership … of the reassembling.’

‘There are many more opportunities now for users to engage and to participate. Sometimes user impact is quite trivial, but it can also be very profound. For a lot of content, there is somebody out there who knows much more about it than we do and he is able to get in touch with us. Just think of the vast volumes of audiovisual content from our living memory. But user generated content does raise issues of trust: to what extent will we, memory organizations, be able or willing to vouch for this content?’ And there is more, ‘These participants may want to contribute more than just tags, they may bring us their own archives, expecting that there should be a place for the memories of all of us.’

Cliff Lynch is an acute observer of what goes on in our information world

Such a development will have a fundamental impact on our acquisitions policies. Many new choices will have to be made. We must talk about those choices, document them, share them with our peers, and thus develop a sense of what is happening. ‘I find that exciting and promising,’ Lynch concluded.

So the question becomes: are we adapting to this new environment? I attended a workshop session on 'national infrastructures' and heard Marco de Niet of the DEN Foundation say: 'We should have done this ten years ago.' He was commenting on Dutch plans to use the Europeana structure and tools to aggregate content from a variety of Dutch institutions on one discovery platform. They call it the 'Netherlands Cultural Heritage Collection' - but really, it is metadata only and, if we are lucky, we will get some thumb nails. A workshop attendee asked the critical question: "Will our users be satisfied with just metadata?" Joyce Ray of the US IMLS figured that no-one would be able to find the money to aggregate the content as well.

Marco de Niet (left) and Hans de Haan addressing the workshop
on national infrastructures. Time ran out before we were able to start
discussing the usefulness of national discovery platforms in an international information space.

But should such practicalities stop us from making bold moves?

In order to give us a sense that all of this is doable, the conference organizers had contracted strategist Michael Edson of the US Smithsonian Institution to give us a final pep talk the American way. His advice: stop thinking and talking in terms of ‘the future’. The pace of innovation is so quick now that we simply cannot spend months or even years talking about strategy. Because if we do, we will fail to recognize the things about digital culture that we can bank on now. In other words: ‘It is all a matter of going boldly into the present.’ Strategy should do work. It is a tool. (The text of his entire speech is on slideshare (edsonm).

Michael Edson

This is a spirit that can work - just look at what the Internet Archive has done with a shoestring budget of $10-15 million. But can it work for us, for you and me?

This is what Edson offered to take with us into the office this Monday morning:

1. What world am I living in?

2. What impact does my organization want to have in that world?

3. What should I do today?

I would say: good luck to all of us!

Grabbing digital preservation by the roots - #DISH2011, 3

2011-12-08T13:28:00.005+01:00

Day 2 starts out with a plenary keynote by Samuel Jones about the importance of culture for our lives. Interesting and entertaining, no doubt about that, but like yesterday’s keynotes (see earlier post) the viewpoint is rather philosophical, and thus it is difficult to determine what heritage institutions can take home from it in terms of concrete advice as to how to deal with all this on a day-to-day-level.

So, after coffee I am heading for something completely different: a workshop by Karin van der Heiden, a freelance Dutch adviser on matters of digital archiving, especially for graphic designers.

Karin van der Heiden (right) with Job Meihuizen of Premsela.

In cooperation with Premsela, a Dutch archive for designers, she has recently published a brilliantly clear brochure entitled ‘Save as …’ giving graphic designers some really basic and helpful advice about how to organize their information and archives (see post in Dutch). A really good initiative, because we all know that choices well made at the production stage can really help keeping the stuff usable over longer periods of time. Focussing on production gets to the problem of digital preservation at the root. There is a Dutch-language version and an English-language version. The website is still only in Dutch, but I have been told that a US edition with website is forthcoming from AIGA, the US professional association for design.

On Twitter, Karin advertised her workshop as ‘digital archiving for dummies’. Perhaps there were a lot of archivists who thought 'that is not for me', because there were only a handful of attendees. But Karin's intention was not to train the archivists, but to train the archivists to train the producers - to train them to get away from complex terminology like OAIS and TRAC, and to enable them to explain to their producers what to do, at the level of explaining to your father what he should do if he wants to be able to look at his grandchild's pictures in 10 or 20 years.

If you think that this is perhaps too basic a level, just remember this: more and more digital content is being produced outside the sphere of influence of heritage institutions. Can you see the boxes of junk coming your way in 10 or 20 years' time and the troubles and expense they will cause? Educating everybody is therefore important to all of us. Karin's mission is to make basic preservation measures doable, enable designers, artists, researchers and everybody else to easily integrate basic measures into their workflow:

Great stuff. I'll let you know when the US edition becomes available.

Playing the 'digital lifecycle game' #DISH2011, 2

2011-12-08T09:08:00.000+01:00

With no set body of knowledge or best practices, training our staff to deal with digital objects remains quite a challenge. The European DigCurV project promotes the availability of vocational training for digital curation. At DISH 2011 DigCurV's Kate Fernie and Katie McCadden presented a really cool training tool: the digital curation lifecycle game. Loosely based on a Monopoly board, the game presents players with real-life questions:

Rony Vissers (Packed, Belgium) searching for answers

'Half-way through your digitization project, the chosen file format is replaced by a new standard. What do you do?' and: 'You get funding to hire a manager for your preservation department. What skills will you require?' ‘You want to digitize your collection of sound recordings, but you do not have the necessary equipment, what do you do?’

Shawn Day, Digital Humanities, seems to feel the threat

These, of course, are no yes or no questions. They are intended to prompt discussion - and in our workshop session there were plenty of inspiring discussions.

The (physical) trial version we played still has a few flaws and Kate and Katie welcomed feedback to improve the game, but even as it was, the game had us working hard and having fun for over an hour. DigCurV intends to make the game available as a member’s bonus, so I would say: by all means, check out the website and join the DigCurV network! DEN Foundation and my own NCDD are members already, and Marco de Niet and I instantly decided that we must make a Dutch version.

How mature is your organization when it comes to digital preservation?

At the same DigCurV workshop Marco de Niet of DEN foundation drew attention to another tool that can be helpful in educating staff. It is a ‘maturity’ model developed by Charles M. Dollar, which was recently used as the basis for Dutch librarian Enno Meijers’ thesis Stapsgewijs naar duurzame toegang (on the NCDD website). The model is used to measure your own organization’s progress against a number of key criteria, and then decide where improvement is needed. Next month Charles Dollar will be speaking about his model at the PASIG Austin conference in Austin, TX, and I will write more about it then.

Are heritage institutions 'living the digital shift'? #DISH2011, 1

2011-12-07T21:30:00.001+01:00

Today and tomorrow I am attending the DISH 2011 conference, or: Digital Strategies for Heritage, a biannual international conference organised by the DEN Foundation and The Netherlands Institute for Heritage (Erfgoed Nederland), two Dutch institutions with remits to promote (ICT) innovation in the cultural heritage sector. Paraphrasing first keynote speaker Katherine Watson of the European Cultural Foundation (ECF), the question apparently still must be asked: 'Are the arts, culture and heritage living the digital shift?'

The conference covers many angles of the digital shift, but obviously I will be on the lookout for sessions and papers dealing with long-term access. Having that focus makes it easier to make choices at this conference, which boasts three blocks of no fewer than fifteen (!) simultaneous parallel sessions - which means you always miss 14/15th of what's on offer. That's a lot to miss, and somebody tweeted: I hope the three plenary keynote presentations make up for the 'sacrifice'.

Did they? I can only give you my own answer: yes and no. Yes in the sense that they gave us a powerful picture of what present-day digital culture is all about. But for me, personally, the three keynotes pretty much covered the same ground and thus I would have been happier with just one keynote and more opportunity to attend a workshop. My preferred keynote would not have come from Katherine Watson or from Charles Leadbeater (although they made good points), but from Amber Case, a ‘cyborg anthropologist’ studying the tools we make for ourselves. These are 'no longer extensions of our physical selves, but extensions of our mental selves'. Case described the collections of digital photographs that we all have as a ‘Mary Poppins bag’ that is weightless – and because it is weightless and sheer unlimited, we do not really feel an acute sense of loss when it goes to waste .... Ah, so that's why it is so difficult to find funding for digital preservation! Yet, the stuff we put on Facebook and Youtube is as much a historical record of our lives as the murals in the Egyptian pyramids were. Only much more fragile and ‘suspended in mental space’.

Amber Case, 'living the digital age', despite her admittedly 'analogue' upbringing.
'In my own back yard, I understood the limits of my mental and physical capabilities.'

Other things we have to come to terms with are the digital age’s ‘simultaneous time’ – there is always somebody awake somewhere in the world and they may be filling your inbox to the brim. Physical proximity to people is losing relevance (‘everybody is always looking at their mobile phones and laptops’ - this conference is no exception) to ‘virtual proximity’ online. We have ‘second selves’ online which we must groom like our first selves. The games we play give us immediate rewards which are addictive. And we shed things and apparatuses like a tree sheds leaves – no more hand-me-downs from previous generations.

This world is fast, it is non-linear, it is mobile; users are actively engaging and have a million choices (Katherine Watson). And, what’s more, according to Amber Case they demand an information/cultural environment in which the interfaces and platforms and websites which we, heritage institutions, have so painstakingly built, disappear entirely into the background. Actions are reduced, queries are eliminated. 'The best technology is invisible; it gets out of the way and connects people.' Users want interfaces to make them feel 'superhuman', 'powerful'.

Charles Leadbeater: Users no longer want to be passive receivers, they want to 'search, enjoy, make, share, do'. The world is becoming 'asymmetric': small investments may have big impacts, and vice versa. Traditional roles and responsibilities no longer work. English football, with fixed roles for footballers (offense) and kicking pushing muscle (defense), has been transformed by (Dutch footballer) Johan Cruyff. At Barcelona FC, everybody must be able to play football.

Chair Chris Batt with a breakdown of the audience of
more than 300 attendees, 75% (my estimate) from the Netherlands

Pfffffhhh ... I turn my head and look at the many familiar faces in the audience. Colleagues from museums, from archives, from academic libraries. I happen to know about the average age of their staff. I happen to know about their budgets. I happen to know about their closets full of floppy disks and cd-roms. I happen to know about their ageing IT systems. And I think, wow, we've got a long way to go ...

BTW: my 1/15th of the afternoon programme, a DigCurV workshop about digital curation education, was well worth it. More about that tomorrow!

Een "infrastructuur": wat is dat en hoe bouw je het?

2011-12-05T10:52:00.003+01:00

Velen van jullie hebben de afgelopen jaren de missie van de NCDD op een van mijn powerpointdia's zien staan:

Dat is een hele mond vol, en dus zeg ik er altijd maar bij: wat wij daaronder verstaan is een landelijk dekkend netwerk van voorzieningen: mensen, kennis, opslagfaciliteiten, software, hardware, opleidingen, en, niet te vergeten, financiering.

Over een mond vol gesproken ...

Hoe vlieg je zoiets groots aan? De ene methode is de deltaplanmethode: grootschalig, hoog-boven-over. In polderland Nederland zie je zoiets maar zelden. Dan moet het water ons écht aan de lippen staan, zoals in 1953 letterlijk gebeurde. Voor de duurzame bruikbaarheid van onze digitale bestanden is zo een beweging (nog) niet tot stand gekomen. Wij (zeg maar, informatieprofessionals) weten wel van de tijdbom die onder onze digitale informatie tikt, maar die urgentie wordt nog lang niet door alle bestuurders als dringend ervaren.

Dus hebben we binnen de NCDD gekeken of we de uitdaging in min of meer behapbare brokken kunnen opdelen. Daar zijn vier werkpakketten uit gekomen:

Opslag (het betrouwbaar en vooral zo efficiënt mogelijk opslaan van de bits en de bytes, inclusief netwerkverbindingen)
Preservering (wat moeten we nu precies doen om die duurzaamheid te waarborgen - monitoren van de ontwikkelingen (preservation watch), plannen van preserveringsacties, de softwaretools die je daarvoor nodig hebt, R&D, en vooral veel kennis)
Afstemming collectiebeleid (digitale informatie laat zich lastig vangen in de traditionele taakverdeling tussen instellingen, daar moet je nieuwe afspraken over maken)
Kwaliteitszorg en certificering (wanneer is een archief een 'trustworthy digital repository'? Hoe bewijs je dat?)

Voor de eerste twee werkpakketten zijn sinds juni dit jaar NCDD-werkgroepen aan het werk. Zij hebben de opdracht gekregen om diverse scenario's te bedenken voor een landelijke infrastructuur. Die moet a) zo efficiënt (lees: goedkoop) mogelijk zijn, en b) de kwaliteit van duurzame toegankelijkheid in Nederland op een hoger plan brengen - met name ook voor de kleine instellingen die zelf geen digitaal depot kunnen betalen. Alle officiële informatie daarover staat op http://www.ncdd.nl/over-beleid.php.

De werkgroep Preservering aan het werk: rond de tafel Jata Haan (EYE), Giovanna Fossati (EYE, voorzitter), Frédérique Vijftigschild (NCDD support), Paul Doorenbosch (KB), Aad van der Valk (Beeld en Geluid), Mette van Essen (Nationaal Archief), Jeanine Tieleman (DEN) en Andrea Scharnhorst (DANS); ontbreken: Barbara Sierman (KB), Robert Gillesse (DEN) en Gaby Wijers (NIMk). Beelden van de werkgroep Opslag houden jullie van me tegoed.

Aan mij de eer om al dit werk te ondersteunen vanuit de NCDD, en ik kan je zeggen, de werkgroepen hebben het niet gemakkelijk. Duurzame toegankelijkheid is een jong vak met heel veel onzekerheden. Wie kan voorspellen wat voor computers we over 10 of 20 jaar zullen hebben? Wie durft te voorspellen hoe snel het web blijft groeien? Wie durft te selecteren wat we wel en niet moeten bewaren? Wie durft er vandaag definitief te zeggen wat de beste duurzaamheidsstrategie is? En hoe zit het met alle bestuurlijke en juridische complicaties?

Ga er maar aan staan. Niettemin zijn we vol goede moed aan het werk gegaan. Er wordt hersenkrakend nagedacht en geschreven. Ideeën worden geopperd en soms weer van tafel geveegd. Om soms later opnieuw op te duiken als andere alternatieven niet haalbaar zijn gebleken.

Maar we hebben hulp nodig. Van jullie. Daarom organiseren we:

NCDD symposium "Bouw een huis voor ons digitaal geheugen",
24 januari 2012, KB, Den Haag, 10.30 u tot 16.30 u, toegang gratis, wél even aanmelden

Programma en aanmelden op http://ncddsymposium.eventbrite.com.

Digital preservation basics in four online seminars

2011-11-30T18:11:00.001+01:00

If you are new to digital preservation, you may want to check out four ‘webinars’ organized by the California State Library and the California Preservation Program. The one-hour webinars promise to give you a basic understanding of what digital preservation is all about, of interest especially to librarians and archivists who are involved in developing digital projects.

The first webinar is scheduled for December 8, 12 PM Pacific time (which is 21.00 hrs in Holland). Topics include: ‘storing digital objects, choosing and understanding risks in file formats, planning for migration and emulation, and the roles of metadata in digital preservation.’ See http://infopeople.org/training/digital-preservation-fundamentals.

‘Mind the Gap’ and Archive-it – on web archiving (iPRES2011, 9)

2011-11-23T19:46:00.000+01:00

At a reception the other day, I heard a rumour. Because preserving web sites is so difficult, the Internet Archive was rumoured to consider printing all of its content. I will not disclose the informant’s name – he would not have a future in the digital library where he works (OK, it was a guy, a young guy, and he works for a Dutch library.) Needless to say, it could not even be done if the Internet Archive wanted to do it. Lori Donovan told the iPRES audience that a single snapshot of the www nowadays results in 3 billion pages [for the Dutch: 3 miljard pagina’s].

Mind-boggling numbers, especially if you think of the Internet Archive’s shoestring budget.

Anyway, iPRES2011 is over, but I still have some worthwhile stories waiting to be told. One of the issues tabled at iPRES was whether we can (and/or should) safely leave web archiving to the Internet Archive and national libraries.

Logistics put the Internet panel members much further apart than their viewpoints would warrant: they agreed that web archiving is important, not just for national libraries. From the left: Geoff Harder, University of Alberta, Tessa Fellon, University of Columbia, and Lori Donovan, the Internet Archive.

No, said Geoff Harder of the University of Alberta and Tessa Fellon of the University of Columbia. There are compelling reasons for research libraries to get involved as well. Harder: “This is just another tool in collection building; we should not treat it any differently. You begin with a collection policy and an expanded view of what constitutes a research collection: build on existing collections; find collections where research is happening or will happen.”

I would say that perhaps there are even more compelling reasons to collect web content than, e.g., printed books, because web content is extremely fleeting. Harder told his audience: “Too much online (western) Canadian is disappearing; this creates a research gap for future scholars and a hole in our collective memory.” He encouraged research libraries to: “Mind the Gap – Own the Problem”.

The University of Alberta’s involvement in web archiving started with a rescue operation: a non-profit foundation which created some 80+ websites, including the Alberta Online Encyclopedia, went out of business. This was extremely valuable content, and it needed to be rescued fast.

When a time bomb is ticking …

The University of Alberta decided to use Archive-It, a service developed by the Internet Archive. It is a light-weight tool that is easy to get up and running immediately. Plus, said Harder, there is a well-established tool-kit including dashboard and workflows, you become part of an instant community of users and your collection becomes part of a larger, global web archive. Because that is a precondition for working with Archive-It: by default, everything that is harvested becomes publicly available globally. Harder: ‘It is an economical tool for saving orphaned and at-risk web content … where we know a time bomb is ticking.”

Have a look at the collections built with Archive-It, I would say to research libraries’ subject specialists. You can include anything that is interesting in your field, such as important blogs, for as long as they are relevant.

Yunhyong Kim of HATII, Glasgow, takes blogging very serious and is doing research into the dynamics of the blogosphere.

Q&A

Is Archive-It durable enough? asked Yunhyong Kim of Glasgow (HATII). Donovan appeared confident that Internet Archive would be able to continue developing the tool. And I would repeat Harder: when a time bomb is ticking, you have got to go with what is available.

What about preventing redundancy, was another question. Should we not keep a register somewhere of what is being archived? Fellon thought that was a good idea, but perhaps it was too early for that. 'There are many different reasons for web archiving, different frequencies.” Sorting out what overlaps exactly and what does not is perhaps more work than just accepting some “collateral damage”.

If you want to know more about Archive-It, you can sign up for one of their live online demos. There’s one scheduled for November 29 and one for December 6. See the website

Archive-It Singapore-style

PDF/A-2: what it is, what it can do, what it cannot do, and what to expect in the future

2011-11-21T19:10:00.000+01:00

There is a new PDF ISO standard, 19005-2, or PDF/A-2, and therefore the Benelux PDF/A Competence Center decided to organize a seminar. When one of the organizers, Dominique Hermans of DO Consultancy, asked me to do the warming-up presentation, I readily agreed, because I had been hearing some bad things about PDF these last few months, and was eager to find out more. While preparing my own talk (slides at the end of this post) I decided to quote those very criticisms (see LIBER2011 blog post), just to get the ball rolling and challenge the experts to comment:

This slide of mine is a mash-up of three slides by Alma Swan at the LIBER 2011 conference, Open Access, repositories and H.G. Wells

These criticisms come from people who want machines to analyse large quantities of data in a semantic-web/Linked Data-type environment. Are the criticisms justified? For those of you who, like me, are sometimes confused about what is and what is not possible, I will summarize what the experts told the seminar.

The key one-liner came from Carsten Heinemann of LuraTech:

“PDF was designed as electronic paper”

‘It was designed to reproduce a visual image across different platforms (PC, Mac, operating systems), and for a limited period of time.’ As such, PDF was a really good product, because it was compact and complete and it allowed for random access. But there were also many issues, and Adobe has been working on fixing those ever since. This has resulted in an entire family of PDF formats with different functionalities.

PDF/A is the file format most suited for archiving purposes. The new standard, PDF/A-2 is not a new version of PDF/A-1 in the sense that one would need to migrate from 1 to 2, but rather a new member of the PDF family tree that has improved functionality over PDF/A-1. In order words: migrating from PDF/A-1 to PDF/A-2 is senseless, but if you are creating new PDF documents you may want to consider PDF/A-2 because of the new functionality to incorporate more features from the original document (e.g., JPEG2000 compression, possibility to embed one file into another, larger page sizes, support for transparency effects and layers).

To make matters more complicated, PDF-A/2 comes in two varieties. Compliance level 2a and compliance level 2b. Level a allows for more access by search engines such as used in semantic web techniques, because it requires that files do not only provide a visual image, but that they are structured and tagged and include Unicode character maps.

Heiermann concluded: XML is for transporting data; PDF is for transporting visual representations. To which I may add: XML is for use by machines, PDF is for use by humans.

Misuse of PDF is easy

Raph de Rooij of Logius (Ministry of the Interior) told his audience that one should not be too quick to say that something is “impossible” with PDF. A lot is possible, but you have to use the tools the right way – and that is where things often go wrong.

Raph demonstrated that most PDFs put online by government agencies do not meet the government’s own requirements for web usability – including access by those who are, e.g., visually impaired. “The many nuances of the PDF discussion often get lost in translation,” he said. The trick is to pay a lot of attention to organizing the work flow that ends in PDFs.

PDF is no silver bullet

Ingmar Koch, a well-known (blogging) Dutch public records inspector, has seen many examples of PDF misuse. “Public officials tend to think of PDF as a silver bullet that solves all of their archiving problems”. But PDF was never designed to include anything that is not static (excel sheets with formulas, movies, interactive communications, etc.).

From the left: Caroline vd Meulen, Ingmar Koch, Bas from Krimpen a/d IJssel and Robert Gillesse of the DEN Foundation.

From a preservation point of view, I heard some shocking case studies from public offices. An official will type the minutes of a council meeting in Word, make a print-out, have the print-out signed physically, then OCR the document and convert it to PDF for archiving. I dare not imagine how much information gets lost in the process. But then again, we all know that data producers’ interests are often different from archives’ interests. Public offices just want to make a “quick PDF” and not be bothered by all the nuances.

How about validation?

There is a lot of talk about “validating” PDF documents. First of all, PDFs are created by all sorts of software, and what they produce often does not conform to the ISO standards and is thus rejected by validators. Things get more confusing when validators turn out different verdicts. Heinemann explained: “That’s because some validators only check 30%, whereas some will check 80%. The latter may find something the first did not see.”

At the end of the day …

It seems that, indeed, there are millions and millions of PDFs out there that can only provide a visual representation and are no good when it comes to Linked Data and the Semantic Web. But PDF is catching up, including new features all of the time. I understand that we may even expect a PDF/A-3, which supports including the original file format in the digital object. Ingmar Koch did not seem to be too happy about such functionality. It would make his life as a public records inspector even harder. But from a preservation point of view, that just might be as close to a silver bullet for archiving as we will ever get.

Meanwhile, if you want to use PDF in your workflow, getting some advice from an expert about what type of PDF is appropriate in your case is called for!

Comments by Adobe

Adobe itself was very quick to respond to this blog post in an e-mail I found this morning. Leonard Rosenthol, PDF Architect, was not very pleased with the picture painted by the above workshop – as a matter of fact, he used the word “appalled”. He asserted that PDF and XML/Linked Data go very well together and that various countries and government agencies have already adopted a scenario that ‘presents a best of two worlds’. Here is his link to a recent blog post by James C. King that describes how it is done: <http://blogs.adobe.com/insidepdf/2011/10/my-pdf-hammer-revision.html.

That blog post is an interesting addition to the workshop results (confirming Raph de Rooy’s assertion that “nothing is impossible”), but it does not take away the fact that PDF is often misused. I would guess that is because it is complicated stuff. “Making a quick PDF” just does not do it. The recommendation to seek expert advice, therefore, stands!

Lastly, here is my own presentation: a broad overview of developments in the digital information arena to start off the day – in Dutch:

20111117 pdfa angevaare

View more presentations from ingeangevaare.

For the Dutch fans: Ingmar Koch has blogged about this event here, and the slides will become available here. Thanks also to KB colleague Wouter Kool for helping me understand PDF.

‘Bewaar als …’: glashelder advies over digitaal archiveren

2011-11-20T14:33:00.001+01:00

Karin van der Heiden heeft met Premsela (Nederlands Instituut voor Design en Mode) een glasheldere leporello (uitvouwbrochure) ontwikkeld om vormgevers praktische handvatten te bieden om hun informatie goed te ordenen en goed op te slaan – en dat is het begin van alle langetermijntoegang. Niet alleen belangrijk voor vormgevers, maar voor iedereen die digitale documenten maakt en die goed wil bewaren!

Gefeliciteerd, Karin, met deze productie!

Kijk op de bijbehorende website, http://bewaarals.nl/, en zegt het voort!

PS: Hieronder de hele vellen, in .jpeg.

An English edition will be made available in the US in a few months. I will keep you posted.

Aligning with most of the world (iPRES2011, 8)

2011-11-08T16:53:00.001+01:00

iPRES is organized alternately in Europe, in North-America and in Asia in order to include people and discussions from all continents – Africa and South-America are still on the Steering Committee’s wish list. However, when you looked at the list of presenters at iPRES2011, it was the usual suspects that dominated: Europe, North America, Australia/New Zealand. I asked a Programme Committee member about that, and he told me that some papers had been submitted from Asia, but they were deemed not good enough to make it to the programme.

To my mind, there is a bit of a contradiction in this. Of course we want high-quality papers at iPRES, but it is a bit risky to take our (western) stage of development as a yard stick for what constitutes “quality”. As Cal Lee phrased it: “Digital preservation tends to be quite regionally myopic.” I would suggest that the next iPRES organize a special track or workshop day for those that are just beginning to think about digital preservation, or that work from a very different context than a “western” one and focus on their specific circumstances and challenges.

Fortunately, there was one workshop that expressly invited members from “other” countries. It was the workshop “Aligning national approaches to digital preservation”, a follow-up from last May’s Tallinn conference (see my blog posts), put together by Cal Lee from the University of North Carolina. Yes, there were usual suspects presenting as well (including yours truly), but in this post I shall mostly ignore them in favour of new input:

Özgür Külcü, from Hacettepe University, Ankara, Turkey, described Turkish participation in the AccessIT project, whereby an online education module with practical information about digitisation issues and protection of cultural heritage was developed. And in the context of the InterPares 3 project the Turkish team is helping translate digital preservation theory into concrete action plans for organizations with limited resources. But many issues remain:

Masaki Shibata from Japan, revealed the results of a DRAMBORA 2.0 test audit carried out at the National Diet Library in Japan:

Shibata admitted that, unfortunately, the risks mentioned in the final report largely remain unsolved. ‘We were caught up in an illusion that there was an ideal solution to ensure long-term digital preservation,’ he said. ‘We tried to address the risks only by means of systems development.’ Also, specific Japanese and NDL circumstances played a role, such as the rigidness of the fiscal, budget, employment and personnel system; language difficulties and geographical constraints; lack of digital conservators; and a cultural context of preservation. Shibata concluded that an international alliance for digital preservation ‘would become a boost/tailwind for national policymaking in Japan.’

Daisy Selematsela from the National Research Foundation of the Republic of South Africa, described the outcomes of An audit of South African digitisation initiatives before focussing on “Managing Digital Collections: a collaborative initiative on the South African Framework”, a report published earlier this year, which is meant to provide data producers with high-level principles for managing data throughout the digital collection life cycle; and the Train-the-trainer programme:

As for international alignment, Selematsela concluded:

Raju Buddharaju of the National Library of Singapore (photo right) suggested that we first need a better understanding of what we mean “alignment” and what we mean by digital preservation (what do we include, what do we exclude) before we can try and come to workable initiatives.

The workshop was originally designed as a one-day event, but in the end the conference organizers only gave us 3 hours on Friday afternoon. The good news was that despite the time of day and conference fatigue, more than fourty participants showed up and they conducted animated discussions on such topics as: costs; public policy and society; and preservation & access.

But it was difficult to reach any concrete conclusions. There are many good intentions, but it continues to be difficult to find the common ground that leads to practical results. Steve Knight of the National Library of New Zealand (photo left) questioned whether there is any real will to collaborate, e.g., on putting together a much-needed international format (technical) registry. Talking about education, finally, Andi Rauber suggested that because there is no well-defined body of knowledge, we might prefer a range of “friendly competing curricula” rather than an aligned body – for the time being.

Which only goes to show that, like Singapore itself, alignment comes in many shapes and sizes.

Disaster planning and enabling smaller institutions (iPRES2011, 7)

2011-11-08T08:20:00.001+01:00

As this iPRES was moved from Tsukuba, Japan, to Singapore because of the earthquake and tsunami in Japan in March this year, it was only fitting that iPRES2011 should include a panel session on disaster planning. Neil Grindley (JISC) asked if digital preservation does not implicitly include disaster planning, but Angela Dappart (DPC) argued that with an entire infrastructure going down, the problems will be massively larger. Plus, as Arif Shaon (STFC) observed, ‘Grade A preservation should include it, but we have not reached that stage yet.’

Shigeo Sugimoto of Tsukuba, who would have been iPRES’s host in Japan, took a forward-looking view at disaster planning. Many physical artefacts were lost during the earthquake, and having lots of digital copies at different locations can certainly help rescue cultural heritage, provided the metadata are kept at different locations as well.

Shigeo Sugimoto (right) with José Barateiro of Portugal during the disaster planning session.

There is one catch, though: many smaller institutions do not have the means (money, staff) to build digital archives. Therefore, in Japan the idea has been put forward to design a robust and easy-to-use cloud-based service for small institutions:

In the Netherlands, I am involved in two Dutch Digital Preservation Coalition (NCDD) working groups who are looking at the same problems: how to enable smaller institutions to preserve their digital objects. Professor Sugimoto and I have agreed to stay in touch and exchange information and experiences.

‘At scale, storage is the dominant hardware cost’ (iPRES2011, 6)

2011-11-06T15:13:00.001+01:00

It is not uncommon for conferences to be ‘interrupted’ by sponsor presentations. When I say ‘interrupted’, I do not necessarily mean that such talks are unwelcome. Conference days tend to be packed from early morning to late at night, and such sponsor interventions can be quite pleasant – a moment to doze off or to check your e-mail. Robert Sharpe (photo) of Tessella (vendors of the Safety Deposit Box or SDB system) gave us no such respite. In an entertaining presentation he shared some scalability experiences with us.

The case study was Family Search, which ingests no less than 20 Terabyte of images a day. That was quite a scalability test for the Tessella Safety Deposit Box system, and it tested some of Sharpe’s own assumptions:

Tessella expected that they would need faster, more efficient tools, but it turned out that existing tools (DROID, Jhove, etc.) were easily fast enough.
Tessella expected reading and writing of content to be fast compared to processing, but it turned out that reading and writing were not fast enough; the process required parallel reads and parallel writes. Thus the hardware cost is dominated by non-processing costs.
Tessella (and most of us) expected storage to be cheap, but at scale it turned out to be the dominant hardware cost. Reading and writing hardware came to about GBP 80,000. The storage costs came to GBP 100 per Terabyte content (3 copies), which amounted to GBP 730,000 a year, each year, and without refreshment costs.

Sharpe concluded that we do not need faster tools – but we do need better & more comprehensive tools. We need systems engineering, not just software engineering. And we need enterprise solutions: automation, multi-threading, efficient workflow management and automated issue handling.

All of which, of course, Rob will be happy to talk to you about.

PS: In response to this blog post, Rob wrote to me: ‘A further point I was trying to make in the rest of my talk is you don't need especially powerful application servers to do this: you can do it fairly cheaply (certainly when compared to other costs at such scale).’

Scale Singapore-style: the Marine Bay Sands Hotel. The ship-like contraption on top of the three towers holds lush tropical gardens, a 150 meter swimming pool, restaurants, and a bar.

Taking emulation another couple of steps further (iPRES11, 5)

2011-11-04T05:48:00.001+01:00

It is Friday morning, 9 pm. In the other room they are talking about cost modelling. I have opted for the emulation session, continuing the thread from the KEEP workshop I blogged about last week. Judging by the number of participants, emulation is not yet a “hot” topic in the DP community. But perhaps it is only because it is Friday morning, the third conference day – people keep sneeking in with cups of black coffee in their hands.

The first paper conveys a really bright idea, which has also come up in the Netherlands (Maurice van den Dobbelsteen at the National Archives). If your job is archiving large amounts of data from a controlled environment (e.g., a government ministry), would it not be a great idea to simply make a virtual copy of the entire hardware/software environment in which the objects are produced? Then, when the data come into the archive, all you have to know is when they were produced and at which ministry, and your emulation environment is ready to go. That would save loads of work at the preservation stage.

A similar procedure could work if you want to harvest the archive of significant persons. For example, there was a project to emulate Salman Rushdie’s computer environment, to be able to access his files later.

Euan Cochrane (left) and Dirk von Suchodoletz

Euan Cochrane of the Archives of New Zealand and Dirk von Suchodoletz of the University of Freiburg present this approach at iPRES2011. They ran some tests and inevitably found some technical problems, but nothing that cannot be overcome. And I can imagine that, if it works, it can really be a time and money saver, especially in the long run. However, as always, there are challenges. Some are inherent to all emulation strategies: you need workable emulators and emulators themselves become obsolete – for which, of course, there is the KEEP approach which I blogged about earlier. Bram Lohman is presenting KEEP here in a minute.

One obstacle is unique to this particular approach: the data producers should include ‘emulatability’ in their calls for bids for computer systems. Technically this should not be too difficult, but in terms of licenses, there may be catches. I will blog about the copyright problems later. The KEEP project did a lot of work on that.

The next presentation raises the level of complexity a bit – at least for non-techies like me. It is entitled: “Using emulation as a tool for migration”, by Klaus Reichert et al.

Emulation developers: from the left Dirk von Suchodoletz, Klaus Reichert and Euan Cochrane

Because I did not understand it very well, I asked Klaus during coffee break. This is my layman’s version of how it works: almost every software programme comes with “little” migration tools, e.g., between Word 2003 and Word 2010. If you want a file that does not run on your present software/hardware combination and there is no direct migration tool, you can re-create the original environment (emulation), perform the “little” migration there, and then use the resulting file on your present system. The advantages are that you do not need to write new migration routines and you can use the file within your present context – without the old “handicaps”, if you will. See the slide below.

The case studies in this session include Mark Guttenbrunner, on home computer software emulation, Roman Graf & Reinhold Huber-Mork on braille conversion, and Geoffrey Brown (Indiana University), on emulating some interactive Voyager (1989-1997) publications on cd-rom, classic Mac applications such as Robert Winter’s Interactive Beethoven’s 9th (you could play themes, play notes, play synthesizer versions without certain rhythms, etc.). Let me give you the screenshot as a reminder of what we do all our hard work for (you have to imagine the music). You can find the technical details in the proceedings.

With vintage American pragmatism, Brown said: ‘Our goal is to demonstrate that emulation is practical’ – and that is what we all want. But he also said: ‘At the moment, a lot of this is hobby stuff.’ And there seems to be lot of that in the emulation environment. To really make it work we need more sustainable initiatives such as, perhaps, the Open Planets Foundation (OPF Director Bram van der Werf is in the room …).

Geoffrey Brown answering questions.

After this semi-live blogging session, we are off to lunch. This afternoon there is the workshop on International Alignment in Digital Preservation, in which I am involved myself. As it is Friday afternoon, the last conference day, and as we have competition from a session on web analytics and of organized visits to the National Library and National Archives, we are not expecting the greatest turnout. But quality can do a lot to make up for quantity ;-)

Expect a few more posts, though. Because of time constraints I had to forego some good stuff. I will report on that in the course of the next few days.

What to Wear is not only a ladies’ question here – air conditioning sometimes works too well for some of us.

‘Metadata is a love note to the future’ (iPRES2011, 4)

2011-11-03T09:46:00.001+01:00

The quote in the title comes from a tiny sticker I found here in the conference room. I almost missed it, it is that small. I do not know who put the stickers out there. Perhaps the barcode could tell me, but I haven’t downloaded the app yet. In any case, the quote is one to pass on to you, and in a (bit of a creative) way, it brings together the two keynote speeches on day 2 of iPRES2011. [Post script: Henk Koning of DANS tells me that this is the url: http://mialnttf.tumblr.com/].

Mick Newman of the Australian National Film and Sound Archive (photo left) spoke about ‘Preserving motion film, so much to do and so little time …’ Mick made no secret of his ‘lovely chauvinism’ for analogue film – the film experience for him includes a buzzing reel of acetate film (nitrate being a bit too flammable, after all). Like Richard Wright (BBC) last week in Hamburg, Newman reported that the transition to digital is slow. And not as easy as, e.g., for documents, because films are complex objects containing high-quality images and high-quality sounds, and an ISO file format such as motion-JPEG2000 works for the image, but does a mediocre job with sound. And the files are huge: a 35-mm film with 350,000 frames will turn into a 10 TB digital object. A complex object at that, with formidable metadata challenges (here’s the link to the title), with complicated intellectual property regimes for script, sound, music, set design, etc. etc., causing Newman to quote Karen Van Malssen saying ‘It doesn’t matter what the question is, the answer is metadata.’

At the end of his talk, Newman showed a slide with the pros and cons of analogue preservation vs. digital preservation, and the very fact that he showed the slide at all was notable. In the library and archives community nobody thinks in terms of those ‘or-or’ terms anymore. But, then again, they tend to keep the physical originals whereas Newman made clear that preserving analogue films is very demanding and thus expensive.

In the next keynote the focus was shifted to research data, with Ross Wilkinson of the Australian National Data Service ANDS. Like Seamus Ross yesterday morning (see earlier post) Wilkinson made the case for preservation in terms of added value rather than controlling risks (see last week’s post). With research driving innovation, however, I think his argument is a lot more compelling (towards funders, that is) than just referring to our ‘memory’ and ‘identity’ – which are hard to express in terms of dollars and euros.

Wilkinson (photo left) focussed on increasing the value of research data, a.o. by connecting different databases to generate new information to enable new types of research (and some of that value must come from metadata, hence another creative link to my title). Wilkinson’s added value list is as follows:

‘Data is more valuable, if …

it can be used later
it is able to be used by more researchers
it is able to be used to answer new questions
it is able to be integrated to explore new data spaces …’

‘To do so, it must be managed, connected, discovered, and then re-used – it has to move out of the lab.’

Who is going to do all that work? Wilkinson believes in partnerships – many have a role to play in the process. The researcher, the institution, ‘data carers’ in all shapes and sizes (data librarians, data scientists, etc.), discipline-specific repositories and more general archiving institutions such as ANDS (and DANS in the Netherlands).

Moving research data from the private to the public sphere: each partner has a role to play in adding value to the data

Plus, of course, a number of stakeholders that have a role to play in motiving researchers to share, such as funders and key players in scholarly communications such as publishers. If data become citable, more researchers will be willing to share.

To conclude, this is Wilkinson’s favourate case study of research data reuse:

Q&A: Arif Shaon of STFC in the UK

Conference dinner in the library’s plaza. Yes, the food was again wonderful. And the lemonade kept us sharp for the next day (the barley flavour was especially tasty)

Our kind hosts put an umbrella in the conference pack …

On governance, trust and certification (iPRES 3)

2011-11-02T14:59:00.000+01:00

I am going to blog about this parallel session in reverse order, because that way it makes more sense to me.

At the end of the session (but at the beginning of this post!) Devan Ray Donaldson of the University of Michigan reminded us what ‘trust’ (as in Trustworthy Digital Repositories, or TDR’s) is all about: end users (those that have had no involvement in either production or archiving of a document) need some assurance that the document they are getting from an archive is, in fact, authentic, that it is what it is supposed to be, and has not been tampered with or altered in any way. [BTW: that does not mean that the archive guarantees that the information in the document is reliable. The archive does not know that. The only thing an archive can do is assure that what the end user gets is the same thing that originally came into the archive.]

Archives know that end users care about trust, about authenticity. So Donaldson wants to study how we communicate with the end user about that authenticity. If we put some seal of approval on a document, will the end user trust it more than if we do not put any seal on it? That is an interesting question. Donaldson intends to use HathiTrust documents to test this, and, to me, that is the only ‘flaw’ in his plan – if such is the word, that is. HathiTrust contains digitized book pages, and that type of document is a lot easier to trust and be regarded as “authentic” than, e.g., e-mail. Donaldson agreed, but, as he said: you’ve got to start somewhere.

Next (in whichever order) came Olivier Rouchon of CINES, a large data centre in France (photo right, “Cannot I even have lunch without being photographed?” J). CINES finds itself in a strange political situation: as an organization CINES has a remit for only four years, but it also has the express mandate to do long-term preservation and its clients ask for 30-year guarantees. This is a strange dichotomy and CINES has decided to seek certification as a trusted repository to a) lock the mission, and b) attract larger volumes of data to be preserved.

CINES went through various (self-) audits to attain ever higher levels of certification. That took a lot of work. Rouchon estimates that 1 fte of his 11 fte’s is constantly busy with audits. But, says Rouchon, ‘that should not stop you from doing it.’ First of all, it is mostly a lot of work the first time around. Once you have a good system in place, the next audits become business as usual. Secondly, CINES is using the audit system as an internal quality assessment instrument to keep improving the quality of the service. By comparing the outcome of audits over time the organization can measure its progress.

The EU is now building a three-tiered certification system: the first level is the relatively lightweight Data Seal of Approval, then comes a self-audit, and the highest level of certification is awarded by an external audit. The APARSEN project recently did a number of test audits, a.o. at CINES, and will publish the results shortly.

Steve Knight from the National Library of New Zealand enquired how we know that we can trust the auditors doing the auditing. Rouchon trusts his own (internal) auditors and part of the aim of the APARSEN test audits was to train auditors.

Having talked about trust, and about auditing trust, I now come to the last (first) presentation. Basically, it was about building all the capabilities you need to assure trust and prove trustworthiness into your system. It was also about not dealing with digital preservation as an issue (and a system!) that stands apart from the rest of your organization, but to build an information system for your organization that integrates digital preservation requirements, make them ‘ubiquitous’. Christoph Becker of TU Wien(photo right) told his audience that we have lots of models and concepts and frameworks (OAIS, TRAC, RAC, Drambora, Platter, etc. etc.), but ‘we still lack a holistic view.’ His team takes its cue from frameworks from the IT industry, such as ‘enterprise architecture’, and COBIT (goal-oriented, process-oriented, control-based) to build a Maturity Model based on CMM – you measure your maturity by a set of criteria to identify places for improvement … and then I lost the story. My mind tends to switch off when the discussion becomes abstract and high-level. It is a flaw, I know, but one I have to learn to live with. The basic idea, however, integrating digital preservation, is a good one, and so is using existing industry frameworks, so for those of you who are better at high-level discussions, do check out Becker’s paper in the proceedings which come online soon. The paper is called “A Capability Model for Digital Preservation: Analysing Concerns, Drivers, Constraints, Capabilities and Maturities”.

Parallel session ‘Governance’

iPRES2011: Justifying the cost (2)

2011-11-02T10:58:00.001+01:00

The first day of iPRES2011 is coming to a close. As always after a conference day, the mind is full and most of the information is not yet digested. But here is a first impression. Seamus Ross of the University of Toronto opened the conference with a keynote entitled: ‘Why should today’s society pay for the benefit of society in the future?’

Ross agreed that justfying the costs of digital preservation is difficult. The cost-benefit ratio is unclear, we really have no tangible ground to stand on. We have but a ‘hazy notion of benefits that will accrue to future generations.’

Although, of course, cost modelling work must continue, if only to find the most efficient way to do digital preservation, Ross argued that the most compelling argument for digital preservation is a very human emotion: our innate need to remember, to tell stories, be they of a personal, a historical or a scientific nature. Ross gave some great examples of these types of storytelling, and of course it is true that our identity is very much dependent on traces from and knowledge about our past, be it a personal past or a regional or national past.

In other words: future generations expect us to preserve our history. ‘They will value the information ecosystem just like the environmental one.’ And like environmental curation, digital curation benefits from little, medium and grand actions and initiatives.’

While Ross’s talk was a powerful reminder for us in the preservation industry of why we do this work, and while his appeal to ‘find a place in the public imagination’ helps remind us to build the case not only on technicalities, I must admit that I personally do not think this is enough to convince funders and society at large. We still have a LOT of education to do, about why digital preservation is as expensive as it is now, why it is so complicated, and why a deadly serious suggestion by a Dutch government official only a year ago, to simply ‘print the content and preserve it that way’ is absolutely not an option … Even worse, the suggestion was made in Singapore as well, I was told.

Demonstration: Re-awakening the Philips Videopac

My dislike for parallel sessions was ignited again, forcing me to miss half of what followed. I did attend sessions on governance, risk assessment, trust & certification, and will write about those soon. First things first, dinner is served in the National Library’s plaza. Outdoors, mind you, is where Singapore eats. And eats well! As Mr Beh of the Singapore National Library Board explained to me: ‘Food is our national indulgence.’ The catering at this conference is beyond anything I have ever seen at any conference. Expect most of us to come home with extra pounds put on!

And this was just for mid-morning tea break!

iPRES2011 off to a tropical rain forest start (1)

2011-11-01T14:27:00.001+01:00

This year’s international conference on preservation of digital objects, aka iPRES2011, kicked off today with two tutorials and a welcoming reception in a tropical rain forest on the fifth floor of the National Library of Singapore’s imposing building. Some 170 delegates are expected – some new faces, some familiar faces (in the photo at right those in the know may recognise Andreas Raupert (back), Angela Dappert, two Czech colleagues, Paul Wheatley, and behind him Cal Lee and Adrian Brown.

Considering the size of Singapore (small), the building of the national library is huge (picture below). And I am not kidding about the rain (city) forest – the library organizes special garden walks through the tropical gardens scattered throughout the building.

As for the rain: thunderstorms flood the city like clockwork every afternoon between 2 pm and 5 pm. But the city is prepared: almost every street has an arcade over the sidewalk, so it is only for street crossing that you need your umbrella. And when you enter a store, a plastic umbrella bag is on offer to keep the floors from getting wet.

We are in the modern part of town, with American-style malls, Starbucks and McDonalds, but a mere 15 minute subway ride away there is little India, and in another direction Chinatown, and in another direction again there is the Arab quarter. The government buildings, street names, school uniforms and orderly queues are colonial British. The Chinese (Taoist) temple stands shoulder to shoulder with the Hindu temple, which leans against .. a concrete modern Ibis hotel.

Forgive me the touristy chitchat; this was most amazing jetlag (have-to-stay-awake-at-all-costs-after-flying-in-at-5.40AM) day possible. From tomorrow on you’ll get all the serious stuff from me again.

Emulation is becoming a viable option (#KEEP) (1)

2011-10-27T19:41:00.001+02:00

Just about everybody in our community has heard of emulation as an alternative preservation strategy to migration. But emulation is mostly talked about -- and in terms that are not very encouraging: emulation is rumoured to be very difficult to do, to require lots of expert knowledge and to be really expensive. On 26-27 October the road show of the European KEEP project came to the KB in The Hague and some 45 attendees were given the opportunity to learn more about emulation and test the tools developed by KEEP.

The resulting workshop (led by Jeffrey van der Hoeven, KB, at right) was very much worthwhile and enjoyable, with almost the entire KEEP project team sharing its knowledge. For all those interested in the deep technical details, I refer you to the KEEP website where all the slides will appear soon. For those of you who would like a broader overview, here are my experiences from the last few days (with a selection of slides from the presenters; thanks, KEEP team, for allowing me to use those).

The audience included representatives from libraries, archives, a technical university, two The Hague-based criminal courts, a criminal information service, a film museum and a museum for media art.

What emulation is

In my general introduction to the workshop (what is the problem, etc.), I gave a definition for dummies that was not contested by the experts in the room, so I dare repeat it here. Our problem is that digital objects really only work on the hardware/software combination on which they were made. But these platforms change quickly. Migration is a strategy whereby you change the digital object to make it run on a newer platform. In the long term this is risky, because migrations are never 100%. There are always little changes. Until one day the object no longer is what it is supposed to be. Plus: it only works for a limited number of relatively simple file formats (text, images).

An alternative is emulation. You leave the digital object as it is, but you build software to make the new hardware/software combination pretend it is the old combination.

One can imagine this requires a lot of technical expertise about the old platform and the new platform and this for every possible file format around, of which there are thousands, plus all the dependencies within more complex objects such as websites.

Hence, it does not seem very practical. But when it works, you get a more authentic look and feel than with migration. Two participants in the project come from the computer game industry (Computer Spielemuseum and the European Games Developer Federation, EGDF), where look and feel is everything. Quite a few single emulator programmes have been developed already and new ones are developed all the time. But these are just pieces of the puzzle.

Emulation according to KEEP in four basic steps

Bram Lohman of Tessella summed up emulation according to KEEP in four basic steps, and I will let his slides speak for themselves:

The KEEP approach

The KEEP project wants to make emulation more practical by building a number of black boxes that take care of the most technical aspects of emulation; in this slide they are the green boxes.

The Transfer Tools Framework is your assistant during the ingest phase. It helps you get content from different media (floppy disk, cd-rom, cassette, etc.) and store it into your digital archive.
At the access stage, there is the Emulation Framework, that analyses the digital object, finds the proper emulator software and delivers everything you need to run the object in a new computer environment.
The last building block is the KEEP Virtual Machine. This is the most daring goal of the project, and you can see in the slide that it is only pencilled in for now. Emulators themselves are pieces of software which become obsolete as hardware platforms change. The KEEP Virtual Machine is a piece of software that is so basic and universal that it will ‘port’ into many hardware platforms and thus make emulator software much less prone to technology changes. This is the ‘portability’ part of the KEEP acronym.

A special component of the Emulation Framework is TOTEM, which helps users find technical information about the digital object to be emulated in various available databases such as PRONOM (see earlier post):

What emerged from the workshop

The attendees at the workshop were impressed by the work of the KEEP project. Jacob Taekema of the Amsterdam Municipal Archives (previously Rotterdam) concluded at the end of the two-day event that he could now see emulation developing into a really viable option. Aminata Kpewa and her colleagues of the Special Court of Sierra Leone were really happy to discover a network of people and a tool framework that could finally help them sort out the mess of obsolete databases in their archives.

The important accomplishment of the KEEP approach is that it takes care of a lot of the most technical, expert work (finding the best emulator, configuring the original environment), thus making emulation more accessible for less technically expert staff. The good news is also that the framework can be included into any existing preservation solution – Safety Deposit Box producer Tessella is one of the partners in the project. In addition, all of the work of KEEP is available in open source.

Representatives of the KEEP team discussing archival problems with representatives of the Special Court for Sierra Leone – and making plans for on-site help.

But the KEEP team itself readily admits that much, much work still needs to be done:

The framework now works for a limited number of platforms and file formats (slide at right), and, obviously, many more should be added to really make the framework practicable. As yet the framework cannot yet handle complex objects, although the emulation strategy is considered the most promising for those. It is a matter of finding the resources to put in lots of more R&D work.
At the ingest point (the tool transfer framework), the approach still depends on the availability of all sorts of old hardware to read obsolescent media carriers (floppy disks, tapes, etc.). Such devices are being collected by private individuals and sometimes by computer museums, but it was suggested that perhaps we need a more structured approach to safeguarding these essential machines and the knowledge about how to use them (nationally, perhaps, such as within the Dutch Digital Preservation Coalition?)
Whoever wants to work with an emulated object will have to know how that programme worked, what the commands were, etc. Thus we need to collect and keep as many manuals and specifications as possible. KEEP is building a knowledge base.
The KEEP project itself will end in February 2012. But much work still needs to be done. The KEEP team are doing everything they can to transfer the project to an organization such as the Open Planets Foundation (OPF), which can continue to develop the approach. Let us hope that this transfer is successful and more resources will become available to develop emulation as a strategy, e.g. from the European Community.

Digital preservation stumbling block: copyright law

A major stumbling block for KEEP project, for emulation, and, indeed, for all of digital preservation, is copyright legislation. David Anderson of the University of Portsmouth took upon himself the ungrateful task of finding out about possible legal constraints, and, unfortunately, he found (too) many. In fact, the deliverables of the KEEP project itself will be handed over to the European Commission in loose components because of copyright restrictions. Which is, when you come to think of it, pretty ridiculous. I will write a separate blog post about this important subject and David’s work soon.

More KEEP workshops planned

If this post has wetted your appetite for emulation, you are most welcome to attend one of the forthcoming KEEP national workshops: 10 November Zagreb, 29 November Rome, 24 January Cardiff. Details on the website.

Alexander Fernandez on behalf of the European Games Developer Federation EGDF: ‘Involve game developers in preserving their work and building the emulation knowledge base.’

(For those who question the need to preserve games: they are part of our cultural heritage and they play a growing role in education. Plus: there is a keen and creative community out there developing those games, and they may well help projects such as KEEP to develop means to preserve other complex objects.)

PS: Not to complicate matters too much, and also because it was not mentioned during the workshop, I did not speak, in this blog post, of a third preservation strategy emerging from the CASPAR project, the ‘representation information network’; everything about that in an earlier post.

Are we battling risks or adding value? (#DPS2011) (4)

2011-10-24T16:44:00.000+02:00

Despite all our efforts, the digital preservation community is still having a hard time “selling” its activities to funders and to society at large. As Bohdana Stoklasova of the Czech National Library said at the Goportis Digital Preservation Summit: our work is unglamorous and undervalued.

Bohdana Stoklasova making sure that the funders of the Czech National Library’s digital projects get their credit’s worth – as specifically required by funding deals.

Recently, the argument has been raised that we should stop describing our work in terms of ‘risk’ and ‘loss’ and ‘threats’, but rather speak in terms of ‘adding value’. During the panel discussion and coffee breaks at the Goportis Digital Preservation Summit, however, the consensus was that bad news sells better than good news. By suggesting that we organize a little data disaster I may have gone a little too far, but to my mind risk management is still the best approach to thinking about digital preservation.

At the conference, Angela Dappert (previously British Library, now at the UK Digital Preservation Coalition) made the case for the risk management approach. In her view, risk management is central to all digital preservation activities. ‘Risk’, Angela began, ‘is uncertainty of outcome.’ It is something that might happen, and if it happens, it will become an issue’ (a problem). Risk management is about preventing risks from becoming issues. In that sense preservation is proactive whereas conservation is usually reactive (something has happened).

Below are some slides from Angela’s presentation (thanks for these, Angela); if you are interested in the subject be sure to check out the full presentation at http://www.digitalpreservationsummit.de/presentations.html.

It is important to note that not all risks need to be addressed immediately. It is a matter of balancing the risks against the available means to combat them. In some cases one may wish to accept a risk (e.g., when the collection is not essential for the organization, or when it concerns masters of digitized collections for which the analogue original is still available):

Angela gave an overview of available tools for risk management, including the most well-known framework, Drambora, and a new kid on the block in which Angela is involved, the Timbus project.

Just try and manage the risk of losing cultural heritage in a city like Hamburg …

An encyclopedia of file format information? (#DPS2011) (3)

2011-10-22T11:05:00.001+02:00

Ingest being as complicated as it is (see previous post), let alone what comes after …, would it not be a terrific help if there was an encyclopedia where the best knowledge of the community was continually being assembled, structured and made accessible for all of us to work with? An encyclopedia telling us, e.g., what .pdf is, what hardware/software combination we need to render it, how the computer can recognize it, what versions and variations are known, what complications are known and how to deal with those, and, last but not least, what our colleagues’ experiences are with migrating or emulating this file format? Answers to the question: what is the best preservation strategy for this file format?

Would not that be heaven on digital preservation earth?

Format registry ‘ecosystem’ workshop at Goportis DPS2011.

Well, in fact, the community has been working on such an encyclopedia (more commonly known as a ‘file format registry’) for quite a while now. PRONOM of the UK National Archives was considered a great start, but the upkeep of the system was just too much work for one institution. Then came initiatives such as the Global Digital Format Registry (GDFR) and the Unified Digital Format Registry (UDFR) – which is, as yet, in the planning stage.

The Goportis Digital Preservation Summit organized a workshop on this issue, led by Bill Roberts of the Dutch National Archives. In his introduction he listed the issues involved in setting up and maintaining such a registry: if the work is too much for one or a couple of organizations and we go for a decentralized approach (Wikipedia-style): Whom Do We Trust? How do we know that the information in the registry is reliable?

Workshop leader Bill Roberts: ‘The most common preservation strategy I know of is “stick it in the archive for now”.’

The workshop discussions clearly demonstrated that there is a lot that is difficult to agree upon across the (broad) community. The question: ‘Do we even have an standard for a file format?’ prompted laughter from the workshop attendees.

But some undaunted souls can always be found to give a good idea a new chance. Bill Roberts reported on such an idea, arisen within the Open Planets Foundation. The idea is to set up a ‘Registry Ecosystem’. The objectives are:

And the approach is:

The workshop attendees supported the idea and agreed on a number of issues:

if possible find and mobilize the talent of ‘lonely geeks’
include as much prior knowledge as possible (although vendors such as Microsoft will be extremely frustrating partners to work with)
there should be common core data models
a test corpus is very much needed
the scope of the ecosystem should be broad and include such things as preservation policy procedures and software tools.

On the spot a highly qualified working group was set up to work on the registry ecosystem (including Bill Roberts, Leo Konstantelos of Glasgow, David Anderson and Janet Delve from Portsmouth, Adrian Brown of the Parliamentary Archives and Michelle Lindlar of Goportis; some other names were mentioned, but have yet to be approached). The working group is to meet virtually and deliver results within three months.

And, to quote Bill’s last slide: ‘The most important thing: Lots of people out there have pieces of the puzzle; we must encourage and enable them to share.’

PS: Leo Konstantelos suggested dropping the term ‘file format registry’, because the scope of what is described here is much larger. He suggested ‘technical registry’.

See also Bill Roberts´own blog post at the Open Planets Foundation website.

On ingest, or: “receiving” is a complex word (#DPS2011) (2)

2011-10-21T11:15:00.001+02:00

The second day of the Goportis Digital Preservation Summit was all about ingest, or 'receiving' content in your repository or archive. In his keynote, Seamus Ross, formerly of the digital preservation taskforce at Glasgow and now at the University of Toronto, was quick to dispell any illusions that 'receiving' is an easy thing to do.

Interoperability troubles – from Open Office to Powerpoint – causing stress before Ross’s presentation (Nina Stoffers, left, Ross right).

Ross’s presentation was a complete Ingest 101 course, and so, I will let his slides tell most of the story.

Ingest is about “receiving” content from producers:

Ideally, we would want to create a work flow that is consistent, error-free, well-documented, in accordance with our organization’s policies:

Preferably, you know who the producers of your content are and you start negotiating with them so that they deliver the best possible quality. However, keep in mind that whatever makes our lives easier, is most likely to make the producer’s lives more difficult. That is where the bargaining begins. Ideally, you get this:

But in practice, this is most likely what you get most of the time:

Seamus Ross: ‘Most of the work we do during ingest is about fixing all these errors, is about compensating for the communication failures between producers and archives.’ So, what do we do?

How do we do all this? Ross: “You are a craftsman. You must accept that your tools are blunt.” Present tools for identification and validation are far from perfect. They still require a lot of manual work and the people who work with them must be very knowledgeable. Also, “You may be sure that producers will deliver error-laden stuff, no matter how well you train them.”

Ross stressed that policies are an essential part of the equation:

But even policies cannot guarantee smooth sailing:

Having said that, Ross did have a list of useful reference material for the audience, including an instructive case study at http://artefactual.com/wiki/index.php. Check out his slides when the complete set comes available via the event website. He also mentioned the useful NDIIPP tools and services directory at http://www.digitalpreservation.gov/partners/resources/tools/index.html and the Cairo Tools Survey. But remember Ross’s warning that working with these tools requires quite a bit of prior knowledge.

Seamus Ross: “But do not worry too much – digital archaeology will play a role in the future.”

During the Q&A Adam Farquhar of the British Library offered his more optimistic view of the state of digital preservation (see yesterday’s post). Ross’s reply: ‘But that concerns only a narrow range of object types.’ Databases, for instance, are still a very real problem to deal with.

Goportis co-organizer Yvonne Friese checking #DPS2011 tweets.

More good stuff from this densely packed conference in the next few days. About whether OAIS is still helpful, about tools, about file format registries. And about thinking before you act, the New Zealand version.

Digital Preservation Summit (1): how far have we come?

2011-10-19T19:19:00.003+02:00

We have gathered in cold, rainy, windy Hamburg today and tomorrow for the Goportis Digital Preservation Summit (#DPS2011). So the sunny note on which Adam Farquhar of the British Library started off the conference was quite welcome - except that some of us (including the undersigned) saw a few more clouds in the sky than Adam. A matter of the cup being half full or half empty?

Adam Farquhar (left) with Angela Dappert (DPC)

Farquhar said our progress to date is 'pretty encouraging'. 'Digital preservation has become business as usual,' he said, 'for large memory institutions.' Now that I reread my notes, the addition about large memory institutions is probably crucial to Adam's argument, but what stuck in my mind, and also in the mind of Steve Knight from the National Library of New Zealand (both of us presented during the day), is that in our opinion, digital preservation is still quite a long way off from being business as usual for most of the stakeholders - including data producers, funders, and all but the very largest memory institutions.

Adam also said that we are doing digital preservation 'at a substantial scale', citing recent BL projects involving the migration of millions of objects - whereas in my community (the Netherlands Coalition for Digital Preservation) I hear much grumbling about the (lack of) scalability of the tools we have at our disposal at present.

Fortunately (I mean in terms of agreeing on issues), Adam also saw a number of challenges:

changes in digital materials (flash, social media with short urls)
content in context - when a publication is commented upon over the years, it changes
dynamic content - complex objects such as 3D interactive views of crystals; html5 (which incorporates javascript elements)
a lack of skills in memory institutions - which is getting worse because of the budget cuts.

And I agree with all those (and surmise that Steve Knight will too).

At the end of his keynote, Adam Farquhar said two things:

Do not wait until we know everything to get it right, but do whatever you can now
Within our community, we need to become more honest about what works and what does not. That is the only path to true learning.

To which I can only say: hear! hear!

There is much more good stuff to report from this conference (including instructive disagreements between presenters), but this time live or even semi-live blogging is difficult because I am presenting and moderating myself - plus: this conference is very well organised and the audience does not get any (boring) time off for blogging. Also, it is a 9 to 6 programme, and your blogger needs time to eat and sleep. So, dear readers, I must ask for a little patience. But I assure you: ALL shall be revealed ... and in a matter of days even, because then we have the KEEP workshop coming up, and iPRES ...

On a more practical note: powerpoint and laptops are wonderful inventions, but can somebody PLEASE come up with a solution whereby every presenter is visible to the audience?
(Thanks to Natalie Walters for allowing me to use this image)

Is emulation something for you (2)

2011-10-15T13:19:00.000+02:00

If you tried to register for the The Hague Keep workshop (26-27 October, see last post) but were told the workshop was sold out, you may want to try again. Twenty more seats have been made available.

Is emulation something for you?

2011-10-08T16:18:00.001+02:00

In their efforts to come up with catchy acronyms, project managers sometimes think of wonderfully sounding names that, however, tell you too little about what is really going on. The European KEEP project is a case in point to me: KEEP stands for ‘Keeping Emulation Environments Portable’. Perhaps it is just my slow brain and/or the often fuzzy official project language, but for some reason I kept getting visions of shopping bags and briefcases …

On the occasion of the KEEP road show, which is coming to The Hague on 26-27 October (and to Zagreb, 9-11 November, Rome: 29-30 November), I asked KB colleague and KEEP participant Jeffrey van der Hoeven (at left, emulating KEEP user satisfaction) to explain it to me. Here is my version of what he told me:

The most well-known method to deal with software and hardware obsolescence is migration: you change the bits and the bytes of a digital object to make them work on a new platform. However, migration turns out to be not at all as risk-free as we would hope. Plus: it does not work for complex objects such as video games, websites, etc.. An alternative is emulation: you do not change the bits, but write software to make a new computer function as if it were an (old) computer. This means writing emulators for every possible combination (which is a lot of expensive R&D work), but if it works, there are fewer risks involved than with migration.

However, working with emulators is not for dummies. It is technically challenging work for specialists. The KEEP project developed an ‘emulation framework’ that takes care of that. It automatically selects the right emulator and configures the software required to render the object. That sounds quite handy.

Now, what about the ‘portability’? Emulators themselves are pieces of software that become obsolete over time. Therefore, KEEP is developing a KEEP ‘virtual machine’ – that will allow for execution of any software on any platform at any time.

Does this sound too good to be true? Come to The Hague (or Zagreb or Rome) and find out for yourself. Be sure to bring some old obsolete floppies with you to test the systems hands-on.

'The Really Foolproof Solution for Digital Preservation ....

2011-09-21T19:13:00.002+02:00

... is ... money ... enough of it, and for an indefinite period. If this cannot be guaranteed, the rest of this book will be essential for you', writes David Giaretta (Alliance for Permanent Access/STFC) at the beginning of his new book. I am not giving you the full title yet, because I am afraid that it might scare you off. Certainly, if I had just seen the title without knowing the book or the background, I would have thought that this book is not for me. And that would have been a mistake, because, although I am not a technical expert, I am thoroughly enjoying it and learning a lot in the process. And so may you, especially if you are one of the many readers that read my post about Giaretta's workshop at the LIBER conference in June. This book gives you much more of where that came from.

David Giaretta with his book.

Contrary to what the title would have you expect (allright then, here it is: Advanced Digital Preservation), there is lots of good solid basic digital preservation information in this book. OAIS for example. Everybody has seen the functional model's diagramme, but how many of us have actually read the standard and understood the philosophy behind it? Giaretta guides us through it in detail. And in pleasantly understandable language. Plus: the importance attached to a clear definition of the designated community in the OAIS model - this is crucial to what follows.

Inevitably, things do get technical in the course of the book; after all, if we did not have technical problems we would not have a digital preservation problem, but the not-too-technical-reader is always warned in good time that this perhaps is a section or a chapter to be skipped. Yet the essence of Giaretta's theory is worth noting for everybody. In his view, migration and emulation, our most well-known preservation strategies, are perhaps good enough for simple objects (PDFs, tiffs, jpegs), but are inadequate for many complex objects which can be found among research data, Giaretta's main focus (hence the title Advanced Digital Preservation). No-one will doubt that scientific research often generates very difficult objects to preserve - they are complex, dynamic, often non-renderable, and so forth.

If you do not preserve research data, this book is still important for you, because other sectors (cultural heritage, archives) that started out with simple objects will increasingly be faced with more complex varieties, as content producers are discovering the extra possibilities and putting them to good use.

The 'Droste' effect

To tackle the problems of more complex objects, Giaretta, and the CASPAR project team, developed a theory around the Representation Information Network. Simply put: a (or rather: any) data object is nothing but ones and zeros; they must be accompanied by representation information in the metadata to tell you what you need to 'independently interpret, understand and use' (in OAIS language) the data object. The data object can be a single file or multiple files, and the representation information can be anything from a scribbled handwritten note to a complex machine readable formal description (pp 17 ff). In Giaretta's more accessible advocacy language: you have something that is unfamiliar (ones and zeros) and the representation information gives you what you need to make it familiar. However, representation information is not a straight-forward thing: it is more like a set of Russian babushka dolls (in Dutch we would refer to the 'Droste effect', after the cacao nurse that serves from a cacao tin that has her own image on it which serves from a cacao tin that ...): a Word document cannot be understood with Microsoft Office software alone, you will need the operating system, and the programming language, and so forth and so forth. You will need every dictionary, every definition, every standard, every specification that is used somewhere along the line - until you connect with the knowledge base of your designated community, that is: you make the connection with what your designated community has at its disposal in terms of software, hardware and knowledge to work with those.

Over time, as technology evolves, the 'unfamiliarity' of a digital object will increase and the the amount of representation information needed to connect with your designated community will increase with it. Our job is to manage that process and make sure there is always enough representation information to connect with our users. Preferably in an automated way, because there is no way we can do this manually (unless of course we have an truly endless flow of money ...).

Giaretta and his CASPAR team argue that this is the only method that will work for all digital objects, no matter how simple or complicated. The trick will of course be to build that automated process that will keep our digital objects "fresh".

More research is needed to turn this theory into something practical. Meanwhile there is this book to enjoy and learn from, including excursions into non-technical territory: repository audits, preservation chains, business models, stakeholders analysis, and more. Giaretta's fluid style of writing, the many cross-references, summaries, and warning signs have enabled me to delve deeper into the technical level than I thought possible. And I am still learning.

What I would like to see next, however, is more interaction between what Giaretta is developing and what the Open Planets Foundation led by Bram van der Werf (and the related SCAPE project) is working on. What would be really great to have for the community is their joint views on what works and what does not - and in which circumstances, and the direction R&D should take. How about it, gentlemen?

David Giaretta [et al.], Advanced Digital Preservation (Springer, 2011, isbn 978-3-642-16808-6, €99.95).

Linked data (3): antwoord op lezersvragen

2011-08-24T12:32:00.000+02:00

Het is hoog tijd dat deze blog weer uit zijn zomerslaap ontwaakt, en dat doen we met een thema dat blijkens de vragen die binnenkomen actueel blijft: linked data. Eerder publiceerden we Irene Haslingers inleiding in het thema (Linked Data: wat is dat nu eigenlijk precies?). Daarop kwamen vragen, o.a. van Genoveva Leppaart. René van der Ark van de KB, die onderzoek doet op dit gebied, geeft antwoord:

Overzicht van datasets die al opengesteld en gelinkt zijn – stand september 2010 (bron: http://wiki.dbpedia.org/About)

[NB: De software van blogger weigert dienst als we punthaken gebruiken. In onderstaande voorbeelden hebben we die vervangen door [punthaak open] en [punthaak sluit]. Minder fraai, maar dan komt de tekst tenminste op je scherm.]

Voor ik [=René] de onderstaande vragen probeer te beantwoorden moet ik eerst in zijn algemeenheid zeggen dat de eerste stap voor het bereiken van linked data bestaat uit het openstellen van bestaande data voor de buitenwereld via het internet. Het daadwerkelijke linken kan plaats gaan vinden wanneer genoeg partijen hun data open hebben gesteld (bij voorkeur in een veelgebruikt formaat zoals RDF/XML. Dit kan plaatsvinden met een simpele conversie en hoeft dus geenszins de brondata aan te tasten). Hier zit zo onderhand schot in, maar op alle vier de vragen hieronder is nog geen eenduidig antwoord te geven, omdat de discussie hierover nog volop gevoerd wordt.

1. Is het mogelijk Linked Data ook te gebruiken voor de inhoudelijke ontsluiting van bijv. artikelen uit vaktijdschriften, rapporten, boeken, krantenartikelen e.d.? Zo ja, kan je uitleggen hoe dat werkt (op hoofdlijnen)

Ja. Je kunt een open vocabulaire/thesaurus op het web gebruiken om bronnen mee te ontsluiten/verrijken. Het meest bekende voorbeeld, prominent aanwezig in de linked data ‘cloud’ is DBpedia (de wikipedia in RDF/XML formaat), dus deze zal ik voor het gemak als voorbeeld gebruiken. De meest eenvoudige manier om een object te ontsluiten is door de URL van een concept uit (bijvoorbeeld) de DBpedia toe te voegen aan de metadata van het object. Dat zou er dan ongeveer zo uit zien:

[punthaak open]dc:author rdf:resource=”http://dbpedia.org/resource/Albert_Einstein”[punthaak sluit]Albert Einstein[punthaak open]/dc:author[punthaak sluit]

De url waarnaar verwezen wordt in het attribuut blok ‘rdf:resource’ is een verwijzing naar een open data-bron waarmee je effectief ‘linked data’ hebt gecreëerd.

Het idee is dat wanneer je vervolgens de metadata van dit object als open data op het web beschikbaar stelt, andere partijen jouw object geschreven door Albert Einstein kunnen vinden in jouw data, omdat je het hebt ontsloten met het concept Albert Einstein van DBPedia. Dat werkt natuurlijk ook andersom.

Ben je al in het bezit van een eigen thesaurus waarmee objecten ontsloten zijn, dan kan het de investering waard zijn om deze thesaurus te ‘mappen’ (of ‘alignen’) met andere open data-bronnen zoals DBpedia, of, voor persoonsnamen VIAF, of, voor plaatsnamen (wereldwijd), http://geonames.org.

Dit betekent dat je een indirecte link hebt gemaakt (object - ontsloten met dc:author 123456 - geefMeDeMappingVan(123456) - dbpedia:Albert_Einstein).

2. Het nadeel lijkt me dat je alleen relaties kan terugvinden die vooraf gemaakt (en dus bedacht) zijn. Hoe bepaal je tevoren wat de gebruiker zal willen weten en dus welke relaties je legt? Hoe ver ga je met het leggen van relaties?

Dit hangt heel erg af van de aard en bruikbaarheid van de relatie. In essentie staat elke ontsluitingsterm al in relatie met een object. Dit drukken we in de semantic web/linked data wereld uit als een ‘triple’:

publicatieID dc:author [punthaak open]http://dbpedia.org/resource/Albert_Einstein[punthaak sluit]

Bovenstaand voorbeeld is in de informatiesector in ieder geval een bruikbare relatie. Wanneer het echter om een relatie tussen concepten gaat, wordt het een stuk lastiger:

[punthaak open]http://dbpedia.org/resource/Albert_Einstein[punthaak sluit] dbpedia-owl:spouse dbpedia:Mileva_Marić

(Albert Einstein - heeft huwelijkspartner - Mileva Marić)

Maar: van wanneer tot wanneer waren ze getrouwd? Had hij meerdere vrouwen? Omdat de structuur atomair is, kun je dit soort aanvullende informatie alleen met meer ‘triples’ vastleggen.

3. Hoe kun je in een database met linked data gegevens zoeken? Is dit werk voor de (informatie)professional of kan de eindgebruiker dat ook?

Hier benoem je één van de moeilijkste problemen met semantische zoekmachines. De gemiddelde eindgebruiker, van leek tot wetenschapper, gaat niet de moeite nemen om een zoekvraag semantisch uit te splitsen naar een query die de computer begrijpt. Bovenstaand zoekvoorbeeld zou er dan versimpeld zo uit komen te zien:

Select ?pub Where {
?pub dc:author ?auth .
?nat skos:broader [punthaak open]http://thesaurus.org/natuurkundigen[punthaak sluit] .
?auth skos:related ?nat .
?auth hasName ‘Albert Einstein’ .
}

Als een eindgebruiker al de vaardigheden bezit om zo’n zoekvraag te formuleren, dan moet de eindgebruiker ook nog genoeg kennis hebben van de inhoud van de database om erin te kunnen zoeken. Er wordt al sinds deze technologie is bedacht, gezocht naar manieren om googleachtige zoekvragen automatisch te vertalen naar een semantische query, maar dit heeft m.i. nog weinig bruikbaars opgeleverd.

Als je echter alleen de dwarsverbanden tussen thesauri gebruikt, kun je met computers wel een hoop voorwerk doen in het uitbreiden van de kennis over een object en het dus beter ontsluiten; zij het met traditionele en niet met semantische zoektechnieken. Linked data levert dus wel degelijk wat op. Een voorbeeld is een archeologische vondst waarvan alleen de plaatsnaam van de vindplaats in de metadata stond. Als die plaatsnaam wordt gekoppeld aan de database van geonames, dan heb je geocoördinaten tot je beschikking en kun je de vindplaats tekenen op google maps - hiervoor heb je geen semantische database nodig.

4. Wie moeten de relaties gaan aanbrengen? Bibliotheken, informatiecentra en archieven, of uitgevers en/of auteurs? Hoe denk je dat dit geregeld gaat worden?

De consensus in de linked data community is dat dit proces een natuurlijk verloop zal krijgen wanneer genoeg partijen hun data openstellen zodat iedereen ermee aan de slag kan. We kunnen niet voorzien in dit stadium welke partijen kwalitatief en/of kwantitatief de beste links zullen gaan opleveren, als het automatisch gebeurt. Hier is een hoop vertrouwen voor nodig en wanneer de kritieke massa dan bereikt zal zijn, is absoluut niet in te schatten. Wel begint duidelijk te worden dat het openstellen van data in andere sectoren interessante nieuwe technologieën kan op leveren: denk aan de brandweer die gegevens van brandveiligheid vrijgeeft, gekoppeld aan de huizenwaarde in kadastergegevens - dit soort applicaties worden nu op grote schaal gemaakt dankzij het bestaan van ‘open data’.

Of dit proces zich überhaupt gaat voordoen in de informatiesector is iets waar we alleen maar naar kunnen gissen, net als of het iets oplevert voor iemand. Maar het is toch een beetje een kwestie van meegaan in de vaart der volkeren in de hoop dat er iets gebeurt.

In de toekomst is het denk ik wel zo dat bibliotheken/informatiecentra/archieven zich kunnen blijven onderscheiden met de kennis van de eigen collectie en door het faciliteren van betere vindbaarheid; het leggen van relaties tussen verschillende collecties is een onderdeel hiervan. Wie de relaties moet gaan leggen en beheren kan ik niet overzien; wel dat alleen mensen betrouwbare relaties kunnen leggen (machines doen het met een betrouwbaarheid van 80%), maar dat er voor die benodigde menskracht vaak geen budget is.

René van der Ark is projectmedewerker Innovatie & Ontwikkeling bij de Koninklijke Bibliotheek, rene.vanderark@kb.nl

How can we prove that digital preservation systems will deliver? (LIBER 5)

2011-07-03T11:29:00.001+02:00

This blog post is about the LIBER2011 workshop with the poorest attendance (14 out of 400 conference participants having a choice between three parallel sessions). Attendance may have been poor, but the subject matter was important and thus I can only conclude that I and others who plead the cause of digital preservation still have a lot of work to do. (Or are the other 386 counting on me blogging about it in sufficient detail ;-)

Why testing?

Over the past 15 years or so we have been building preservation systems and putting our digital collections (or, more precisely, ‘digitally encoded information’) into them. But how do we know that they will deliver? Last month in Tallinn, Michael Seadle called our present systems ‘a leap of faith’ and with Andreas Rauber he pleaded for more testing and more exchanges of testing data (see post).

But what do you test? And how?

That was what David Giaretta’s workshop was about, in the context of the APARSEN project (a major European project with 32 partners) (slides in this post courtesy of David Giaretta).

Giaretta explaining the four phases of APARSEN: Trust, Sustainability, Usability and Access. Testing is part of the trust package.

‘We need more than migration and emulation.’

The most well-known preservation techniques are migration and emulation. The results are tested on the basis of ‘significant properties’: Is the information an organization regards as essential still there after the object has been changed or, alternatively, in the new computer environment that purports to emulate the old computer?

Giaretta asserts that these techniques are useful for some digital objects – and they have a role to play in determining authenticity -, but the techniques do not work for all objects. APARSEN has developed a three-dimensional model to characterize objects technically to be able to determine what tools can be applied:

Which leads to these conclusions:

So, we need other techniques in addition to migration and emulation. Especially if we want to our information to be part of the Global Brain of Linked Open Data Herbert van de Sompel spoke about on Wednesday.

First question: who do we preserve for?

Giaretta has developed a very elegant way of describing what we do all this work for: we have ‘unfamiliar’ stuff (rows of ones and zeros) which we must make ‘familiar’ for people to be able to use it. We must do that now, and we must continue to do it in the future. Over time, the job will become more difficult.

Second question: what do they need to use the object?

What ‘familiar’ means, depends on the context, on a central concept from the OAIS reference model, the ‘designated community’, the user group an institution works for and their knowledge bases. If the target audience is a group of five-year-olds, our rendering techniques must be very sophisticated so the five-year old only has to push a button. If the audience is a group of computer specialists, less help will be needed.

In this view, the representation information which is part of the OAIS is included in the AIP (OAIS term for the archival information package that includes both the object itself and all the extra information needed to process and render it) becomes the focus of testing the systems (see OAIS Information Model). Is everything there that the designated community needs to be able to use the information?

The ‘representation information network’

As we saw above, the representation information varies between designated communities. But it will also change over time. A present-day computer will understand the information ‘this is XML’. But in 2080 XML is perhaps an archaic file format, and the rendering information will have to be much more specific in telling the computer how it can render XML so a human (or machine) can use it. And if the manual for the programme happens to be in PDF, it will need to include the same information about PDF. Discipline-specific information must also be included, such as vocabularies and ontologies. And when the information package contains a series of dates one must be able to determine the time zone, summer or winter time, etcetera.

It is a network, to which new information must be added as time goes on:

This network can be tested. Is all the required information being preserved?

In the past month, APARSEN has been doing a series of test audits in Europe in preparation for the ISO16363 standard which is in the making. The tests were also designed to test prospective auditors. The provisional conclusions are as follows:

most audited organizations do a good job at preserving the bits;
quite a few organizations lack succession plans (what happens to the data when my organization ceases to exist?);
quite a few have not defined their designated communities;
typically, the representation information networks are insufficient or non-existing.

Giaretta concluded:

Plenty of empty chairs … (Photo: Jordi Aguilar)

Here is David’s impressive list of references for those of you who want to know more:

1. CCSDS. (2002), Reference model for an Open Archival Information System (OAIS). Retrieved from: http://public.ccsds.org/publications/archive/650x0b1.pdf

2. OAIS update (at the time of writing under CCSDS review), http://public.ccsds.org/sites/cwe/rids/Lists/CCSDS%206500P11/Attachments/650x0p11.pdf

3. Knight, G., 2008, Framework for the definition of significant properties. Retrieved from http://www.significantproperties.org.uk/documents/wp33-propertiesreport-v1.pdf

4. Wilson, A., 2007, Significant Properties Report. Retrieved from http://www.significantproperties.org.uk/documents/wp22_significant_properties.pdf

5. J. Rothenberg and T. Bikson, 1999, 'Carrying Authentic, Understandable and Usable Digital Records Through Time' report to the Dutch National Archives and Ministry of the Interior. Retrieved from http://www.digitaleduurzaamheid.nl/bibliotheek/docs/final-report_4.pdf

6. M. Hedstrom and C.A. Lee, “Significant properties of digital objects: definitions, applications, implications”, Proceedings of the DLM-Forum 2002. Retrieved from http://ec.europa.eu/transparency/archival_policy/dlm_forum/doc/dlm-proceed2002.pdf

7. Cedars project, http://www.leeds.ac.uk/cedars/

8. Investigating the Significant Properties of Electronic Content over time (InSPECT) http://www.significantproperties.org.uk/

9. The InterPARES project, http://www.interpares.org/

10. Wison, A., 2008, Significant Properties of Digital Objects, presented at “What to preserve? Significant Properties of Digital Objects”. Retrieved from http://www.dpconline.org/docs/events/080407sigpropsWilson.pdf

11. DELOS Digital Preservation Testbed. Retrieved from http://www.ifs.tuwien.ac.at/dp/testbed.html

12. OCLC/RLG Working Group on Preservation Metadata, 2002, Preservation Metadata and the OAIS Information Model, A Metadata Framework to Support the Preservation of Digital Objects. Retrieved from http://www.oclc.org/research/projects/pmwg/pm_framework.pdf

13. Derek Sergeant, 2002, Interpretation of the OAIS Model. Retrieved from http://www.erpanet.org/events/2002/copenhagen/presentations/dmserpanet.ppt

14. CASPAR Access Model, http://www.casparpreserves.eu/Members/cclrc/Deliverables/report-on-oais-access-model/at_download/file especially section 2.

15. Michael Factor, Ealan Henis, Dalit Naor, Simona Rabinovici-Cohen, Petra Reshef, Shahar Ronen, IBM Research Lab in Haifa, Israel and Giovanni Michetti, Maria Guercio, University of Urbino, Authenticity and Provenance in Long Term Digital Preservation: Modelling and Implementation in Preservation Aware Storage, TaPP ’09. First Workshop on the Theory and Practice of Provenance. San Francisco, 23 February 2009, http://www.usenix.org/event/tapp09/tech/full_papers/factor/factor.pdf

16. CASPAR Conceptual Model, http://www.casparpreserves.eu/Members/cclrc/Deliverables/caspar-conceptual-model-phase-1-1/at_download/file

17. Giaretta, D., 2007, The CASPAR Approach to Digital Preservation, The International Journal of Digital Curation, Issue 1, Volume 2, http://www.ijdc.net/index.php/ijdc/article/viewFile/29/18

18. CASPAR – Cultural, Artistic and Scientific knowledge for Preservation, Access and Retrieval. See http://www.casparpreserves.eu

19. Mike Coyne, David Duce, Bob Hopgood, George Mallen, Mike Stapleton. The Significant Properties of Vector Images. JISC report, 27 November 2007. http://www.jisc.ac.uk/media/documents/programmes/preservation/vector_images.pdf

20. Mike Coyne, Mike Stapleton. The Significant Properties of Moving Images. JISC report, 26 March 2008. http://www.jisc.ac.uk/media/documents/programmes/preservation/spmovimages_report.pdf

21. Brian Matthews, Brian McIlwrath, David Giaretta, Esther Conway. The Significant Properties of Software: A Study. JISC report, March 2008 http://www.jisc.ac.uk/media/documents/programmes/preservation/spsoftware_report_redacted.pdf

22. Kevin Ashley, Richard Davis, Ed Pinsent. Significant Properties of E-learning Objects. JISC report, March 2008. http://www.jisc.ac.uk/media/documents/programmes/preservation/spelos_report.pdf

23. PARADIGM project, Workbook on Digital Private Papers. http://www.paradigm.ac.uk/workbook/preservation-strategies/file-properties.html

Setting priorities: value shift from printed to digital … and vice versa (LIBER 4)

2011-07-01T17:38:00.001+02:00

‘Digital and print is an and and proposition for libraries’, said Graham Jefcoate during the special collections session. But the budgets have not increased. So how should we allocate resources? How should we prioritize?

The Dutch KB has been working on a model that takes an integral view at printed and digital collections. The model can help decide what to spend our money on. At LIBER it was presented by Sophie Ham (photo left) and Tanja de Boer. As it has not been published yet, I gladly offer it here with some detail, because I think it can really help libraries make tough decisions (slides courtesy of Sophie Ham).

The model rates the value of (parts of) collections to determine which ones should be conserved or preserved with priority. Here are the rating criteria:

Primary criteria:

Secondary criteria:

And this is how the process works:

Multiplying primary and secondary criteria might look like this (just an example):

Sophie gave two extreme examples of how such a value assessment can turn out. This example concerns printed versions. Obviously, newspapers score higher on informational value and medieval manuscripts score higher on uniqueness and historic value.

	Newspapers	Medieval manuscripts
Informational value	8	5
Aesthetic value	2	9
Historic value	3	9
Use	8	3
Uniqueness	4	10
Condition	2	8
TOTAL	27	44

What happens when these collections are digitized? Some values are transferred to the digital copy (e.g., much of the use and informational value for newspapers), but other values cannot be transferred (e.g., the uniqueness of a medieval manuscript). As Claudia Fabian of the Bayerische Staatsbibliothek demonstrated, the use value of a physical object may even increase when it is digitized, because more people become aware of its existence and become interested.

And here is the end result of this (extreme) example:

As you can see, the value of the physical medieval manuscript has remained unchanged, whereas some of the value of the printed newspaper collection has been transferred to the digital copy, especially the use value; the KB’s online newspaper database is in great demand by users of all kinds in the Netherlands.

Discussing the KB model, from the left: Sophie Ham, Claudia Fabian and workshop chair Graham Jefcoate.

So, if push comes to shove, and painful decisions have to be made about whether to build new stacks for physical newspapers or invest in expanding the e-Depot that holds the digital newspapers, this (limited) analysis clearly points in the direction of investing in the e-Depot and perhaps deciding to keep only representative selections of printed newspapers.

Obviously, lots of questions remain. At what granularity should one assess collections? How can one make the assessments as objective as possible?, etc. But it is a promising beginning. If you want to know more or contribute to developing the model, please get in touch with sophie.ham@kb.nl.

KB colleagues at Sophie’s presentation: from the left, Els van Eijck van Heslinga, Lotte Wilms, Lieke Ploeger, Victor-Jan Vos.

Value shift: cool conference bag being put to alternative use. Those with short legs thank the sponsors for the abundance of printed promotional material.

Global Web of Data depends on machine-actionable XML (LIBER 3)

2011-06-30T19:52:00.001+02:00

Two inspired keynotes today about the vast new possibilities that machine-readable data – or more precisely: data that machines can act on - open up for the advancement of science. There is so much (digital) data out there that no human can comprehend it all. Fortunately, we have or are developing tireless machines that can publish, merge, search, reason, predict and integrate information. They can establish relationships between fields of science which have never even contemplated getting together and make way for new cross-disciplinary work.

Herbert van de Sompel, at right, with yesterday’s keynote speaker Rick Luce.

It was, of course, Herbert van de Sompel of Los Alamos who treated the audience of 400 research librarians to a peek into this fascinating world of emerging research possibilities (slides available from slideshare). All based on some rather basic building blocks:

Enthousiastically Van de Sompel reviewed some of the projects to make all of this possible, starting with his own OAI Object Reuse and Exchange, Open Annotation, and Memento, and then on to other developments, such as the ‘nano publication’, the smallest entity of information that can be searched, merged, read, etc. etc. by machines – a somewhat extended version of an RDF triple. And what to think of Executable Papers – articles that include the software and underlying data so that the reader can repeat the original experiments and draw his own conclusions.

Mind boggling! Van de Sompel explained that this is what we need to make it all happen, ‘and it irks me that we have that at our fingertips’:

open access to all data
permissive (i.e., non-restrictive) creative commons licences
money to pay for these tools
persistence in identifying the objects – and that is still a challenge (see last week’s post)

From a digital preservation viewpoint, however, there is a complication. As Alma Swan of Enabling Open Scholarship explained a few hours later, it does not work with PDF. PDF was developed for humans to read. Machines cannot read PDF, they need XML.

Alma showed that universities are building digital repositories at great speed – there are now almost 2000 of them. But what are we filling them with? Mainly PDF’s … Because they are nice and robust from a preservation viewpoint. And humans can read them.

Alma Swan (seated) awaiting her turn to speak with session chair Bas Savenije.

There is good news as well. As Alma pointed out, digital repositories are attracting new users, mostly users from outside the university who do not have access to licensed digital content from major publishers. Companies, for instance, and private citizens who are making use of the new digital possibilities and getting involved in scientific efforts, such as these:

However, much of the material these new users are interested in, is not available in open access, and thus cannot be used by either humans or machines.

So what are we preserving all of the stuff for?

I’m going to sleep on that one …

Yours truly will spare no effort to tell you everything; here is my paparazzi shot at the VIP room: LIBER President Paul Ayris (left) and Executive Director Wouter Schallier.

More (digital) wake-up calls for academic libraries (LIBER 2)

2011-06-29T18:01:00.001+02:00

Today it was Rick Luce (Emory University, US) who had the (questionable) honour of issuing a wake-up call to research libraries. This time the topic was not cultural heritage (see yesterday’s post), but the core business of academic libraries: serving researchers and the scientific research process. Check out his slides when they become available on the LIBER website. It is a dazzling summary of all the changes taking place in the sciences: zetabytes of data; dynamic, complex data objects that require management; communities and data flows becoming much more important than static library collections, etc. etc. Luce’s warning: somebody will develop the services the new researcher needs. If the library does not develop those, there is no future for the research library. Luce called for a fundamental transformation process that will affect every aspect of the ‘library’ business.

In this vision, the role of the ‘library’ is to deliver a layer of middleware between the scientific process and IT infrastructure.

Luce’s advice for libraries on how to bring all this about: ‘Radical Cooperation’:

Such change, of course, does not come easy. Disruption of set patterns makes people nervous – but instability also contributes to disruptive innovation:

Luce warned that ‘Culture will have strategy for breakfast every time.’ It takes years to turn the culture of an organization around.

At the end of this inspiring session, Kurt de Belder of Leiden University asked a crucial question: ‘Are libraries in fact the type of organizations that can make such drastic changes?’ In his response, Luce implied that some will make the change successfully – those who do not will be out of business by 2020. One of those libraries might be that of the librarian who asked how to make researchers deliver the metadata that the library needs during a data workshop this morning. Or was the wake-up call clear enough?

Norbert Lossau of Gottingen asked for advice on how to convince the University Board and staff of all this. Luce’s advice: ‘Look for small victories; do it step by step; work with early adopters within your staff.’

Will research libraries have enough time to get all that done by 2020?

Neelie Kroes of the European Commission addressing the conference by video. Her agenda: a) open access; b) open data (all public-sector information); c) digital culture (‘By 2025 all European cultural heritage should be digitized and available through Europeana’)

Tomorrow will be too late – (born) digital in library special collections (LIBER 1)

2011-06-28T20:43:00.001+02:00

“Heritage collections in the digital future” was the revealing title of the first workshop I attended at the annual conference of LIBER, the Association of European Research Libraries, in Barcelona. The title was immediately attacked by Marco de Niet of Digital Heritage Netherlands: ‘Digital Future? Come on! The digital reality is today.’

Was it really necessary for De Niet to talk in terms of ‘Shame on you!’, I wondered when he started his presentation? Have we not reached the stage now where we all know about the digital present and are making plans for it even if we cannot act upon them immediately?

Unfortunately, the answer must be no. Starting with Ivan Boserup’s summary of the findings of a recent poll amongst LIBER libraries, followed by statistics gathered by De Niet himself (from projects such as Enumerate), and on through Jackie Dooley’s summary of recent OCLC research (Taking Our Pulse: the OCLC Research Survey of Special Collections and Archives) – the evidence is overwhelming that (research) libraries are still pretty much functioning within an analogue paradigm. This is not to say that they are all still about lending physical books. Of course not, quite a few libraries have digitized collections and provide online access. But their digitization efforts mostly lack strategic planning, access is still mostly provided in a controlled way (for a limited group of users), preservation issues are still not being addressed adequately, and born-digital material (including audiovisual content) is blatently missing from collections and collection plans.

‘Shocking’, De Niet called these findings, and I second that. De Niet summarized the changing value propositions as follows:

Most libraries are still in their comfort zone, that of digital in a controlled network. According to De Niet, libraries cannot afford to stay there. If they do, their role in the information business become insignificant. The key factor is still the libraries’ desire to be in control, according to De Niet. ‘Libraries have to let go.’

De Niet sees opportunities for libraries, but not if they stick to their traditional values.

During the Q&A a library director remarked that he did not find this presentation particularly helpful. He thought it was rather confusing …… Quod erat demonstrandum?

Spain is suffering from a heat wave; fortunately the conference pack includes a sponsored fan – Jasmine Honculada of WIPO was one of the first to discover how that funny plastic object could be put to excellent use.

(There is more to tell about this session – more to follow soon.)

Persistent identifiers: policy and ‘will’ vital ingredients (#kepoid)

2011-06-24T13:22:00.000+02:00

The world of internet is changeable and volatile. If we are to secure long-term access to content on the internet we have to find mechanisms to bring order to the seeming chaos. Standards, for instance – although I learned last month in Tallinn that we may be rushing into those (see blog post). Persistent identifiers are another type of building blocks for long-term access to digital objects, because PIDs make sure that we can find the object that is being preserved, even if it is moved from one URL to another. But I learned last week that the persistent identifiers are not as persistent as one might hope for. Another illusion down the drain?

In front of a famous painting by Rembrandt (The Anatomy Lesson of Dr. Nicholaes Tulp), a working group led by Andrew Treloar (standing, at right) dissects the truth about persistent identifiers and their complex relationship with Linked Open Data.

The setting was a two-day seminar on persistent object identifiers (or POID, thus #kepoid) organized by Knowledge Exchange, the PersID project, SURFfoundation and Data Archiving and Networked Service (DANS) in the Hague (14-15 June). Regrettably, I managed to attend only the second day, but it was enough to make me understand how complicated this business is.

This is how it should work: a (national) organization (national library, scientific organization) assigns a unique identifier to a digital object, a so-called persistent identifier. If the object is moved from one URL (internet location) to another, the PI remains the same and a resolver service links the new URL back to the PID.

Borrowing from Andrew Treloar’s presentation (Australian National Data Service), here are the main complications associated with object identifiers:

Granularity: what do you assign a PID to? In FRBR terms: to the work? to the expression? to the manifestation? to the item? Or, I may add, to a chapter? to a paragraph? Perhaps we even need multiple PIDs at multiple levels.
How do you assign PIDs to objects that are not static, but that change all the time (e.g., databases)?
How trustworthy is the object that is being identified (e.g., short url services)?
How to point to something inside the object?
Who owns the binding between the PID and the object?

And then there is the problem that there are a number of different PID systems (e.g., URN, DOI, PURL), which are not interoperable (comment by Juha Hakala: ‘It is encouraging that it is quite a long time since someone came up with a new PID system.’). And PID’s do not go well together with Linked Open Data (LOD).

‘Why is it so hard?’ – notes from Jeroen Rombouts’ computer (3TU.Datacenter)

Both Clifford Lynch and Andrew Treloar concluded that solving the technical problems of the PID challenge is the easiest part of the work to be done. Andrew built a pyramid of key success factors (photo above): at the bottom of the pyramid is a sustainability model, the second layer is about policies, the third is about procedures, and the top layer is about will or the intention of individuals to follow the rules and make the system work.

A room full of persistent identifiers – at right seminar chair Bas Cordewener (SURFfoundation).

In the end the attendees concluded that building interoperability between the existing PID systems is not a top priority. But getting PIDs to work with Linked Data is. Treloar proposed a 'Den Haag manifesto’ to bring this about:

The Hague Manifesto on persistent identifiers and Linked Open Data (LOD) (draft version)

Make sure PID’s can be referred to HTTP URI’s including content negotiation
Use LOD vocabularies, for schema elements
Identify the minimum common set of schema elements, across identifiers in scholarly communication space.
Use same-as relations to help PID interoperability across PID systems/schema’s
Work with the LOD community on simple policies/procedures to improve persistence of HTTP URI’s.

Treloar will work with anybody who is ‘ready, willing and able’ to develop these principles.

Some other recommendations from the meeting:

Do an inventory of different PID systems and make transparent how they work, so that organizations contemplating using PID’s know how to choose a system
Find the common ground between the systems and use these to widen awareness of PID problems and systems
Organize regular meetings between those who are involved in building PID infrastructures to facilitate alignment.

The work is being continued, within PersID and also within the European APARSEN project.

On alignment (when we do and don’t need it), and on the Elephants in the Room (ANADP11–10, Evaluation)

2011-06-18T15:31:00.000+02:00

ANADP11 was a conference about (inter)national alignment – and organizing a conference around such a theme implies that we need more of it. I offer you my tentative conclusions, a fortnight after the conference:

Sometimes aligning is as easy as hopping on a plane and attending a conference in Tallinn. Nothing tops meeting people face to face to exchange ideas and information. Much of the (technical) information can, of course, be found on the internet, but in our day-to-day lives we rarely have or make the time to actually find and read it all. Also: having an actual conversation about something is much more informative than one-way traffic. This type of alignment really requires no more than a bunch of enthusiastic people who are willing to put in a lot of time into putting conferences such as these together (thanks! Matt Schultz, Katherine Skinner, Martin Halbert, Aaron Trehub, Abigail Potter, Martha Anderson, Michelle Galinger and, last but not least, Mari Kannusaar of the Estonian National Library): the Tallinn conversation was an important type of alignment in and of itself in the way it was organized around themes, with panels discussing the issues before the conference.

Alignment networks become a little more formal when organizations actually start doing projects together: the Open Planets Foundation, the Alliance for Permanent Access, the US National Digital Information Infrastructure and Preservation Program (NDIIPP), the national digital preservation coalitions (nestor, DPC, NCDD). These are foundations with by-laws. Typically, they will run outreach and R&D programs together. A more informal, but very successful type of network is the International Internet Preservation Consortium (IIPC). It is fluid, it is informal. One wonders whether these need more international alignment than they already have. Perhaps more of these groupings are called for, especially in countries that have not yet embraced the issue of digital preservation, but if so, then these must really be bottom-up initiatives.

The organizational panel’s breakout session pleaded for fluid, informal international alignment rather than an international steering committee, as proposed by Laura Campbell in her keynote address.

Alignment of necessity becomes more formal when organizations start sharing the burden of digital preservation: LOCKSS, the MetaArchive, etc. Taking care of each others’ collections requires governance contracts to be drawn up. However, one wonders if these types of initiatives are in need of more or further international alignment; they seem to work best when similar organizations (similar as in: close together, with similar remits, of similar size, within the same scientific discipline) group together to do a very practical job. I thought the Alabama Digital Preservation Network (blog post) was a particularly powerful example which can inspire others. We definitely need to continue to organize conferences to highlight such initiatives for others to learn (and enable our staff to come to these conferences despite budget cuts …), but whether they need more (international) alignment as such … I wonder.

Sharing the burden of preservation requires more robust support, but often at a local and/or disciplinary level (St. Catherine’s passage, Tallinn)

Technical development, then, and standards. Are we reinventing the wheel over and over again? Will alignment help us save money? Yes and no. Of course we would save a lot of money in the short run if we were all to adopt the same technology. However, digital preservation is a moving target. Michael Seadle and Andreas Rauber assured us that all our present systems are untested; they are, indeed, “a leap of faith”. So this is no time to “rush into standards” (Bram van der Werf). In other words: exchanging experiences, exchanging test data: yes; throwing all of our eggs into one basket at this (early) point in time: definitely not. We need to allow for diversity in preservation strategies and tools to develop. This will, of course, lead to some redundancy, but that is all in the game.

So far, we have made a strong case for more conferences, more conversations, more debates, more projects, more comparing of notes, more sharing of experiences (test data, best and, yes please, worst practices), the establishment of fluid, informal affinity groups to share knowledge. Plus more formal alignment on a local or national scale when organizations actually start collaborating in preserving their digital collections. But we have not yet made the case for more international alignment. Unless … one thinks about …

An international digital preservation registry?

During the conference the call was heard for some type of international registry of projects, initiatives, knowledge, competences, etc. I have heard that call before, and it would be great to have such a facility? A one-stop portal where everything that we know about what is being preserved by whom, everything we are studying (including everything we have found out that does not work), and perhaps even more importantly, everybody involved in DP and their special expertise, is linked up and a brokering services brings supply and demand together. Not a project, but a sustained effort by the international community.

The trouble with that is that we had something like that, the PADI service, run by the Australian National Library. But, as Maurizio Lunghi (photo left) explained at the conference, that service was discontinued last year. The website is still up, but it is not being maintained any more. PADI went down, Lunghi told us, because it started out as an international effort, but in the course of time it became isolated, and the Australian National Library was doing all the work alone. PADI became too isolated. And that is the trouble with this type of registry/competence center. Everybody wants it, but establishing and maintaining one is very labour-intensive. Especially in difficult economic times such as the present, our institutions’ management will not prioritise active knowledge sharing. Which is understandable in the short term, but obviously unwise in the long term. I have been involved in organizing something like this at a national scale, and apart from the staff effort involved, it is also very difficult to make all that knowledge and experience available in a form that is useful to users – simply because there are so many users with so many different needs.

But wouldn’t it be great … I am not giving up the idea and I welcome anybody to think this through with me!

The Digital Elephants in the Room

So much for the problems that we did discuss. Now for the ones we did not discuss (enough). Clifford Lynch called them the Elephants in the Room, the issues we chose not to talk about because … yes, well, why did not we discuss them? I have a couple of theories. For one, the conference room was dominated by librarians – there were very few people from the archives, museums, or from scientific repositories. That was a shame, because it limits the scope of issues to be discussed. Particularly, those issues that transcend traditional borders between domains and sectors were not talked about nearly enough. And I would argue that these are the very issues where we absolutely need international alignment because there is no other way to deal with them. Here is my list of these Key Issues (which I will be happy to develop further with anybody who wants to join the debate):

Lots and lots of digital content is being produced at the moment that no heritage institution is collecting, because it does not fall within traditional collection profiles. It is high time we get together at a high policy level, with representatives from a broad range of organizations (museums, libraries, archives, scientific institutions) to talk about this huge challenge. This involves talking about data deluges (in science, but also in social media; audiovisual output for which there is no Public Records Act and no Legal Deposit Scheme), it definitely involves talking about selection (what to keep?), and who is going to do the keeping (distribute the work? establish new organizations?)
Making the case of digital preservation towards funders and the public at large. This is a big one. It may force us to take another look at the assumptions underlying our work before we can get this right. Which is not something people particularly like doing. But we’ve got to do it, make sure that we talk with one voice. Can we write the success cases to prove our point?
Where do we put our money? Both in Europe and in the US substantial amounts of money are invested in digital preservation research. But is R&D attacking the right problems? Where should we put our money to really start bringing costs of digital preservation down?
Should effective copyright action be in this list? Clifford Lynch suggested that we get some smart people from both sides of the ocean together to really get to the bottom of the issue and organize an effective lobby to truly bring about a fundamental change in thinking about the individual (producer’s) rights versus public rights.
Dare we think of an international registry that will really allow the international community for reinventing wheels only where alternatives really are beneficial?

At the end of the day I think I can say that no, we did not get everything right at this first ANADP conference. But we made a good start at filtering out the issues and deciding where they should be addressed and by whom. That in itself is important. There is talk of a follow-up workshop at iPRES and there is talk of a follow-up ANADP. Let’s try to include more parties there (more countries, more domains), let’s try to bring more decision makers to the debate, and let’s try to narrow the issues down to those areas where alignment is truly essential.

My favorite conference recommendation, by Jeremy York of HathiTrust: ‘Developing killer digital preservation apps’

This is the last of a series of 11 blog posts about the Tallinn conference. The others were:

Martin Halbert (left) and Matt Schultz of Educopia, two of the driving forces behind this initiative to open up a transcontinental policy debate.

Many thanks to Mari Kannusaar of the Estonian National Library (right), thanks to her colleague Leila and all the other colleagues for a smooth conference!

This is the last of a series of 11 blog posts about the Tallinn conference. The others were:

Clifford Lynch’s synthesis – ANADP11 (9)

2011-05-31T19:34:00.001+02:00

Clifford Lynch of the Coalition for Networked Information has a remarkable talent for analyzing a lot of information and bringing it together in a synthesis. Throughout the conference you could see him sitting quietly somewhere in the audience, digesting what happened, pondering it, and drawing his conclusions (photo right and photo below).

I am not sure that it is entirely appropriate to synthesize the synthesizer, but I do want to give you some highlights from Cliff’s ‘opinionated synthesis’ (the highlights themselves, of course, being opinionated as well …).

‘Yes, alignment is very necessary,’ Cliff started out, because:

Working in the same direction makes much needed collaboration possible;
Everybody will benefit from mutual support and shared learning.
What Cliff did not hear enough about at the conference: Collectively we can make a much more effective case towards governments, funders and society at large. (Cliff: ‘It is of fundamental importance that we clarify and educate.’)

The six axes of alignment dealt with in the conference interrelate in very complicated ways, of course, but Cliff took the conference outline as his point of departure anyway.

Technical panel

Testing & benchmarking are very important topics. We need to learn to do this better and we need to introduce scale into the equation. This ties into the economic angle: we need to test commercial products.
‘There is some glib talk about interoperability’; we need to be more specific: what? how? what goals?
What Cliff did not hear enough about at the conference, was the danger of a monoculture. We must be careful not to align too much; we need diversity to further develop the field.
Two areas deserve more attention: the bit storage layer (how to trade off between partners? what is the role of commercial services? what about standards?; and, secondly, security and integrity. ‘Just imagine the impact if somebody would raid and Wikileak all the embargo collections we hold …’ The stakes are substantial!

Organizational panel

Cliff thought that the honesty displayed by yours truly (see blog) was ‘wonderful’, and contained ‘lessons for all’. I humbly thank Cliff for this comment – and hope that others will be motivated to follow the example.
We are moving from cooperation in short-term projects into cooperation in fundamental, on-going programs, which is a very good thing.
What Cliff missed: information about aligning strategies within national borders: what variation is there in assigning levels there. He also missed more conversation about the question of replication – what choices have to be made? What about autonomy and interdependence?

Standards panel

Some standards may not be DP-specific, but as they characterize the materials we deal with, they are relevant.
There is a thicket of analytic standards – and maybe we are rushing prematurely into some of them. Perhaps it is better to talk about ‘best practices’ for the time being.

Legal panel

The legal issues are becoming more dominant all the time.
Cliff was impressed by the report of the Comité des sages and the efforts it reflects to define collective EU principles on copyright.
Cliff suggested that a bunch of smart people from both sides of the ocean get together to drive policy on copyright matters. [Would not that be great – is it feasible?]

Educational panel

A lot of curricula feed back into national programs.
We are focused on educating professionals – but we should also start training people outside our field and the general public. The ‘national preservation week’ launched by the Library of Congress is a great initiative to Make the Case for digital preservation.

Economic panel

A very important distinction: digitization is very different from digital preservation when it comes to funding. The latter has no immediate payoffs and is thus harder to fund.
We did not talk much about scale; more attention required, in combination with cost models.
When it comes to sustainability, the very fact that we define certain goods as public is in itself a sustainability strategy.
Instability in public funding is a nightmare!
We should pay some more attention to the connection between risk management and costs both for physical and for digital materials.

The elephants in the room

According to Cliff, the elephant in the room (a topic that is avoided by implication) is the data deluge in e-science, Big Data. As e-science drives both education and technology, this is very relevant. But national libraries are not paying much attention to what happens in Big Data.

Perhaps a smaller elephant is audiovisual content, both physical and digital. This information is expensive to maintain, and AV materials seem to be the step children of national collection strategies.

Another elephant: What is the extent of born-digital material – government, commercial, social media. We do not have a good assessment of this content yet.

If we ever have another conference like this, Cliff suggested that we add two more axes to the discussion: Making the Case towards funders and the public at large; and: Strategic alignment of collection policies (personal records, social media, games, news: who preserves what?).

(PS: Just in case you wonder: ANADP11, blog post no. 8 is still to come; I am waiting for some necessary information.)

The right mind-set: training and education at ANADP11 (8)

2011-05-30T14:04:00.000+02:00

It is no secret that very few of those presently involved in (R&D in) digital preservation have been formally trained to do so. As Meg Phillips of the US National Archives said at the conference: ‘Present staff at cultural heritage institutions have patched it together – they have degrees in the arts or humanities, and what they know of digital preservation they have mostly learned on the job.’ This is no wonder, of course, since DP is a new field and it takes a while for supporting industries such as education and training to catch up. Judging by the presentations of the educational panel this work is still ongoing. Because it does not only involve defining new skills, but also combining those skills with existing skills (library, museum, etc.), fitting all those skills in job descriptions that real-live people can actually fulfill and fitting all those job descriptions into effective work flows and organizational settings. Then develop training programs and curricula that can deliver those people.

Educational panel, from the left: Andreas Rauber (TU Wien), Sheila Corrall (Univ. of Sheffield), Joy Davidson (UK DCC), George Coulbourne (US LoC)

To make things more complicated, the ‘new’ skills and the ‘old’ skills do not always go well together. As Joy Davidson emphasized, this is where the library meets IT and these two have very different service-mindsets: whereas libraries are about very personal, individual service, IT is about rolling services out to lots of people. Service requires softer skills (communication), whereas IT is more about facts and figures.

And there is more. When asked to define her ‘fantasy staff member’, Meg Phillips of the US National Archives and Records Administration (and one of the few archivists present), said that NARA needs staff with broad knowledge of formats and digital preservation tools, staff who can think, who can help make decisions. Howard Besser of New York University added: ‘We do not need tinkerers; we need people who can deal with the scale of the digital challenge, who can deal with new, emerging problems.’ Some existing staff can be trained new skills, Howard said, but as this is more about mind set than actual skills, we must accept that some of the staff we have will not be able to make the change. It is the younger people, the students, who have the right mind frame.

Students attending the conference: ‘Please continue to invite students to these meetings; it develops our thinking at a strategic level.’

The panel reviewed a number of training and education initiatives (check out the slides here). Joy Davidson said she was happy about the level of international cooperation between the different projects and institutions. Yet there is still a lot of diversity in the language used by stakeholders in the different contexts, which makes it hard to develop cross-disciplinary curricula. It is is continuing to be difficult to determine what is generic about DP (education) and what is specific to the contexts.

Here is the educational panel’s answer to the question “Where do you want to be in five years?”

foster inter-disciplinary group projects in undergrad and postgrad courses (making use of tools like Planets Testbed and Plato). This could be a quick win
ability to describe and compare data management courses across international postgraduate research courses
establishment of frameworks for assessing data management skills and professional development progression paths for skills development for a range of disciplines
greater practical experience opportunities for students
greater engagement with professional bodies and industry to cooperate on raising awareness amongst researchers of the need to acquire sound data management skills
ability to match skills across data curation lifecycle and the various roles
development of information science courses that complement and bridge the gaps in the data management skills of researchers and information technologists. More real-life examples of data management and curation as it relates to day to day activities on the job
more induction level data management training for all secondary, undergraduate, postgrad and PhD students.

That is some to-do list! But a crucial one. In the end, even in a technical field such as digital preservation, it is the people that make the difference – between ‘tinkering’ and a truly digital mind-set.

Who pays for the invisible and unglamorous but absolutely necessary? – ANADP11 (7) & ESDI11 Round table

2011-05-29T14:15:00.008+02:00

‘Economic sustainability’ is a buzz word in digital preservation. It is about what it costs and who pays. It is about users expecting everything that is online to be freely available while the people who do the actual work of providing preservation and access need to feed their families.

On the eve of their session, the ANADP11 economic panel does not (yet) seem daunted by the fact that they have to deal with difficult questions about money – from the left: Aaron Trehub (Auburn University; Alabama Digital Preservation Network; Neil Grindley (JISC), Bohdana Stoklasová (Czech National Library); Maurizio Lunghi (Fondazione Rinascimento Digitale).

There was an ANADP11 panel about these issues as well as an ‘ESDI Roundtable discussion’ (ESDI = Economic Sustainability of Digital Information) the next day. The latter was organized by Neil Grindley of JISC as a follow-up to earlier work by the US/UK Blue Ribbon Task Force.

I am going to bring some threads from both meetings together in this post. First of all: in a financial context it is essential that we distinguish digitization (the act of making digital copies) from digital preservation (the act of preserving digital objects). Every time I think we have made this point clear enough, some speaker will mix them up again. So, I begin by citing Bohdana Stoklasová of the small but very active Czech National Library (remember those wonderful young web archivsts at IIPC11!):

‘Digital preservation work is invisible, and therefore it is hard to find funding for it. Digitizing new material is much more appealing to funders, because the benefits are immediately visible.’ (Bohdana Stoklasová)

The conversation that followed was mostly about funding and the sustainability of funding (we need long-term money rather than project money), and less about what digital preservation actually costs (the jury is still out on that, as witnessed by a workshop report from an international cost modelling meeting in The Hague last year).

Neil Grindley from JISC offered the audience some valuable insights into how funders think: ‘When a funder sees a problem, he wants to throw money at it and solve the problem. But some problems are too large and intractible.’ Comparing DP to an asteroid (slide right), the question is where and when to hit the asteroid to disarm it. Timing is very important, as is scale. Laura Campbell (Library of Congress) agreed that this is an important issue that funders have to address: exactly where will funders’ money be most effective in bringing DP costs down? She agreed with Grindley that funders from all over the world need to talk about this and align their efforts so that the money will be used in the most effective way. Grindley’s conclusions merit further discussion:

Grassroots initiatives from the US

Americans tend to have a wonderful way of ‘cutting to the chase’. Aaron Trehub of Auburn University Libraries (photo left) was a case in point:

‘An unofficial definition: Digital preservation is the flipside of digital collection-building. Like most infrastructure, it’s invisible, unglamorous, and absolutely necessary’.

‘DP is necessary because: – things (hurricanes, etc.) happen; – hardware breaks; – software crashes; – files get corrupted (‘bit rot’); – somebody hits the wrong button; – somebody drops something; – burning files to CDs and DVDs isn’t enough; – it costs money to restore lost collections.’

‘One solution: distributed digital preservation (DDP): – multiple copies of digital content; – at geographically dispersed locations; – with at least 75-125 miles between them; – preferably out of hurricane pathways; – and across different power grids'; – with different system administrators; – on live media, with audits for integrity; = and in at least 3 copies (LOCKSS = six copies).’

‘DDP basic ingredients: – institutional commitment; – quorum of at least six institutions; – hardware (LOCKSS boxes), – in-house IT support; governance structure; – enough $$$ to cover membership and/or LOCKSS fees (your institution might already be a member!); – enough ‘sweat equity’ to keep it running.’

Such a DDP set-up works even for a poor state such as Alabame, which established the ‘Alabama Digital Preservation Network’ (or ADPNet), in which libraries, museums, archives and other cultural heritage organizations in Alabana share the work of preservation. The network now holds some 3TB of data and has been self-sustaining since 2008. Here is Aaron Trehub’s assessment of ADPNet’s significance and his economic advice for anyone thinking about doing something similar (and do check out the remainder of Aaron’s slides when they become available through the Educopia event website!):

ESDI11 – Economic Sustainability of Digital Information Roundtable

On Thursday, a selection of ANADP11 attendees defied increasing bouts of conference fatigue and gathered to meet with some extremely early risers across the ocean who attended via Skype (also see the tweet threat on #ESDI11 with excellent work by cardcc). Neil Grindley explained that a mini-taskforce consisting of Chris Rusbridge (present LinkedIn profile: ‘retired’ at ‘None’) and Brian Lavoie is following up the work done earlier by the Blue Ribbon Task Force, by trying to turn their 100+ page report ‘Sustainable economics for a digital planet’ into a ‘reference model’, a tool that can help us understand the economic issues and the dynamics of sustainability, provide us with a joint vocabulary, and help us make the case vis-à-vis funders and the public at large. In that sense the model is to work much like the OAIS model did for technical issues – although Chris Rusbridge was quick to add that he does not like the OAIS model much (‘In reality digital archiving is a much more fluid and continuing activity than the closed box idea that is at the centre of OAIS’).

But first a few words on the BRTF work. Sabine Schrimpf of the NCDD’s German colleagues, nestor, reported from the nestor working group which studied the relevance of the report for Germany (Report in German. The working group agreed with many of the Task Force’s findings, but was ‘reluctant’ to accept the report’s recommendations for more public-private partnerships (PPP’s). I felt much the same way when I read the report – the US economy’s dynamics are different from much that goes on in Europe. We accept much more readily that preservation is a public duty and should be funded accordingly. (But note Clifford Lynch’s assessment of PPP’s: ‘Mostly, when there are profits, they go to private partners; if there are losses, they to to the public sector.’)

About the reference model: Chris Rusbridge (at right, with Brian Lavoie standing by via Skype, see cable mess above) stressed that the present text is very, very draft (see slides forthcoming on JISC website), but what I saw from it looks promising. It charts all those roles and relationships that make the digital world so different from the analogue world. E.g., what are the properties of sustainability? We do a lot of the work for ‘future users’, but they have no voice yet, so who can function as their ‘current representatives’? Who benefits, who pays; what about free riders? Where do rights holders come in? Etc. (‘Nobody wants preservation; everybody wants the digital asset.’ Chris Rusbridge)

The slides are very much worth reading – all of them will be made available through the JISC website, but here is a taste for you (thanks, Chris, for making these available to me on Sunday afternoon!)

Wouter Schallier of LIBER (the European Association of Research Libraries) reviewed the work that LIBER is doing to make the case for digital information, focussing on the (access) project Europeana, where digital content from all the EU member states is being made available. As an access project, Europeana is presently sustainable, but the future is uncertain. What will happen to Europeana if one of the major aggregators stops doing the work? Off the record doubts could be registered as well as to the added value of Europeana in comparison to Google. Is the Europeana system user-friendly enough?

Matthew Loy and Nancy Maron from Ithaka joined the conference by Skype to report on the sustainability research they did for JISC a couple of years ago, with a number of cases studies (full documentation here). Clifford Lynch (Coalition for Networked Information, CNI) commented that in his view the case studies are not classical digital preservation sustainability situations.

In the end, everybody agreed that the work Rusbridge and Lavoie are doing on the ESRM can definitely have value for the community, especially in the difficult task of Making the Case vis-à-vis funders and the public at large. But it is complicated business that Lavoie & Rusbridge are trying to chart, and a possible pitfall is that people will start using it as a standard (‘we are ESRM-compliant’). And there is need for a community to support the work and test it. Sabine Schrimpf suggested the nestor working group, but more testing is obviously needed.

Chris Rusbridge (furiously tweeting) and Clifford Lynch (CNI).

Clifford Lynch commented at some time during the meeting that ‘the BRTF report is cited a lot, but I am not sure how well it has been read.’ Matthew Woollard drew attention to the option strategies mentioned in the report. As we do not know what future demand will be for what we are presently curating, the way to save money might be to invest a little money into preservation work now, for say five years, and then decide later what selection from the material merits expensive curation measures. Clifford Lynch added that this strategy is being promoted by the US National Science Foundation as well, as short-term options tend to be cheap.

I leave you with Neil Grindley’s key words from the meeting. To be continued, absolutely.

Legal alignment? Forget it! - ANADP11 (6)

2011-05-28T13:17:00.001+02:00

The most succinct summary of the legal status quo with regard to digital objects came from Dwayne Buttler (University of Louisville and legal adviser to MetaArchive): ‘Copyright law prevents you from making copies, and I cannot imagine a digital world without copies.’ It did not make the legal panel (photo, from the left: Dwayne Buttler, Adrienne Muir, Wilma Mossink) any less cheerful – after all, legal complications are their bread and butter.

Adrienne Muir of Loughborough University compared national deposit arrangements with regard to web archiving. Some are legal deposits, others are voluntary (although not necessarily less effective, as witnessed by the Netherlands); some include audiovisual materials, others do not. ‘This variance’, she said, ‘leads to gaps in what is being preserved for the future.’ Adrienne does not think that legal alignment is the most appropriate approach to filling this gap. She expects more progress from aligment in implementation (which is just what Laura Campbell and I were talking about - see previous post).

The community’s struggle with copyright provisions is well-known. Wilma Mossink of the Dutch SURF Foundation explained that preservation in itself is not the biggest problem. Providing access to what you have preserved, however, is complicated. She reviewed the situation in Europe and then concluded that legal alignment at the European level is difficult because of dependency on unpredictable factors (lobby by rights holders; member states unwilling to give up their legal systems). Wilma expects more benefit from communal defined legal requirements, and said we should strive for one voice in those. Access to preserved digital objects, Wilma asserted, shall be more easily arranged by agreements between the stakeholders than by legislation.

Wilma referred to ‘The New Renaissance’, a recent report by a ‘Comite des sages’ (committee of the wise) that pleads more access to Europe’s cultural treasures through the Europeana project. Strangely enough, it says that one copy of all European digital cultural treasures should be archived at Europeana – although Europeana is not, at present, an archive at all. It is an aggregator for content from member states. Europeana is about access, not about preservation.

Dwayne Buttler (above) undoubtedly won the contest for the most entertaining presentation of the conference. His characteristic assessment of the situation:

The law is imperfect, but perhaps US institutions can expect some help from the “fair use doctrine’:

Woops. I just realized that Buttler drew up the permission-to-reuse-conference-stuff-by-Educopia forms all speakers signed, but did not sign one himself, ‘I am a lawyer after all’, he said with a broad smile. Am I in trouble now? Or can the fair use doctrine be made to apply in this case as well?

Perhaps it is safer for me to refer to writings by Buttler that are on the MetaArchive website anyway: the governance contract underlying the MetaArchive cooperative, and which anyone with similar intentions can consult:

In summary: in my experience it is pretty rare for lawyers to agree on anything, but lo and behold: it happened at this conference. Their joint advice: do not expect fast progress in legal alignment to make it easier to provide access to preserved content (which is what memory institutions want to do). As a community of heritage institutions, we should pull together and develop a joint stance and voice to lobby more effectively, but meanwhile – and at the same time – voluntary agreements between the stakeholders offer the best prospect for opening up those digital treasures.

Tallinn main square on a glorious spring evening

ANADP11 - corrections & additions

2011-05-28T04:36:00.000+02:00

I welcome anyone who does not agree with something I have written about the conference to send me an e-mail (inge.angevaare@kb.nl). There was a lot going on at the conference, and I may have gotten some things wrong. I slip any changes back into the original text, but for early readers, I will list them specifically in this blog post.

Blog no. 5, about standards: I rewrote Hans Hofman’s comments on Saturday, 17:00 Dutch time) after an e-mail from him which clarified his viewpoint:

Hans Hofman of the Dutch National Archives (also attending on behalf of the International Council of Archives) suggested that there are many standards already available that also relate to digital preservation, such as international record keeping standards. He feels that we keep reinventing the wheel if the DP community does not make use of these. As an archivist, he also felt dat the discussions at this conference were being dominated by libraries and their collection approach to digital preservation, and if there ever is a next conference (there was talk of that possibility), we should find the common ground between libraries and archives (active management over time of digital objects) and focus on those.

Are we rushing into standards? – ANADP11 (5)

2011-05-28T01:54:00.001+02:00

In a conference about alignment a session on standards is very well placed, for standardization is a form of alignment in and of itself. And yet, we have all heard the adagium with which Raivo Ruusalepp (left) opened the session: ‘Standards are great; everybody should have one.’ There are so many standards: digital preservation standards, external standards that we really do not know the value of (e.g., PDF), standards from specific domains – ‘it’s like a Chinese menu we have to choose from.’ OAIS is of course the most well-known and well-used DP standard (but for organizations to advertise that they are ‘OAIS-compliant’ is utter nonsense; OAIS is a reference model, a framework of concepts, nothing more, nothing less).

Speaking also on behalf of Estonian colleages Mihkel Reial and Kai Idarand, Raivo went on to describe how the Estonians reacted to a massive cyber attack four years ago. They agreed to step up security measures – but whether the words have always been turned into actions is up for debate.

The other members of the standards panel, from the left: Cal Lee (UNC School of Information and Library Science, Chapel Hill), Bram van der Werf (Open Planets Foundation) and Matthew Woollard (UK Data Service). In the background Robert Sharpe of Tessella, vendor & conference sponsor.

Matthew Woollard of the UK Data Service (photo right) told the conference why his organization went through an ISO 27001 certification process that cost it no less than GBP 100,000: for pragmatic reasons, in order to secure annual funding to the amount of GBP 1 million. ‘A reasonable investment,’ he concluded with a quite British sense for the understatement.

Matthew Woollard went on to make a pitch for the Data Seal of Approval in the context of present European efforts to develop a three-tiered certification system (see also previous post). ‘Read the DSA,’ he said, ‘it is simple, it is understandable, and it may help you understand your organization better.’ He expects that at some point in time, some level of certification may be required for any organization participating in EU-funded projects.

Cal Lee of the UNC School of Information and Library Sciences dealt with contextual information (metadata) and the conventions and standards involved in those. I must admit that some of it was too technical for my (policy & communications) brain – my apologies, Cal (check out his slides on the conference website in due course). But he made a few fundamental assertions with regard to metadata standards that merit reproducing:

Bram van der Werf of the Open Planets Foundation (photo right) raised some fundamental issues of his own with regard to standards. He said: ‘Most major challenges are about people. Standards are basically designed to allow less clever people to perform activities that clever people have designed. Thus quality requirements and standards are most suited to deal with static challenges. Digital preservation is by no means a static field; rather it is a moving target. Thus one may wonder if standards really help to improve digital preservation. I would propose that we start thinking in terms of current best practices and accept that they will evolve continually.’

Hans Hofman of the Dutch National Archives (also attending on behalf of the International Council of Archives) suggested that there are many standards already available that also relate to digital preservation, such as international record keeping standards. He feels that we keep reinventing the wheel if the DP community does not make use of these. As an archivist, he also felt dat the discussions at this conference were being dominated by libraries and their collection approach to digital preservation, and if there ever is a next conference (there was talk of that possibility), we should find the common ground between libraries and archives (which is active management over time of digital objects) and focus on that.

David Giaretta, APA (left): ‘I never thought I would say this, but I actually agree with Hans’ [Hofman, right in the photo]

It was also argued that digital preservation is not an activity in itself – it is part and parcel of an entire work flow and standards should reflect that. Matthew Woollard would like to see them become more context- and culture-specific.

In conclusion, Cal Lee prepared this slide with standards success factors:

Diversity in approaches – ANADP11 (4)

2011-05-26T19:15:00.002+02:00

The organizational panel was a showcase for the many different ways in which institutions develop methods to cooperate in digital preservation.

From the left: Martin Halbert (MetaArchive), yours truly (NCDD), David Giaretta (APA), Michelle Galinger (NDIIPP/Library of Congress).

David Giaretta of the Alliance for Permanent Access to the Records of Science (APA) drew a picture of the many initiatives in the European research arena. Digital preservation, David asserted, is not a separate activity, it must be fitted into the research infrastructure. The European Commission is funding many R&D projects in digital preservation (Planets, ParseInsight, CASPAR, Shaman, etc.; you can google them for more info). If truth be told, the variety of institutions and projects is quite overwhelming – perhaps that is why David’s slides have a tendency to be that as well ;-) – here’s an example that can keep you busy for a while.

Present efforts are focused around APARSEN, with over 30 participating organizations. The intention is to establish a Europe-wide competence center for digital preservation. A new acronym for me was SCIDIP-ES. Here is the accompanying elucidating slide:

Europe is also working on a three-tiered audit and certification system that makes a lot of sense to me: it allows institutions to grow into certification: there is a light-weight instrument for basics (the Data Seal of Approval), a second level of self-audit, and then a super-level of external audit.

While the Alliance clearly takes the technical requirements as its starting point, the US NDIIPP program draws its inspiration from the social context, from ‘shared values’ and ‘common goals’ – Michelle Galinger (left) told the conference. NDIIPP began as a project organization, but as time went on the need was felt to build something more committed, something more sustainable. This led, in 2010, to the establishment of the National Digital Stewardship Alliance (NDSA), in which partners ‘commit to formalized roles and structure within alliance’.

A third genre of DP organizations are the national coalitions, mostly bottom-up voluntary networks of custodial institutions across the borders of traditional domains (archives, libraries, museums, research institutions). The UK DPC and German nestor are networks to promote awareness and share knowledge and expertise. The Dutch NCDD goes a step further and intends to build an infrastructure (see yesterday’s post).

All of the above are enabling organizations, facilitators. To my mind things get really interesting when institutions actually start sharing the burden of digital preservation. Not surprisingly, reciprocal hands-on networks like these have mainly sprung up in the United States. Martin Halbert of North Texas University summarized the distributed preservation approach:

The American version of the distributed preservation approach is not undisputed. LOCKSS is about bit preservation, and especially in Europe there are those that believe that without proper preservation strategies the job cannot be done. At this conference I got the distinct impression that for economic reasons, thinking is turning in the direction of spending our money on relatively simple preservation methods for the short term, especially because it is very difficult to tell what will be valuable in the future. Matthew Woollard of the UK Data Service spoke out in this direction. After some years the material can be reappraised and more expensive preservation strategies can be applied to selected material only.

Lively discussions about organizational aspects during the break-out session.

One of the nice features of this conference, I thought, was the room for conversation, for discussion. During this break-out session Laura Campbell’s idea for a global preservation body (see blog) was pretty much rejected. It was felt that while the organizations doing the actual preservation must have continuity and long-term commitment, interaction between these organizations should be more fluid, more spontaneous. The International Internet Preservation Consortium (IIPC, see recent blogs on their General Assembly) was repeatedly mentioned as an excellent example of how fluid, voluntary groupings of people with similar problems can work in the 21st century.

The conclusion was that in DP (not unlike in many other situations) one size does not fit all. Commonalities, shared problems, are basic to successful types of cooperation, and sometimes these must be context- and culture-specific. It was also said, during another session, that in a young field such as digital preservation we need to experiment with different approaches in order to discover what works and what does not.

At the end of the day, Laura Campbell and I, over dinner, thought of a very fluid and informal network to address some issues that really need addressing on a high international level, such as how to redefine our collection profiles within a digital paradigm. To be continued!

PS: Meanwhile the conference has come to a close. But there is good stuff yet to share with you. So, if you will bear with me, I will go on blogging about it for a few more days. Until the story is told.

Conference reception: We were so captivated by the singing of the National Library’s Women’s Choir, that the excellent food remained untouched for quite a while …

To share or not to share … our weaknesses – ANADP11 (3)

2011-05-25T10:10:00.002+02:00

Something I thought a lot about in preparing for this conference is that presentations tend to be about how wonderful our organizations and projects are. The value of such information is limited in the sense that we often learn more from our mistakes and failures. That is why conferences have receptions and dinners – after a few glasses of wine it is often easier to get at least fragments of the real story. But that is is not an ideal route. Major headache risk, for one.

The three fundamental questions in this conference are: What are our accomplishments? What are the gaps? How can we move ahead? We cannot very well do that without honestly looking at ourselves. And share that information, so that we can all learn.

So, I decided to take a risk. Rather than tell the story of my own organization (the Dutch Digital Preservation Coalition, or NCDD) in terms of the information that you can easily find on the website (http://www.ncdd.nl/en), I took the requirements for successfully organizing digital preservation with which I ended yesterday’s blogpost, and measured ourselves against those (by the looks of this photograph by Wouter Schallier, grading is serious business ;-). I then presented the results to the audience. Consider it a mid-term evaluation, for a national coalition that in 2008 formulated a very bold mission ‘to build an infrastructure for permanent access to all digital objects in the public domain – within five years’.

I used simple report card gradings (V meaning ‘doing pretty good’; B meaning ‘we have made a beginning’; +/- meaning ‘got some, need more’; 0 meaning ‘presently a gap’). And came up with this – my own evaluation of things:

That does not look too bad, does it? We seem to be right on track. But after a night’s sleep I decided I was not happy with it. Somehow it did not tell the whole story. And then I discovered why not – this was all about words, about minutes of meetings, published documents, etc. This report card required a second column of ratings, actions. What have we, as a coalition (I am not talking about individual partners) really built that works, that exploits the commonalities, that saves actual money because of the scale, that divides the work according to a digital paradigm?

This report card shows how difficult it is to take the step from words to concrete actions. This report card shows that cooperation is hard and slow work. This is the report card that most projects do not show at conferences. Yet this is the report card that teaches us something, that tells us about the areas where we have to step up our game.

I was quite surprised by the positive feedback from the conference about the report card. So I would say: we are all in the same boat, so let’s set a tradition, align our weaknesses, and learn from them.

Aligning digital preservation internationally – ANADP11 (2)

2011-05-24T20:38:00.006+02:00

At the beginning of the conference (more in previous post), the President of the Republic of Estonia, Toomas Hendrik Ilves (left), provided the attendees with a powerful reminder of why we are all involved in this business of preserving our heritage at all. Referring to successive occupations of Estonia by German and then Soviet troops, he recounted efforts to destroy the Estonian culture, e.g., by destroying 10 million books, and ‘supplanting national awareness with some bizarre fairy tales.’ When Estonia became independent, in 1991, KGB files disappeared by the thousands, in order that occupation misdeeds might not be uncovered. Now Estonia is trying to retrieve and preserve whatever was not lost, because ‘information and knowledge help people stand against totalitarianism.’

This leaves us with the question how we are going to do that, and, at this conference in particular, the question how we stand on aligning our efforts internationally.
The first day’s keynote came from Laura Campbell, Assistant Librarian for Strategic Initiatives at the Library of Congress, and leader of the LoC National Digital Information Infrastructure and Preservation Program (NDIIPP). Under the motto ‘Together we can light up the world’, she recounted the development of NDIIPP from a bag of Congress money into an organization to fund DP projects, and on into a central node, a non-authoritarian team leader, of digital preservation initatives in the US with 185 partners in 45 states. The NDIIPP approach has a strong social element, with mottos like ‘Learn by doing (let’s learn from our mistakes)’ and ‘We are a network of people’.

Laura’s summary of changes in our (digital) environment since NDIIPP began is worthwhile reproducing (see slide at right). And she had some good quotes as well: ‘Seamless integration? You can try it, but it is extremely difficult.’ ‘Diversity & interaction drive innovation.’ ‘Deliberate planning does not work; we need loose collaboration instead.’ ‘As storage costs go down, it will soon be possible to think about collecting everything – is that a wild idea?’

In order to promote international alignment, Laura Campbell suggested the establishment of an international preservation body with a focus on policy, perhaps assisted by an advisory expert group to identify what categories of digital objects are most at risk. The body could promote an international notion of collection, work on standards and tools, and maybe maintain a common index of preserved materials.
Is this idea going to land at this conference? Laura invited ideas from the community.

The organizers (Educopia Institute, see previous post) have set ambitious goals for themselves – to build an agenda for concrete action towards international alignment from this conference. Let’s see what happens!
The conference program includes six distinct thematic panels with corresponding breakout sessions for discussion. The panels are: technical alignment; organizational alignment; standards alignment; legal alignment; educational alignment; economic alignment. Some (would) argue that you cannot really separate these issues. We’ll see about that too.

Let me share some highlights with you (from my own perspective – which is more about policy and communications than hard-core technical stuff; let other bloggers help to complete the picture! e.g., http://digitalplusresearch.blogspot.com/).

Technical alignment panel: our present preservation systems are a leap of faith

Michael Seadle of the Berlin School of Library and Information Science (photo right) opened the technical panel as follows: ‘I want you to become uncomfortable.’ He and Andreas Rauber of TU Wien (and Planets) told the conference straight out that the technical systems that we rely on presently to preserve our digital objects have not been tested. In truth, we have no clue as to how well our present systems function.’

Some way to start the day! Half the audience was still jet-lagged. The others all woke up at 4 a.m. this morning, when the superbright Nordic sun shone into our hotel room windows (and when we all went online to check out the Iceland ash cloud and stayed awake worrying about getting home later this week).

Technical panel doing last-minute homework, from the left: Andy Rauber (TU Wien), Sabine Schrimpf (Deutsche Nationalbibliothek, nestor), Michael Seadle, Adam Rusbridge (EDINA, LOCKS UK).

But Michael and Andy are right, of course. Michael: ‘Vendors are promising too much.’ and: ‘We are buying archiving systems on faith.’ Even though Robert Sharpe of Tessella put up a valiant effort to defend his systems, there is still so much we do not know. The technology has not been proven. So, what we need is two things: a) testing, transparent, objective testing to check the functionality of the systems and uncover the weaknesses; and, b) for the time being expectations must be managed: those in our institutions who purchase the systems must know what they are buying, what the limitations are. (Which sparked a comment by Swedish National Librarian Gunnar Sahlin that leadership in our organizations is being challenged by the very complexity of the systems leaders have to decide about.)

The end of the discussion was that everybody agreed that we need lots of testing and benchmarking. But when asked if the attendees were prepared to put lots of resources towards testing and benchmarking, very few hands were raised.

On the topic of an international body, Andy was not immediately convinced. ‘Perhaps we can work with a small group of organizations that want to work on testing and benchmarking first.’

Organizational alignment panel: changing from an analogue to a digital paradigm

Let me Mirandize you before reading what follows: I chaired this session myself.

I set the scene with the well-known cartoon of Father Dominic transcribing a monastery’s archives on to cd-rom – with a quill (left). I argue that it is true that our organizations have been receiving and collecting digital objects for about ten years now, but fundamentally, the organizations that do the work are still firmly grounded in a sense of the world that is analogue, with at most a thin coat of digital veneer. Our staff are still daunted by the scale of the digital phenomenon; we still think in terms of traditional collection profiles (books in libraries, research data in repositories). And so on. (Notable exceptions are the beta sciences; they went digital very early on.)
More interesting than describing the past is looking at the future: what will it take to successfully organize digital preservation within a truly digital framework? With my co-panellists I came up with six criteria against which to measure a number of case studies and future initiatives:

(I am afraid I must leave you to ponder these overnight. There will be more tomorrow and the day after. At some point in time, I blogged semi-live. Now I am multi-tasking. Which is better overall, but introduces a slight time delay on the blog.)

But before I go: at the end of the first day, the National Library treated us to an Estonian cultural treasure that no occupying force could ever squash: Estonian singing: angelic, moving, wonderful. It was the Women's choir of the National Library itself that performed for us, that just celebrated its 20th anniversary. I cannot blog such beauty, but perhaps, if you look at this picture and close your eyes, you will pick up some notes ....

Finding ways to align internationally – the ANADP conference (1)

2011-05-23T09:08:00.000+02:00

Here is the first post from Tallinn, Estonia, where I am attending the conference Aligning National Approaches to Digital Preservation (23-25 May). As this once again is an international conference, I hope that the Dutch fans will forgive me for blogging in English this week. The title of the conference is perhaps a little bit deceptive – it is not going to be about aligning necessarily national initiatives, but really about taking cooperation and collaboration in digital preservation in whichever form to a higher, international level. Unlike, e.g., iPRES, which focuses more on technical developments in digital preservation and on specific projects, this conference takes cooperation and alignment as its focus – that is the stuff we all know we have to do in order to meet the digital challenge, the stuff that is mentioned in hundreds of big, fat reports (mostly in the “we should …” form – without talking much about how); it is the stuff my own organization (the Dutch Digital Preservation Coalition, NCDD) is made of – and yet it is also the stuff that is often so difficult and slow to realize at any practical level.

And even when we do succeed in cooperating on something, we mostly fail to look at the issues from a truly digital, global perspective. In the US, NDIPP will not fund projects that include non-US partners; in Europe the Commission has the same policy – funding for European partners only. The idea for this conference came from Educopia, the people behind MetaArchive. Martin Halbert of North Texas University told me that one of the reasons for organizing the conference was that too little news about what is going in Europe tends to get through to the US stakeholders. For me, being a Dutch person, it is not always easy to imagine what it is like to live in a country that is so big that the rest of the world does not come into the picture much.

This Monday morning there is a sense of some suspense in the organizing committee. I know that we’ve got a pretty good line-up of speakers, and we’ve been skyping for months on how to fit it all together (Skype, by the way, is an Estonian invention!) – yet I do not quite know what will come out of it. The organizers emphasize that this is the first conference of its kind, and we will have to see how it goes. It is easy to agree on alignment, but to really make it work is another matter altogether.

So, I say: let’s begin by giving everybody who says “we should …” a fine of some sort – I have read and heard so many of those in the past three years as coordinator of the NCDD, that I think we pretty much have our fill of that. We should standardize, cooperate and align – let’s take that as the premise from which this conference starts. No need to repeat the obvious.

Let us instead pamper those people who come up with practical, doable ideas.

twitter: #anadp11.

Tallin, Estonia: where the Middle Ages survive in the Old Hanseatic Town

and Russian influences are visible (above).

This is our venue, the National Library of Estonia (above), an enormous building, which looked rather Soviet to me, but turns out to be a product of re-awakening national awareness in Estonia in 1985. It is the largest building in Tallinn; upkeep is expensive and so the National Library rents out an entire wing.

(I made the collages at the top from the thousands of photos I took at DP events in the past three years, to adorn the conference website.)

‘Memento’ sparks optimism at closing of IIPC 2011

2011-05-15T13:56:00.001+02:00

The IIPC 2011 General Assembly (see previous blog post) came to a close last Friday in an atmosphere of optimism. Martha Anderson of the Library of Congress told me: ‘We did not realize it fully until this conference, but we are very close now to opening up and joining our collections. That is something we have always wanted to do within the IIPC, but we never knew how.’

Aaron Binns of the Internet Archive expressed ‘personal excitement’ when he talked about the Memento project at Los Alamos, which won the DPC Digital Preservation Award last December (see also earlier Dutch blog). ‘Up till now we were still very much working within our own silos. Now for the first time there is a potential framework to work with. Memento can help us realize shared access, which is the whole point of the IIPC.’ Rob Sanderson (aka Azaroth42) of Memento attended the IIPC meeting to report on the progress that is being made.

Binns also reported progress in other areas. Archiving social media and interactive websites remains quite a challenge for the community, he said, but ‘we are making progress. Facebook may be crawlable in the end.’

Another important topic at IIPC 2011 was quality assurance. Checking the quality of harvested websites is presently a labour-intensive manual affair. But Helen Hockx reported on promising work at the British Library to automate quality checks, and Binns expressed confidence that the IIPC community can build on those – although there are still a lot of open questions.

During this General Assembly the IIPC also reached out to its users, as reported yesterday. In retrospect I wonder if the discussions would have gone in another direction if the IIPC had invited humanities researchers in addition to social scientists. Are humanities scholars content with the contents of the web archives? But perhaps it is much too early to assess the value of web archives for researchers in general. As Sophie Ham of the KB, national library of the Netherlands, said during her brief presentation on Monday: ‘If we plant our seeds within the right atmosphere, our diamonds will grow bigger and bigger.’ The KB started harvesting in 2007, so its on-site access to 700 websites is but a beginning of bigger things to come.

During the meeting, the IIPC and the Dutch KB said their farewells to Hilde van Wijngaarden, who in the past ten years developed into one of the driving forces behind digital preservation research & development at the KB and who was an active member of the IIPC steering committee. She is moving on to head the library at the Amsterdam University of Applied Sciences. KB and IIPC will miss her.

Next year’s IIPC General Meeting will be held at the Library of Congress in Washington, DC, the date is yet to be announced, but it is probably going to be May again.

‘Researchers are used to searching in incomplete collections. For them, I guess, that is part of the fun.’

(Barbara Signori of the Swiss National Library comparing their web archiving efforts to a Swiss cheese)

Web archiving: the international arena (IIPC 2011)

2011-05-10T13:23:00.001+02:00

Last month we reviewed the Dutch web archiving landscape (blog in Dutch), and yesterday the international web archiving community descended on The Hague for the IIPC (International Internet Preservation Consortium) annual General Assembly. So, Dutch fans, for once a blog in English, to report on the kick-off, an open conference entitled ‘Out of the Box: Building and Using Web Archives’, which was held at the KB in The Hague on 9 May. Despite cuts in travel budgets, some 100 professionals attended the General Assembly, from all over the world. The day was divided into three sessions: Collection treasures; Web archiving short stories; and The use of web archives. Let me share some highlights with you.

Collection treasures – and how to organize the international treasure hunt

Early in the day, Gildas Ilien of the Bibliothèque national de France (BnF) articulated some fundamental questions facing (national) web archiving institutions: ‘The Web challenges our national boundaries and policies. What’s in? What’s out? There is a need to define consistent selection and scoping criteria while projects and events develop. And: who should take care of what is in between nations, or everywhere? What is the risk? What is the value?’ Ilien referred specifically to event harvesting. The challenges there are emergency, scope and collaboration.

Ilien’s case was illustrated by the 2011 revolutions in Tunisia and Egypt which sparked a buzz of event archiving activity. In Egypt, the the internet was first shut down completely on 25 January, but the moment the ban was lifted, on 9 February, the Library of Alexandria started crawling the web for information, pictures and video to document the Egyptian revolution. Across the ocean, the Library of Congress realized something was going on which deserved capturing. Abbie Grotke of the LoC (standing in for Kris Carpenter of the Internet Archive) reported how the LoC used the informal IIPC network to call around and find out who was capturing what. This led to an impromptu collaborative effort between the Library of Congress, the Internet Archive, the BnF, the British Library, the American University in Cairo and Stanford University, to capture the events in the Middle East. They did not only archive websites, but blogs and social media as well. As Arthur Thomas would say later in the day: ‘It is more accurate to talk about collections of stuff from the Internet.’

As inspiring as this impromptu collaboration surrounding events in the Middle East was, the fundamental questions posed by Ilien are yet to be answered: who does what and how do we collaborate to capture the web that is fundamentally international and interinstitutional, and where, moreover, events develop at a much faster pace than our institutional decision-making procedures can keep up with (emergency!). We will no doubt continue the debate on these issues at the upcoming Aligning national approaches to digital preservation conference in Tallinn on 23-24 May. The slide to the right (by Carpenter, Grotke, Ilien) frames the important questions to be debated, including the many legal restrictions applicable to sharing data from web archiving activities – and the question on how we can organize central operators (such as the Internet Archive, maybe) and a possible World Watch.

The many faces of web archiving were demonstrated by two young archivists from the Czech National Library, Zuzanna Kratochvilova (left) and Lukas Gruber (right). After the turmoil from the Middle East, they reported that web archiving allows a national library not only to capture the traditional media, which specialize in bad news, but also many smaller websites that concentrate on good news, such as Czech citizens sharing their hobbies on the internet. ‘Content is more important for us than format’, Lukas told his audience, and so the team organized a ‘blog of the year’ contest to allow the public at large to help decide which blogs to include in the national collection. And Zuzanna could not help but smile when she conceded that memories from their own childhoods had prompted the team to select a site on a popular Czech children’s programme for archiving. It is part of the national cultural history.

Tjarda de Haan of the Amsterdam Museum recounted enthusiastically all the efforts that are going into recreating a milestone website, ‘The Digital City’ (De Digitale Stad), which was in the air between 1994-2001 and which was designed to help the general public learn to use the new possibilities of the internet. A huge cultural treasure, which tells the story of internet acceptation in the Netherlands, was lost when the commercial interests which owned the site by then simply pulled the plug in 2001. The Amsterdam Museum is now doing everything it can to find whatever content has survived, period hardware, and whatever they can find. Their efforts include a true Gravediggers’ Party, to be held on 13 May in Amsterdam.

To conclude the ‘treasures’ section, Gillian Lee of the National Library of New Zealand reported on a small-scale project to capture the Canterbury Earthquake on 22 February, and Mark Williamson of Hanzo, a commercial web archiving company, showed how Hanzo archives Coca Cola’s marketing history.

Web Archiving Short Stories – a parade of what we’ve got

In seven five-minute mini presentations, a wide array of web archiving projects was presented to the audience. A worthwhile show for anyone still doubting the need for web archiving. The web is where our lives are happening, the web is where our lives are being documented, the web is worth preserving. The parade included the Swiss, Croatian, Dutch, British and American national libraries and the Rotterdam Municipal Archive, and was concluded by a great film documenting an Internet Archive K12 project to train students and teachers about the need for and use of web archiving. Memorable quotes from a high school teenager: ‘We don’t usually realize that we’ll be history in like … two years.’ and: ‘Fifty years from now someone will read a textbook and it might be about us.’

Use of Web Archives – is what we’ve got good enough for researchers?

Having captured all of this wonderful stuff from the web, the question remains how these treasures are being used. More often than not, copyright restrictions keep national libraries from putting the material online so it can benefit the general public. Which means that all that work is being done to serve researchers. What do they think about these collections? The organizers had invited social scientists to debate the question, beginning with Ralph Schroeder, Arthur Thomas and Eric Meyer (picture at right) from the Oxford Internet Institute. They recently completed a draft of a report entitled Web Archives: the Future(s). One of their conclusions: present web archives are harvesting content, especially html/http, while that part of the web is becoming more insignificant with every passing day. Increasingly the web is dominated by the social platforms, location-based services and other very complex two-way and peer-to-peer mechanisms, semantic web/linked data, which are extremely complicated in a technical sense and thus are hardly being archived at all. Meyers: ‘Virtual worlds are our social life’, social scientists want to study how the networks function, who is interacting with whom about what, and that information is not being recorded.

So much for taking pride in our treasures ;-)

Anne Helmond and Esther Weltevrede, of the University of Amsterdam (in first picture of this blog above), demonstrated the type of research the Oxford team refers to. They analyzed the Dutch blogosphere 1999-2009. How did the network evolve? What platforms are being used by bloggers? (increasingly Dutch platforms!) Etcetera. Studies on the workings of the internet itself rather than the content it offers.

The Oxford team had a number of recommendations for web archives: try to archive server logs; try to archive traffic itself (an ambitious project!); allow researchers to trigger (frequency of) crawls; work on mechanisms to organize small and big events archiving (let machines trigger the harvesting, perhaps); think in terms of collections and collections of collections rather than individual sites. Quite an agenda! But Eric Meyer had more down-to-earth advise as well for institutions wishing to promote use of their web resources: ‘The best way to promote use of your digital resources is to provide examples of meaningful uses by others.’ and: ‘Institutions must put more effort into advertising what they have got.’ Gildas Ilien also recommended keeping in touch with the general public. ‘Not many people will use the web archive, but they will pay taxes for it if they know about it.’

Inevitably, the question of selection came up in the discussions. How do researchers feel about selection? The Oxford team: researchers increasingly say: collect everything, because we do not know what the future will need. The fact that we cannot analyze and make meaningful uses of this material presently, should not stop us from collecting. Arthur Thomas drew a comparison with his old field: biology. It all started with people collecting specimens, often they were amateurs. These collections were incomplete but they often generated great ideas, such as evolution and classification. Gildas Ilien of the BnF agrees with this philosophy: ‘We should collect as much as possible – do not wait until you lose it.’

‘Can we please everyone?’ asked Helen Hocks-Yu of the British Library retorically when describing a BL project to encourage use of the web archives and get feedback from researchers. As yet, the question remains unanswered. But help is on the way, as represented by Paul Girard of the largest French social sciences lab (Sciences Po). He described an open source software project ‘Hypertext Corpus Initiative’ in which a community of researches and web experts collaborates to develop mechanisms by which researchers will be enabled to sieve the massive amounts of data from the internet and find exactly what they need – without them having to have too many technical skills. In so doing they hope to deal with the quantity/quality dilemma. [NB: a corpus consists of web entities.]

The panel: ‘Document everything about your web archive’

The concluding panel (photo at the top of this blog), ably chaired by Martha Anderson of the Library of Congress, included the researchers mentioned above and two web archivists: Birgit Nordsmark Henriksen of the Danish National Library and Cathy Hartman of the University of North Texas. The Danish KB both carries out an annual .dk domain harvest (without quality checks) and more selective web archiving projects with better quality assurance – researchers specifically ask for experimental sites to be harvested. The University of North Texas started harvesting government websites when nobody else was doing it (that is vision for you_, and in so doing it gained an important position in the US web archiving community. Cathy explained that their web archive has now become an important part of the university’s identity: it serves the curriculum and attracts students.

In the discussions it transpired that what researchers need most from web archives is to know What is In the Box. How was the collection assembled? What was included and what not, for what reasons? Which harvests failed? Which breakdowns occurred at what times? Even curation activities and preservation actions should be carefully documented. All in the interest of sound research. Preferably, also, researchers should be enabled to use their own tools to analyse the data. And Paul Girard dreams of an index of all web archives in the world.

Dark Archive or Online Access?

As mentioned before, the contents of web archives are mostly restricted to no or only on-site access, because of many copyright and privacy regulations. The institutions are not happy with this situation, but they are reticent to break laws – no matter how old-fashioned they may be. But Martha Anderson had some encouraging news for the group. The Library of Congress has kept meticulous records of the use of its web archives over the past ten years, and now it seems the lawyers are beginning to agree that opening up the archives may not be such a dangerous thing to do. We shall hope to hear from the LoC soon.

And the Dutch KB had good news as well: at the end of the day it opened up its web archive for the first time. As not all technical and legal problems have as yet been sorted, access is limited to 700 websites which have passed (manual!) quality control. 2,300 websites will be added as they too pass quality checks. In about three years’ time, 10,000 Dutch websites should be harvested regularly. As for on-line access: the KB will strive to provide as much on-line access as possible in due course.

The IIPC General Assembly continues this week with a number of technical workshops – I hope that someone else, who is more knowledgeable when it comes to technical matters, will report on those.

(See also next blog.)