Custom OAI-PMH integration
OAI-PMH is the Open Archives Initiative protocol that lets systems offer metadata and lets other systems harvest it. In heritage, libraries, archives and research, it has long been a standard way of making collections discoverable beyond your own system. Forum Standaardisatie recommends it for learning materials and cultural heritage. Appfront builds the integration: an OAI-PMH endpoint on your collection, archive or repository system, or a harvester that collects and processes metadata from others.
What is an OAI-PMH integration?
An OAI-PMH integration lets your system exchange metadata according to the Open Archives Initiative Protocol for Metadata Harvesting. Version 2.0 dates from June 2002 and is the version published by the Open Archives Initiative. The protocol separates two roles: a repository offers metadata, and a harvester collects it. There are six requests for this, such as Identify, ListRecords and GetRecord, made over plain HTTP and returning XML. Every repository must offer at least simple Dublin Core, and can supply other formats as well. It concerns metadata: descriptions of objects, documents or archive items, not the files themselves.
The protocol is designed for selective harvesting. Every record has a datestamp for its last change, so a harvester using from and until only retrieves what is new or modified. With sets, a repository groups its records, for example by collection, and with a resumptionToken large lists are retrieved in batches. A repository also indicates whether deleted records remain visible. In 2017, Forum Standaardisatie added OAI-PMH to its list of recommended standards, for offering and retrieving metadata about learning materials and cultural heritage. In the Dataset Register of the Netwerk Digitaal Erfgoed (Dutch Digital Heritage Network), managed by the Nationaal Archief (National Archives of the Netherlands), OAI-PMH is listed alongside SPARQL and SRU as one of the ways in which datasets are made available.
We build custom solutions because collection systems record their metadata in their own way. A proprietary database, an older collection system without OAI-PMH, or a system that does have an endpoint but does not supply the fields or formats a consumer asks for: an integration helps in all of these cases. Which metadata formats and sets you need, who wants to harvest your data and whether you want to retrieve metadata from others yourself, we establish with you in the first step.
Submission
Your metadata harvested.
Harvesting
Metadata from others, brought in.
Changes only
Selective by date.
How we build your OAI-PMH integration
We start with your collection and your goal: which system holds the metadata, which fields and formats you use, who wants to consume the data and whether you want to retrieve metadata from other repositories yourself. Then we map your fields to Dublin Core and the other formats required, and build the first flow: an endpoint for one collection or a harvest from one source. You check the result against real records, and where possible we test with a consumer to confirm the harvest runs smoothly. Other collections, sets and formats follow once that works well. After that, we keep the integration up to date as your system or consumers' requirements change.
System, fields, consumers.
Fields to Dublin Core.
One collection or source.
Go-live, monitoring, and maintenance when things change.
What an OAI-PMH integration actually does
The components below appear in almost every OAI-PMH integration. Which ones you need depends on whether you offer metadata, harvest it, or both.
Endpoint
All six requests supported.
Formats
Dublin Core and more.
Sets
Records per collection.
Selective
Only what has changed.
Validation
Invalid records reported.
Log
Every harvest traceable.
Who we build an OAI-PMH integration for
The integration is intended for institutions that want to make their collections or publications discoverable outside their own system, and for anyone who wants to bring together metadata from multiple sources. The more records and consumers, the more it delivers.
Museums and heritage
Collections widely discoverable.
Archives
Descriptions harvested.
Libraries
Catalogues exchanged.
Education and research
Repositories linked.
Technology and integrations
This page is about offering and harvesting metadata with OAI-PMH. If you are looking for a system for the long-term preservation of digital archives, see our page on e-depot software; for connecting to an e-depot in line with the archive sector's agreements, there is the DERA integration. If you are dealing with archive metadata under the new standard, read more about the MDTO integration. You can read about our approach on building software.
Why Appfront for your OAI-PMH integration?
A collection that can only be found in its own system, an export that goes to a portal by hand every quarter, or a harvest that retrieves everything again each time and stalls halfway: all of these cost an institution time and reach. That is what we build against: an endpoint that consumers can harvest themselves, changes only, and a log where you can see what happened.
Findable
Your metadata harvested.
Efficient
Only what has changed.
Visible
Errors in the log.
Security and privacy in an OAI-PMH integration
The integration only exposes the metadata you want to make public. We filter out internal fields, and personal data in archival descriptions that must not be public, beforehand, and where the GDPR or your archive's disclosure rules require something, we agree it with you. Every harvest is logged with its time and result.
The integration runs in a European data centre, with encrypted connections. If not everyone needs to be able to harvest, we restrict the endpoint to the consumers you allow. Passwords and keys are stored securely and are not visible to staff.
Frequently asked questions about an OAI-PMH integration
Questions that collection managers, archivists and information specialists ask before getting started.
The Open Archives Initiative Protocol for Metadata Harvesting: a protocol that lets a repository offer metadata and a harvester retrieve it. Version 2.0 dates from June 2002. Requests are sent over HTTP and responses are XML.
An integration that lets your system offer metadata via OAI-PMH, or that harvests metadata from other repositories and processes it in your own system. Consumers then retrieve what is new or changed themselves, and you no longer need to create exports.
Every OAI-PMH repository must offer at least simple Dublin Core, with the prefix oai_dc. You can also supply other formats that consumers ask for. We establish which ones in the first step with you.
No. In 2017, Forum Standaardisatie added OAI-PMH to its list of recommended standards, for metadata on learning materials and cultural heritage. Recommended means it is a common standard, not a 'comply or explain' obligation.
Every record has a last-modified date. Using from and until, a harvester requests only the records that changed in that period, and can limit the request to a set. Large lists are retrieved in batches using a resumptionToken.
A repository states whether deleted records stay invisible, remain visible temporarily or remain visible permanently. If they stay visible, a harvester can see that a record has been deleted and remove it on its side too. We discuss which choice suits you beforehand.
For institutions with a collection system without OAI-PMH, with an endpoint that doesn't deliver what consumers ask for, or that want to bring together metadata from several sources. If your system already has a good OAI-PMH endpoint, configuring it is often enough.
Want to offer or harvest your metadata?
Tell us which system your collection is in and who wants to consume the metadata. We'll show you how an OAI-PMH integration handles it.