sessions, I have felt more fear of missing out, instead of being really joyful about
the session that I've chosen.
But I'm very happy to announce to you that you are among the lucky ones who have really
chosen the correct and the most appropriate session of the entire conference.
And the reason why that is not a joke is that the standards and protocols like SORT, like
OAIPMH, are actually one of the few areas where repository communities like DSPACE and EPRINTS
have actually succeeded in aligning on a standard and working together.
So the things that you hear here this afternoon will very likely have an impact on not one
of the platforms, but on all of them.
All of the fine gentlemen, unfortunately no women will present here, they have done so
much in their lives that I will not have the pretense to know what's most relevant.
So if they want to add some details about their pets, dogs or professional careers,
I would leave it up to them.
But the first speaker is Peter Sefton, thank you.
Hello everybody, can you hear me?
Yeah, you can, good, a bit of life in this room, that's excellent.
So this has my name on it, but obviously a lot of people did the work, have done the
work, I'll try and call out people by name as we go, but there's also some Git repositories
and things where you can actually look at what the contributions from other people were.
This is a really interesting interoperability thing that I wanted to make sure I had the
latest version of this presentation and this looks really different on my version of Chrome,
on my computer.
I'm going through this emoji thing at the moment which some people already know, but
I wanted to tell you a love story.
So this is a story about interoperability, but it's also a story about a love affair
between two different standards that have been brought together.
So the first of those standards is the one that was on the program, so that's Data Crate.
Data Crate is a specification for describing research or other data in detail, I'm going
to run through what that's about, but that's an effort that I've been leading with a team
mostly based in Australia, and somewhere where they have a much better marketing budget and
have much more compelling graphics to go with the standard is the research object work.
So research object, and I'd just like to acknowledge Stian down here who's taking a picture of me
at the moment, so that this is a large collaboration as well, but the point of this one is, is
to do with essentially revolutionising scholarly communications, like lots of us want to do,
and it's about packaging up research data in a way that you have as much context as
possible so that you can put in things like an article that goes with a dataset, this
particular focus on reproducibility and on packaging workflows in with the data.
So these are two different standards, born at opposite ends of the planet, and there's
going to be a wedding, but before we go into the wedding bit and we talk about what life's
going to be like with research objects and Data Crate under the same roof, I'm just
going to go back a little bit and talk about, this is a Data Crate talk, so I'm going to
talk a little bit about the Data Crate specification to give you some context, and then talk about
what we're doing together.
Okay, so we, with Stian and Owen up here, we ran a workshop on Monday about data packaging,
it had lots and lots of contributions from the community, and as part of the preparation
for that workshop, we put together this sort of list of things, it's a contrived list because
all the words start with S, so it was just a sort of device to structure the discussion,
but this kind of summarises about what we're trying to do with data packaging.
First of all, we want to have a really strategic approach to this, we're not talking about
just putting a bunch of files together and making sure that they are not out of date
or corrupted, you can do that with things like Bagot and the new OCFL object spec.
It's about a strategic approach to transforming scholarly practice, which means we have to
come up with ways of describing things really well, knowing how to include code, giving
people guidelines for how do you do metadata around code, how do you talk about versions
of software, how do you include workflows and things in the package, and some principles
about what this packaging effort should be like, so things need to be as self-documenting
as possible, so you want to have a way that people can understand a self-contained bundle
of stuff that you might download as a zip or a tar, or you might find parked on a web
server somewhere, so it doesn't have to be zipped up, but a web person might have put
it on the web somewhere, or you might find a directory on your hard disk that you've
forgotten about, and it would be nice if that actually self-describes, so there's something
like a readme in there that you can look at.
We want to make sure that stuff is kept safe from a preservation point of view, so we do
want checksums and so on, and we had discussions about serialisation syntax and so on on Monday
in some depth, but we think, like a lot of us working in this metadata sphere now, have
settled on using JSON, which is a format that's very easy for any programmer to work with,
particularly JSON-LD, which is JSON with linked data properties, which means we can really
improve the quality of metadata.
The only person who's been to every one of these conferences, and back in the early days
it was all about collecting string metadata for things, right?
You type an author's name into your eprints or your dspace or whatever, and we know the
problems of trying to identify people using string's cause, there's multiple forms of
names, there's misspellings, a lot of people have the same name, or variants of the same
name, so that doesn't work, but using linked data does.
We describe a person using a good ID, like an orchid, it's the obvious one to use for
a person.
So these are some of the principles we're working with.
This is what data crate looks like.
So data crate's pretty easy to find, you can find it on GitHub and you can find your way
to this example.
This is a sample dataset that I put together with a couple of photos in it, just to show
how data crate works.
And if you come to this on the web, you see this, it's like a landing page, it's not being
served by a repository, it's just being served by a plain old web server, and you can see
about this object in HTML, right?
So that's one of the things that data crate brings, is that it's self-describing and it's
self-contained, because you can also click on the thing that says download a zip file
and get a zip file, and inside the zip file is a directory, and this HTML page is in there,
so you can do that.
And I think this is really useful, because I've had this experience of having data downloads
stuff I've got from the internet, sitting on my hard disk, either zipped up or not,
you open it up and it's got stuff in it, right, CSV files, whatever.
Sometimes it has a readme file, the readme file may or may not have really good metadata
in it, but it's a lot of work to have a readme file that describes every file and every piece
of a data package.
That's actually really hard to do.
So that's what I was just saying.
So you download this thing, it's got on your hard disk, there's an HTML file there, you
can click on that and you'll see what I just showed you.
There's also JSON in there, which you can feed to your computer, and the JSON looks
like this, so this is sort of human readable, but it's not designed for human consumption
really, but this is exactly what I just showed you in HTML format, so this is the machine
readable view of that splash page that describes this dataset, which has two pictures and some
stupid random text files in it, just as a demo.
So what you get is an interoperability failure on a grand scale here.
This, if we had interoperability, would be a picture of a computer, and this would be
a picture of a woman scientist with brown skin emoji, right?
So that's a person, this is a machine, and just showing that there's sort of two different
views of the same information.
So let's dig a little bit more into a data crate package.
Inside the data crate package, you can, if you wish, or if your computer wishes, describe
individual files.
So this is a description of a file, there's a picture in there which is of a fence with
an out-of-focus dog in front of it, and there's also all of the EXIF metadata, so that's the
technical metadata that the camera puts into the image file, and there's a couple hundred
fields in there, and if you click on those little arrows, then you can expand it out
and see all the EXIF metadata.
So just put that in there as an example of how you might describe a file, but more importantly
and more interestingly, if you're looking at research workflows, you know, science or
other research, what would be interesting is to be able to map out these kinds of relationships,
right?
So you can read this.
We have an image in the centre, which is that picture, and then there are, so up on
the top, I'm at the top, the most important thing, there's a create action.
So this is just schema.org, which is something the search engine has invented for selling
new things and making sure you can find movie tickets at the right time, but it actually
has enough to do basic provenance.
So there's a create action, sorry, the right-hand side is the one where I took the photo, create
action, made the photo, I can reference what the picture is of, a place called Catalina
Park in Katoombu, near where I live, and if you dig into this, you can go and find this
dataset.
If you dig into it, it's got a description of Catalina Park and uses a geonames URI for
that place, and it gives you the coordinates of the place.
You can see what I used, I took out my antique camera for this one, and you can see, you
know, what were the individual pieces of equipment that we used to create this thing.
This could be really interesting in a scientific context.
If that's a microscope, it would be a particular brand of microscope with a particular lens,
and you would be describing filters inside the optics, filters and a light source, and
filters in front of the light source, and a whole lot of other stuff like solvents and
so on.
So you could have long lists of things that give you context and provenance for the thing
that was created.
And over on the other side, I've got creating a sepia tone version of that using an application
image magic, and if you follow the link to that, you'll get the version of the software
and actually the command that was used to generate the part.
So that's a trivial example, but there's lots of places where we want this metadata.
And expressing metadata in that level of detail, you can't really do very easily using
the way we do it on a, you know, the front page of a, well, you know, when you're doing
a DSPACE repository type thing with name value pairs.
You run out of steam when you're trying to say this thing was created by this other piece
of software with this version.
You can do it fairly well in linked data.
One of the nice things about schema.org is the documentation is pretty complete.
The descriptions of things are not always that great.
Go and look at the description of description in schema.org.
It's not perfect.
But every screen in the data crate has a little question mark on every field, so you can go
and you can see what this stuff is all about.
So if you don't know what a create action is, you can go and follow up on it.
And I'm just showing there where it says type create action.
That's the bit that was on my HTML page, and if you click on the little question mark,
it's going to get, okay, right.
So a couple of points to make here.
We decided to use JSON-LD, but there are some problems with that.
So this top sentence there says that tooling is limited.
So there are some libraries for processing JSON-LD, but we've just been sitting this
afternoon and sort of working through what advice we can give to people about how to
do this, because the libraries aren't quite as good as we'd like to be able to do.
And given the interoperability problems, I can't actually understand my own slide.
This one is saying that the main thing on this slide is a warning
against architecture astronauts, which is Joel Spolsky's phrase.
But there's a lot of activity in this sort of linked data world, and I know some of the
people who have these tendencies are probably in the room, so please don't take this personally.
But when you get into some of the standards that are in our space, like OAORE, for example,
there's a way of building lists of stuff that belong together on the internet.
The way that's actually expressed and the complexity of the way some of those standards
work is not suitable for ordinary programmers who work in my department.
It's too complicated for me to understand.
So we're trying to hit a level with the way we do it, use our JSON-LD and so on that is
understandable and usable for programmers, and we're working really hard in this.
Now back to the wedding, where we're bringing together these two standards.
So the new family name, both parties are changing their name, is ROCrate.
So that's research object and then the crate bit out of data crate.
So this is just a snapshot of the ROCrate page, and it's really bringing together, I mean,
this is really driven by the research object concerns, because they were bigger than the
data crate concerns.
Research object is about ways of writing really good metadata that make it easy to
disseminate and distribute research in a way that it can be rerun and you have lots of
really good context.
So this is a work in progress.
At our workshop on Monday, we collected a number of use cases from people, thank you
for the people who participated, yep.
We collected a whole lot of use cases about how repository managers and a few others
from the community would like to, what you want out of a standard like this.
So now is your opportunity, you can come and see me.
The other two have to leave this afternoon, but you can come and talk to me for the rest
of the conference and tell me your use cases and we can help you capture this, or you can
go to GitHub and you can type it in.
Please read to see if somebody else has already put in something similar and leave a comment
rather than just making 16 copies of the same thing, but that's ...
So the other thing we're doing is doing a rewrite.
So we've taken the data crate spec that I talked about before as a basis of a new ... this
new arrow crate standard and there's a bunch of activity at the moment where we're simply
simplifying that, taking out the things that were confusing and aligning it and making
it more strategic really, sort of aligning it with some of the research object goals.
So what we're working on is ... just guidance for somebody to ... like a recipe to say if
I want to describe a data set, how do I do it?
So how do I link to subjects, how do I say this is a photograph of a fence or a dog,
how do I do the things I showed you, right?
How do you include workflows and how can I do things with this data set?
So the final bit is just if you would like to use this stuff, you can wait.
We should have a sort of alpha version of this out in October for eResearch Australasia
conference in Brisbane, in Australia, so you could wait or you could join in.
So if you want to come and put in issues, put in use cases, you can do that or you can
join the team by going there.
That's it.