A lot of stuff and not a lot of fluff: Aaron Silvers talks xAPI

tessello Chief Technology Officer Jonathan Archibald recently caught up with Experience/xAPI wizard Aaron Silvers to talk about the current state of the Tin Can community, why Amazon are so interested in the specification, and what the future has in store for DevLearn and beyond.
Listen to the podcast, or read the full transcript of their conversation below…


Jonathan Archibald
Hi, I’m Jonathan Archibald, the CTO at Brightwave. Today we’re going to be talking xAPI with Aaron Silvers of MakingBetter. Any of you who knows anything about the xAPI will know that Aaron’s been the driving force since the very beginning behind the xAPI, all the way through to today and is championing adoption of the xAPI. It’s great that we’ve got him here today and we’re hoping to learn lots from him.

Just a bit of background on why we at Brightwave are particularly interested in xAPI is because of our product tessello. tessello was one of the early adopters of the xAPI and we’ve really built out the product based on the xAPI so we’re especially interested about where the standard is going, how it can be used etc.

So without further ado I’m just going to hand over to Aaron who will hopefully introduce himself, tell us a bit about him, and what he’s doing with MakingBetter and all of that!

OK Aaron – over to you!

Aaron Silvers
Hi! I’m Aaron Silvers, as Jonathan introduced me, I have a small firm called MakingBetter with my partner Megan Bowe. Our company does consulting on and around xAPI, in addition to doing a lot of evangelism for adoption of the spec, as well as the groundwork for pooling together an international consortium to continue to drive evolution of the spec itself into international standardisation.

Great. And we’ve been following MakingBetter and especially the xAPI Camp events you’ve been doing. I know you’ve just run an event at Amazon and I was wondering if you could tell us a bit about that and maybe pick out a few highlights from it.

Sure! So just a few weeks ago we had a one day xAPI camp, open to the public, but it was probably half-populated by Amazon employees – they have several different learning organisations throughout all of Amazon.com. Everybody from Amazon Web Services has their own learning function, obviously there’s an HR function that kind of branches out and has some sway over the learning technologies that are used as infrastructure.

But then you have folks like engineering excellence, you have a whole customer service team – all of them have their own education or learning functions. Each of them individually was looking at xAPI already on their own. We ended up talking with Mark Oehlert who I’ve known for years, who’s more on the HR side of things, and we had the opportunity to consult at Amazon as our own organisation MakingBetter, basically to educate them about what xAPI was, and we created it as a community opportunity for *everyone* to get notice of what Amazon’s needs and use-cases were, and what kind of work has been done that would fit the needs they had.

So we curated an event for them – in doing so we all started to find out about just how interested different functions of Amazon were already in terms of xAPI, and the research they’d been doing. So they all ended up starting to have a conversation between these different learning functions that are usually pretty independent, all round xAPI – recognising there was an opportunity for everyone to get their levels set at the same time.
That in and of itself was a huge highlight – where you have different independent parts of an organisation actually coming together and realising they all have something in common in terms of what their goals are, and taking advantage of that synergy was a great win, both for Amazon and obviously for us!

For Amazon to basically come out and say ‘Look, we’re going with xAPI, we feel that it’s a key part of our strategic roadmap going forward’ is a huge endorsement of xAPI and the community at large. Here is a real organisation that we look to as an example of people who are innovative, and people who are doing excellent engineering work, people who really get customer service at a fanatical level, and for them to look at xAPi as more than just as a function of a learning department but also integrated into their work so they can do better engineering and do better software development and do better customer service. There’s a lot of great case studies to be mined there going forward!

Looking at the speakers who came that we presented – we heard from people who we had never heard of within the community before, who came in from the outside, outside of that xAPI bubble but very interested in talking about xAPI and its implications, and the plans that people are already doing with things like the Mckinsey Social Initiative, or Myra Travin’s work in terms of learning futures.

So yeah! There was a lot of great stuff out of that event and it certainly has catalysed interest in adoption since then, even in the few weeks we’ve had.

Great – and how did Amazon feel after the event? Did they feel that they really got a lot out of it?

Well you know companies tend to play things pretty close to the vest! But the feedback we’ve heard had been very positive and very constructive – that they are actually acting on pretty much everything that was worked upon during the day, so…!

Cool! So was there a standout revelation or use-case that was discussed during the day that really caught the imagination of the people there?

That one’s hard to say. I would say that probably the deepest revelation was the extent to which organisations have already been thinking about going with xAPI, and the depths to which they’ve been preparing themselves to go forward in that route.

And do you think as Amazon use the xAPI more and more they will be quite open and share what they’re doing with it, or do you think they will be quite closed about it?

It’s always hard to predict on those things, but I feel like with someone like Mark Oehlert at Amazon, and I’m not trying to put too much pressure on him (but I am!) – Mark’s the kind of guy who was instrumental in Advanced Distributed Learning (ADL) back in the late 90s where he was a contractor who was brought on to help the US government create this new organisation around learning technology and research and development.

So I think Mark actually understands quite well the significance of what’s happening with this collaboration, with this synergy, and I feel like Mark will share as he can the stories that are appropriate to share. And that’s really all we can ask!

Well I certainly hope they do, it will be really, really interesting to see what they can apply their huge engineering brains to and create with the xAPI.

You mentioned a few minutes ago about the excitement around the increase in adoption that other organisations are having with the xAPI. From your side of things, have you noticed xAPI adoption increasing over the last six months to a year?

Well, xAPI adoption has *clearly* increased over the last six months! Certainly sometime around November of last year we began to see in large part that there was at least a lot more discussion and interest that was coming from outside the folks who had been working on creating just the spec itself. Hearing from those folks and seeing their publications and really understanding thaechnically they were right on – like, we were seeing a lot of stuff and not a lot of fluff, which was great.

And then in the last several months there are a lot of indicators on our side that I would say indicate the interest and adoption is increasing. One, we’re getting more phone calls from a business perspective; two, we’ve had two sold-out xAPI Camps in a row, whereas a year ago when we were trying to get xAPI events off the ground we just *could not get* the interest all together in one place, and that was really talking about even a year ago.

So there has definitely been an uptick. Part of it is a solid community effort that has been for the large part very collaborative, and very open about its work and the things that were missing. And then the other part of it is that I think we’re past the point of all the hype around xAPI where people are being realistic about… Like, OK there’s a lot of work that needs to be done right now, that’s like a lot of manual stitching, in order to get something really solid out of xAPI, but there are immediate results that one can benefit from once they do the work. There’s a closer pairing of the labour that goes into it and the reward that comes out of it.

From our side we’re certainly noticing that the xAPI has definitely matured in terms of people thinking about it with our customers and clients – and we’re getting other areas in the businesses we work with coming to us and saying ‘can we use the xAPI for this, can we do this with it, can we put this in it etc.’.

It’s really exciting – it’s moving beyond, I feel, from the massive ‘We’re going to change the world / one system to rule them all’ type things, to actually genuine use-cases which are helping solve real business problems – and that’s really, really exciting.

In terms of adoption, ‘xAPI’ can be slapped on to a lot of products. I’ve seen authoring tools that are saying they’re xAPI compliant, a lot of LMSs are adding a lot of xAPI compliance to them, but some clients who speak to us don’t necessarily know how to make use of those things, or what that actually means? I was wondering if you had any insights into that.

Well I’ll be very blunt to begin with, in that anyone who says they’re ‘xAPI compliant’ is basically giving out a load of bollocks, I think that’s the British term for that!

There is no compliance for xAPI – there’s no law anywhere that says ‘thou shalt use xAPI’ so compliance is really… it’s not even the right word.

There are certainly folks who could say they’re ‘conformant’ to xAPI but that is as you’ve pointed out a difficult field to navigate around. For one, we don’t yet have very awesome conformance tests. Now that said, we do have pretty – I mean actually *really* – solid interoperability among Learning Record Store (LRS) providers. So when you take a look at who’s putting out an LRS – and there’s a handful of organisations who are creating their own Learning Record Stores, we’re even at the point where there are whitepapers around like how they get records from one place to the other, how they validate those records are the same and all that stuff, so from that perspective – from the LRS adoption – xAPI is pretty solid.

If you’re buying an LRS that is from a fairly known LRS provider you can go with pretty good certainty from a market perspective that you’re buying something that’s actually going to work, mostly because the work that goes into creating an LRS is a ton of work, and the definitions of how that stuff has to work is vague enough where people can do their own kind of implementations, but specific enough where there are expected behaviours that are spelled out in terms of like what an LRS is supposed to do. That’s one area where the spec is really solid.

What we’re finding about xAPI conformance on the activity providers side, is a little more I would just say… messy.
You have a number of tools out there – they’re not just authoring tools by the way, I’m very specific about the term ‘activity providers’ because xAPI is not just really about content – so tools or applications that make statements, they generally make statements that to the letter of the spec are ‘a valid statement’, but in terms of what information you might actually be able to get out of there that will actually be useful – your mileage is going to vary from tool to tool.

What we have been seeing is that there are number of mass market tools out there that are doing the best things that they knew how to do when they put the tool out, but they are not exactly keeping up with best practices.

Admittedly, figuring out what the best practice is is sometimes very difficult to make sense of or find!
And that’s one area where the spec community, as well as folks like Megan and I, can and will be doing a lot better on, in terms of helping to make sense of the landscape so that people can be doing more of the right things, or what we are saying are better or best practices, earlier on.

Great – that’s really good, and yes you used the right English definition there!

So let’s say I’m someone who works at a big company and I want to explore how I can use the xAPI. Where’s the best place to start? Is it the community, is it an organisation like you guys, how best is it to avoid the potential pitfalls that a non-xAPI expert might stumble through?

I think a lot of that depends on where in the organisation you’re sitting. If you have a strong technical capacity inside the organisation, the community is a great spot to go to first because you’ll easily find the people and resources that you need, the mailing lists and the forums are generally populated and operated and have a high amount of contribution from people who really do know what’s going on. The community is very good about self-policing the noise out of it. So from that perspective that’s great.

If you don’t have that technical capacity, that’s pretty much how we’ve been getting a lot of our work, which is with organisations who understand what xAPI can do enough to know that they want to do it. They’ve likely even started playing with it on their own, with the downloadable examples, or tools that easily integrate to things that they already have in house.

So, they have authoring tools and if those authoring tools publish to xAPI they’ll do that first, and then they’ll upload those, or play with one of the cloud applications for Learning Record Stores to track statements. Or, they’ll take one of the open source Content Management Systems like Drupal or WordPress and they’ll plug-in plugins that allow them to generate statements or even put in LRSs and…

That’s how they start. What they find very quickly though is on their own those efforts are insufficient to the goals they may actually have, or they start figuring out what their goals really are and realise they don’t know how to achieve them with the tools that they have, and that’s usually where we get called in. I don’t know that we’re the only people that people call, but I know that those are the calls that we get.

I know with the way we position tessello – we have done since the beginning – is that it does all the xAPI stuff in the background, and tessello is a next generation LMS and you don’t really need to think about the xAPI to get going with it, you just start using tessello.

As organisations get used to tessello we can then start to look at more ways to use the LRS functionality, build more activity providers etc., but I can see if you were starting off alone how it might be quite challenging so I think I’d definitely give you a call Aaron.

Well good! My Skype is free.

One of the things we’re looking at with tessello at the moment is we’re starting to dive deep into the data that we’re getting and looking at analytics, and then applying that to personalisation for the learning that’s delivered to learners on tessello. I’m seeing a lot of noise from lots of other companies doing this and it sounds really, really exciting and I was wondering – because you’ve probably spoken to lots of them – whether you could share some of the things that you’ve seen that are particularly impressive or exciting, and how you think this could really help build the personalised adaptive learning movement.

Sure! I would say that we are in an early stage for solid analytics around what we can really do with analytics out of xAPI.

For one thing, you have to get the data out of the LRS to actually work with it, unless you actually have control of the database that you’re building the LRS around, and for the most part most people in organisations don’t have direct access to the database.

They can obviously access the end point, and that’s basically how we at MakingBetter have been building out analytics packages and visual data dashboards and visualisations, is working directly off of the end point of an LRS, so that if people ever want to migrate their LRS provider from one to another they can do so without having to rebuild this visualisation, they just work with a different end point.

That’s one of the nice things about interoperability – you can have that level of abstraction and avoid the vendor lock-in, which is a holdover from the SCORM days that I’m very eager to kill.

The challenge though, what we’re hearing a lot of requests for, is the idea of a Tableau-type of visualisation package for xAPI. To do that, it’s not impossible but there’s a lot of work that needs to happen from a spec perspective and a community perspective, which is one of the reasons why a consortium is so needed.

We can take a look at the profiles that are defined by Communities of Practice (CoPs), or y’know as Rustici Software called them the ‘recipes’ – and that’s great, the fact that those things are even out there is fantastic – but it’s a lot of manual work to figure out how you even normalise the structure of statements so that you can do more general types of queries, or more dynamic types of interfaces as defined by what a CoP thinks is important to track.

Take ebooks, or annotations: there’s a certain amount of behaviours that a community is identified around, what one should be tracking or doing with an ebook. That’s codified to some degree as a formal profile of xAPI, where it’s like OK – so here’s the vocabulary that we’re using, here’s how we define our activities, all that stuff is great and all that information could be used to dynamically drive the expression of data analysis – which is what people are looking for, in a dynamic way.

But how those statements get structured, and the ways in which we use certain parts of statements that are optional – compared to what another CoP might do – that stuff is still very loosey-goosey. Which is one of the strengths of xAPI – its flexibility – but it presents a challenge when you want complete flexible tools that ultimately rely on a little bit more structure across statements.

Or the ability to understand what the structure’s supposed to be in this statement, and how that maps to a different context.

Which is a very difficult long answer to say, to do a Tableau-like tool takes a bit of work yet. It’s easier to create bespoke reporting for specific organisational needs based on the data that you’re tracking and the data you need to track. So that also requires a data strategy. And in the case of personalisation, that requires a) a lot of content and b) a content strategy, because it’s really difficult to personalise and make recommendations if there isn’t enough content to recommend an alternative, you know what I’m saying?

Yes absolutely. Whenever I start looking into all this stuff, the challenges of doing something generic for the whole xAPI are obviously huge – but I think can be solved in time. In terms of personalisation, you can have all the data you want, but if you don’t have the content, if the content is not good, the data’s meaningless.

It doesn’t matter what decisions you’re making on the data if you’re just serving up rubbish content to people that isn’t relevant to what they’re doing.

I’ll go one step further. Even if the best intentions and the best design around the content, if you’re relying on authoring tools to do all this tracking for you, you’re either constrained by the design decisions that they’ve already made about what’s important enough to track, or you could be left to your own devices, if you have a tool that’s flexible enough to allow you to track whatever you want. In that case, you need to make sure that you’re tracking the things that you want to track and packing it with the information that’s going to be useful for telling the kind of stories with the data that you’re going to want to tell later on.

One of the challenges we’re seeing with the people who we’re working with is there are a lot of mass-market authoring tools out there – and again it’s not all about authoring tools, there are plugins for CMSs that are creating statements as well – but they all use for example the verb ‘experienced’. Everything is something that someone’s experienced. That’s not very helpful in picking apart what people are actually doing, and how to make sense of it later on, especially when you’re looking at it en masse.

We have one customer that has 65,000 statements in their LRS, which sounds like a lot. But it’s nothing compared to the millions that you’ll have a year or two years from now as things scale. And, if you think that trying to make sense of 64-and-a-half-thousand statements out of 65 thousand statements saying that someone ‘experienced’ something is difficult, imagine what happens when it’s like 99% of 5 million statements!

Yeah – and a lot of the data from those things is essentially SCORM data but in a different format.

Exactly! So like you get to a point where you think you’re being really on the cutting edge and you’re getting all this great information, only to find out a year or two later when you finally have an analytics package that can try and make sense of it, that nobody can make sense of the data you’ve collected because the data you’ve collected is not helpful, whether it’s ill-structured, or it’s not sharing the right data, or it’s not sharing what you thought it was sharing but now you’re a year-and-a-half in.

That’s one of the things that we’re helping the organisations we work with: data strategy. Because we’re seeing already there are things that we need to do. We’ve enabled a lot of people to be making statements, but when you have an organisation like let’s say Amazon, you’re going to have an increase in adoption. A significant increase.

You’re going basically from ‘Early adoption’ in the Diffusion of Innovations bell curve to ‘Early majority’ where you’re going to have a ton of people in a very short space of time starting to work with xAPI, it’s really important that we start addressing and tightening up some of these things now.

I absolutely agree. So do you think these challenges can be overcome?

I absolutely think they can be overcome. In fact, I’ve been really pleasantly surprised at some of the solutions that are coming in from outside the community and then becoming inside solutions.

One of the ways in which people are addressing vocabulary collisions for example – because it’s a difficult enough challenge to work out which verb to use, let alone, ‘What if I have a slightly different context for that same word, in terms of how I’m tracking?’

Well, there’s a lot of talk about using a technology that’s been well known for years called RDF , which is a very complex technology for us to work with in and of itself to do the things we’re doing with xAPI, but in terms of what it can do in terms of helping us figure out what language to be using when we track things, RDF is extremely powerful and has a very widespread federated architecture that makes it very easy to help suggest within authoring tools that employ the technology what internationally, what all over the world are people doing to talk about these kind of statements? What are similar types of vocabulary that they can use? In which context would it be appropriate?

All of that stuff can be de facto standardised and to a large degree automated – taking a lot of the guesswork about when someone’s trying to create content or create an experience and they want to track something, making it easier for them to figure out what they should be tracking, how they should be tracking it.

Sounds great! So, we’ve talked a bit about the past, the future etc. but what’s next in terms of the community and xAPI camps and MakingBetter and things like that. We’ve got DevLearn round the corner – are you guys doing anything for that.

We have an xAPI camp happening at DevLearn , we have so far probably anywhere between 30 and 35 people in the room, and we’re still a good month and a half of registration happening before the event, so that’s likely to be a pretty good sized event again.

We’re looking forward to that – we have a brand new set of speakers – some people are coming back but we have some new folks coming in. In fact we’re just putting out the schedule today or tomorrow. But you’ll see folks from ADL Andy Johnson and Craig Wiggins, you’ll see CEO Dr Ben Betts from HT2 – I like making fun of him.


I know, nobody ever makes fun of Ben! Sean Putman’s going to be there from Alter Engineering, and TJ Seabrooks from Rustici Software who’s going to be talking a lot about activity providers as well.

And then we have folks like Art Werkenthin who’s going to be talking about CMI5, we have Anthony Altieri who’s going to be talking about Internet of Things, Russell DuHon from Saltbox is going to be talking a lot about looking across what their work is, and seeing basically five patterns of how people are starting to do work with xAPI and he’s going to help accelerate for folks who want to get started, like ‘here are some things you can easily do’.

And then we have folks Brian Dusublon, Nick Washburn, Megan Torrance from Torrance Learning who’s going to be talking about how to get started from an instructional design perspective and obviously Megan and I are going to be there.

So we’re seeing folks register from places like Amazon, Autodesk, Google – so we’re getting a lot of interest from a pot of different organisations and people are coming from far and wide, even internationally to xAPI camp.

Well we’re certainly hoping to be there and I can’t wait myself – it sounds brilliant!

Right, I think I’ve taken up as much of your time as I can today Aaron. It’s been brilliant to hear everything that you’ve been saying – some real passion and insight there that I hope will inspire the people listening to adopt the xAPI or get in touch with the community and really get going with a truly great standard!

Well I just hope it translates to the proper English ‘passion’, so…!

The biggest brands learn with Brightwave.