Solving a customer service crisis with rapid development

The RiksTV TV network is in the middle of a transformation, as internet streaming replaces traditional broadcasting. In this interview their Platform Manager, Anders Kalland, tells us how they solved a customer support crisis: by building exactly the tools the customer service team needed, in record time, using Anvil.

You can listen to our conversation on RiksTV’s episode of the Anvil podcast, Stories From the Workshop, or read the full interview below.

So, who are RiksTV, and what do you do there?

We’re a pay-TV service - the third-largest in Norway. I’m the Platform Manager, and my job is running as fast as I can to keep up.

I used to manage all of the back-end operations for our streaming service, but the teams keep growing with the services we offer, so now I only do most of it. Traditionally my role has been everything from designing solutions for how everything is supposed to work, to ordering technology, to quality assurance and specifications for third party developers.

How does TV broadcasting actually work these days?

Originally we were purely terrestrial, transmitting to old-fashioned antennas outside. But over the last 10 years or so, like the rest of the industry, we’ve started streaming online and on IPTV. Now you can access RiksTV via your TV, on your web browser, or via an app.

We get a live feed from each TV channel we carry, like NRK. It gets transcoded from a high-quality “contribution format” to a consumer distribution format, which will play on a normal TV. For traditional antenna delivery, we encode it in what’s called a Statmux, and we send the Statmux around the country to different transmitters - basically big antennas on top of mountains. You will have an antenna on your roof which is pointed at this antenna, which pulls down this signal and decodes it and shows it on your TV through a set-top box or in-TV decoder. That’s the traditional broadcasting setup.

With internet streaming, we take the same source contribution signal and run it into a different encoder, to create multiple video qualities. For internet broadcast you need to make room for different bit rates (qualities) so that if you’re on a bad internet connection you should at least be able to see something, and if you’re on a good internet connection, you can see something good. We then take those 4-6 different qualities and package them in a standard streaming format (DASH or HLS) and push it to a CDN. The client can then download it from the network onto the mobile device, computer or TV.

So the device is pulling the data from the network, rather than the network sending it?

Exactly. The device chooses the best video quality it believes it can download without causing any interruption to your viewing experience.

How much data do you ship?

We’re talking petabytes per month. We’re streaming twice as much this year as we did the year before, and it doubled the year before that too. So it’s a substantial growth in the amount of content we’re pushing out as people are getting more and more used to using streaming services.

Today more than half of our users are able to stream, but on the whole they still use whatever they used before to watch TV. But if you’re watching RiksTV through a set-top box, then as a consumer you won’t necessarily know when you’re watching a broadcast or a streaming service - it’s just seamlessly integrated.

That’s a fast rollout! What challenges did you encounter?

We rolled out the whole streaming service over only a few months. Suddenly we had many thousands of users using these services, and our customer service centre was struggling to keep up. We didn’t have the tools in ready for the customer centre to troubleshoot ahead of time. In the beginning it was a case of doing the best you can to provide service. I was providing direct service myself, troubleshooting for the team.

We collect a pretty substantial amount of data. So we potentially had a lot of insight into what might be failing, but we didn’t really have any tools for figuring out exactly what was failing for a specific customer. We were using command line tools to figure out if something was working or not for an end user.

Let’s say you are an end user and you’re trying to stream something from our service and it’s not really working. You get a choppy picture quality and it’s just not a good experience. There could be a multitude of reasons for this happening. It could be we’re having some back-end issues, or our CDN network is struggling, or your internet provider is having some issues, or you’re having some internal network issues, or maybe you’re just on a bad Wi-Fi connection. It’s very difficult for a call centre worker to troubleshoot where the problem is when they’re working blind, with only the information from the guy on the other end of the phone – especially when you consider that they might be pretty old or not know what set up they have either.

And so the customer service team was escalating this mystery to you. How do you tell what’s going wrong?

Well, we collect lots of metrics from the devices, so it’s simple to see what scale of problem you’re dealing with. Is the device’s wifi connection bad? Are there are just issues for this customer, or all customers on a specific ISP, or in one city? If it’s the entire country, you can kind of assume the problem is…farther back in the stack.

There’s probably smoke rising from a data centre somewhere.

[laughs] Yes, exactly.

So, you have all these customer service enquiries being escalated to an engineering team. Clearly that’s not going to scale. What did you do?

This is where Anvil comes into play on our end. I thought what we needed was to build some kind of user interface on top of the data, so that a specific customer service representative could get the information they needed to help a specific customer.

I thought a bit about what we could do, and there were three main options. One option was basically to hire a bigger engineering team to handle all these cases, which of course is extremely expensive and also time-consuming in terms of training.

Another option was basically outsourcing it, putting together some kind of development team and having them make the solution. Those would be the normal ways to do it, but they would be pretty time consuming for me because I would have to spend a lot of time writing user stories, describing what this system needed to do. And of course I would have needed a small team of at least a couple of developers, maybe a front end developer and a back-end developer, to build this thing.

So I chose the third option, which was to try and make it myself. And that’s where Anvil comes into play.

So what’s your background? Did you have the skill-set for building a web app yourself, or was this a leap into the unknown?

I’m not strictly a developer. I had some experience with coding: written some Java in University, some mathematical programming languages and so on, but not really a developer. I had some minimal experience with some scripting and knew a bit of Python, but I was by no means an experienced or full stack developer.

To make this kind of solution you would traditionally need a full stack developer or a small development team, so that’s when I turned to Anvil, because it provided basically everything I needed straight out of the box. For me, it was the perfect tool: it allowed me to build a front-end on top of a back-end in a language I almost knew - with no help at all, really.

How long did it take you to build the first version the customer service team could use?

I think I built the first version of the front-end UI with Anvil in a week or two. Maybe a week.

Actually, first I had to work on a back-end component, which had to be in node.js because it used some existing libraries. So I had to learn Node to write that back-end API, and that took me maybe two or three weeks. And then I started looking at Anvil.

One reason Anvil fit so well for my use case was I needed to have secure authentication against my back-end, and Anvil allows you to just add their authentication code, which made it extremely easy. I remember asking you for some help on the way and you were extremely helpful.

What does your solution look like?

There’s four integrated data sources: two different ElasticSearch clusters, a third party service, and there’s also a service for a direct connection to end user devices to run simple diagnostics on the end-user devices. Pulling the data together makes it as easy as possible for customer service. When they get a call from a customer, they will load up this dashboard and insert the customer number. It will find all the devices for this customer, do some simple diagnostics on them, and provide the customer service rep with the output before they’ve really started looking at technical stuff with the customer. If it’s connected to a bad Wi-Fi, and the user is starting to complain, they don’t have to do a long diagnosis - they can just start working on how to fix the Wi-Fi issue.

Are you using Python to pull all this together on the server?

The back-end components are talking to all the elastic search clusters and the specific API to connect the end user device for diagnostics. All that data is to basically just pulled in and then displayed for the user on the front-end. We can just use off-the-shelf Python libraries to do those things – it’s actually surprisingly easy. It’s just taking off the shelf stuff and using it in the right way.

Do you have an idea of how the process compares with using traditional tools?

It would have cost a lot more, because you would have a development team working on it. So using Anvil actually saved a whole development team. It turned me, a somewhat nervous wannabe Python developer, into a full stack developer, which is pretty impressive.

Have you introduced anyone else at RiksTV to Anvil?

I introduced another broadcast engineer to Anvil, who also had a pretty minimal background in programming. Of course, it was extremely fun for him to work with and in a couple of weeks, he was adding features and improving on the tool.

Were there any surprises that you discovered?

I set out not knowing exactly what I wanted, so I knew there were going to be surprises. It was a lot of experimenting: does this work; is this valuable; does this give us anything? And if not try something else. It’s the normal iterative development process. You never know exactly what you want, so it’s very good to have a fast, easy way to try something and if it doesn’t work try something else.

And specifying that for an external dev team would be a lot slower.

Yeah - it wasn’t a tempting prospect at all. It was exactly what I didn’t want to do. That’s why it was so nice actually doing it yourself, in such an easy to start with development environment.

Could you say why you would recommend Anvil?

It’s just extremely easy to get started and to make something. It makes any developer into a full stack developer. You don’t really have to worry about hosting, you don’t have to worry about your deployment pipeline. You can just write some simple code and then start using it.

What do you need to build?

How fast could the right tool solve your business problems? Try Anvil, and find out for yourself: