What we’ve learnt so far at BMJ Group

Q: What have been your key learnings?

A: When we started exploring GenAI, we worried that our own use could lead us to publishing inaccurate information (eg. were we to use summarisation to support the creation of marketing materials). That would be a significant risk for an organisation whose reputation depends on the trustworthiness of our content. We created governance around GenAI use-cases, with a focus on managing risks like this. We have since approved about 17 use-cases across different parts of the organisation, but this initial worry has not materialised. Indeed, the way the tools are being used, carefully and with intent, has meant that this risk has not emerged as feared.

We thought that there might be a case for training a language model on our own content for BMJ-specific use-cases but over the past year, model capabilities have continued to improve and we are making good progress with the out of the box models that are publicly available.

We have also learnt that while a market has emerged for licensing content to LLM providers, this market remains nascent. It’s clear that it’s going to be hard for smaller content players to play in this market at this point in time.

I would say, overall, the thing that we are finding most of a challenge is a combination of navigating the large number of tools that have emerged, as well as figuring out how to support folk in getting started with these technologies. Nearly 70% of folk are confident in being able to identify a GenAI tool, but 35% are confident in using them.

Q: In which use-cases have you had the best results?

A: We have seen a lot of success in supporting our teams with software, from the less formal aspects of that through to the more formal aspects of development.

I’d like to use an analogy from Roy Bahat — the head of Bloomberg Beta. He asks whether AI is like a calculator — something that helps us do what we were doing before, but better; a weaving loom — something that replaces human effort; or a crane — something that allows us to do things that we could never do before. We are in the strong calculator phase right now. These tools are helping us do things we could already do before — but faster and better. They are not replacing our engineers — at least not yet. I hope that they will allow us to do things that we couldn’t do before, and I am optimistic about that for the future.

At the more informal end, many folk across BMJ are finding them incredibly useful for helping with things like getting Excel formulas working well (by any measure, Excel is the most successful programming environment of all time).

On a slightly more formal level folks in our production department who work with XML are finding these tools useful for creating and validating XML, and Schematron, and those working with Salesforce have been able to make macros to help with their workflows, a technical task that they would just not have been able to do on their own before.

For the more formal areas of software development, our front-end teams are all using Github Copilot — which is a GenAI tool that integrates with the tools they use to write code. It’s become an essential tool for them and is used daily. It significantly speeds up code boilerplate generation, streamlining code refactoring, documentation, unit test generation and fixing errors. Critically, they are not asking it to create the core algorithms, or code structure, but as an assistant, it’s giving them a big boost.

Three best practice top tips

1. Allow access to the tools. For the more informal use cases, those came about from folk in BMJ picking up and using GPT or Gemini, and trying it out, so you have to create an environment where folk know that they can use these tools, and in what context.
2. Be willing to pay for the right tools. For some integrations — like Github Copilot — you need to pay to access those tools. The cost is usually around $20 per person per month. You just have to be willing to do that.
3. Get folk to share their experiences. If the tools are useful, they will get adopted. Personal experience is one of the best ways to validate usefulness, so get folk to share their experiences with each other. I have an occasional CTO email that goes out to all of the technical folk in the company (every two to three weeks), and I share any good stories that I hear. We also have a group in Google Chat where folk share experiences or pointers to interesting things going on outside of BMJ.