Elite Bundesliga VARs — how do they make decisions?

Tobias Bauch, technology and innovation manager at DFB Schiri GmbH — a subsidiary of Deutscher Fußball-Bund (and Deutsche Fußball Liga) that is responsible for elite German referees — asked himself this precise question. To help answer it, I provided some support with a trial he decided to run at the Bundesliga’s video assistant referee (VAR) center in Köln (Cologne, Germany). He wanted to see if Tobii’s technology could help him gain insight into how VARs make timely, accurate decisions and, if so, use this information to train new VARs and maintain standards in German refereeing.

In this post, I am going to talk about VAR, the skillset needed, and share a conversation I had with Tobias about how he came up with the idea to use eye tracking, what findings he discovered, and what he intends to do now.

Rapid decision making

Opinion surrounding the introduction of VARs at top-level international and top-division games of the world’s favorite sport — football (or soccer if you prefer) — has been mixed. For the referees, VAR provides an additional layer of security; it improves the accuracy of officiating and helps prevent mistakes that might otherwise impact the outcome of a game. Some fans, however, feel that VAR causes delays, interrupting the momentum of a match. Whatever side of the line you land on, the ability to make accurate judgment calls quickly is an essential VAR skill.

Unfortunately, it can be hard for experts like international referees to articulate their skills because they are often unaware of the underlying decision-making processes. VARs may feel like they act on instinct, but the reality is that they often act on years of experience and practice. The complexity of articulating such skills makes training novices and other experts seem like an impossible task. That’s where Tobii comes in. Because our technology can visualize what an expert does on the job, it’s ideal for helping experts articulate their thoughts and actions. And because it captures what a person pays attention to in a systematic way, our technology makes it possible to identify behavioral patterns that can be used for many purposes, but for Tobias, his aim was training.

Meet Tobias Bauch, technology and innovation manager at DFB Schiri GmbH. Tobias has been at the DFB for about 18 months now. His role is a new one, highlighting the significance the DFB has placed on the use of tech in football officiating.

Q: What made you think about using Tobii’s technology to gain insight into VAR behavior?

A: I am not a stranger to Tobii and eye tracking. In my previous position at the Max Planck Institute, we used the technology to conduct research in the field of computer vision. Our goal was to teach computers to see and understand humans and their behavior. So, when I thought about trying to understand VAR behavior, it was quite an obvious choice for me to turn to eye tracking and Tobii to see if we could gain some insight into how VARs do their job.

Q: So, tell me about VARs, who are they, and how do you become one?

A: In the Bundesliga, all VARs are practicing or former elite football referees. We tend to rotate their schedules so that they are on-pitch referees one week and VARs the next. Some referees specialize as VARs after they end their careers on the pitch. VAR training and refereeing with the support of a VAR is an essential part of our continuous referee education.

Q: Tell me about the operations room. What does it look like?

A: For the Bundesliga, we have one operations center located in Köln (Cologne). In the room, we have six workstations. Each station comprises four positions one for the VAR, one for the assistant VAR (AVAR), and two replay operators. The VAR has two screens. One main screen shows the feed from camera one (the primary action camera), and a second screen placed underneath shows a split screen consisting of four different camera angles from the other cameras in the stadium. The system is programmed with a three-second delay between the two screens to help the VAR make rapid decisions. If they spot a situation that requires attention, they can immediately look down at the second screen to view the replay.

Q: What is the replay operator’s role?

A: The replay operator isn’t part of the decision-making process but is there to support the VAR and AVAR. They operate the equipment, presenting the footage that the VAR and AVAR want to see, what camera, level of zoom, and video speed — allowing the referees to focus on the situation without the additional cognitive load.

Q:  Tell me about the experiment. What did you do?

A: Before I get into the details, maybe I should explain that prior to the live experiment, we spent a lot of time planning. To ensure the authenticity of our findings, we decided to run the tests during live Bundesliga matches. So, I needed to do things discreetly without affecting the environment of the people working in the operations room. In my position, I test a lot of different technologies, so it’s important that the referees do not feel disturbed and will support future trials. This is especially important in live situations like this one when they must focus on a Bundesliga match.

To capture the gaze behavior of the VARs, we used a Tobii Pro Fusion eye tracker mounted on the lower screen, which the VAR uses for decision-making. And we ran our data collection during 12 live matches in Oct-Nov 2022.

Q: Tell me about the security. Before we ran this project together, I had no idea that the VAR was placed in a centralized location and not at the stadium where the match is taking place. So, how does it work with communication? How does the data get from the stadium to the operations room?

A:  The building is not accessible to the public without accreditation, and we deliberately keep the number to a minimum to allow people to concentrate. Only officials, technical personnel, and the VAR coach are allowed in the operations room during matches. We have a subcontractor to take care of the network that delivers the camera feeds from the games and establishes the audio connection between the people in the operations room and those on the pitch. So, as you can see, the setup is complex, and there are a lot of different actors involved.

Q: Is that why only the top leagues have VAR?

A:  Yes, probably. There’s a lot of people, equipment, and infrastructure to take care of. You also need sufficient camera coverage of the pitch and trained referees. All of this comes at a cost. I know FIFA is working on a VAR light solution, so it will be interesting to see how that develops.

Q: But let’s get back to your experiment. Was it easy to set up the eye trackers in the operations room?

A: Yes and no, but that had nothing to do with the eye tracker. I didn’t want to impact the VAR workstations in any way, so although we attached the eye tracker to the bottom of the VAR screen, we connected it to a separate laptop. As you know, under normal circumstances, you would set up the eye tracker on the same computer where the application you want to test is running. But we didn’t want to do this to ensure there would be absolutely no impact on the live VAR system.. As you can see from the images, the eye tracking equipment is very discreet; you can barely notice it — which is good because it means the data we collected isn’t skewed by the presence of equipment.

Q: So, how did you match the context with the eye tracking recordings?

A: We fed the live match feed to the test laptop and overlaid it with the feed from the eye tracker. And I can say that the result is amazing to watch, and the data is precise. You can see exactly how the referee’s gaze jumps around as they decide what camera gives the best insight. As they investigate match incidents, you can see exactly what details of a challenge they focus on.

Q: And what about calibration? How did that go?

A: I would say it was a smooth process. We ran calibration on the eye tracker about half an hour before each match to avoid interfering with pre-match preparations. Once everything was set, we started the recording and collected data for about 2.5 hours. The constant lighting conditions in our operations room ensured no re-calibration was necessary, and data quality remained high.

Q: So, tell us about the findings. Did you get the answers you were looking for? Did you find the insights about how VARs do their job?

A: I think you could describe the trial as a pre-study. I wasn’t sure beforehand what kind of findings we would be able to make. What we discovered was how to best design another experiment that would help us dig into the way VARs do their job. This is standard in research. You have an idea; you test it, you refine it, test it again, and then widen your test group until you are sure that the data you’ve captured provides you with a concrete finding — something you can use to make informed decisions. In our case, that would be developing appropriate training techniques.

Q: What will you do next?

A: I haven’t yet completely refined the next step, but I will run a new experiment at an upcoming training event for referees. They are required to attend regular classroom and athletic training sessions to maintain their licenses, so it’s an ideal opportunity for me to take the next step with them. Looking back, running the initial test in a live match situation was a good idea from an authenticity perspective, but it created some difficulties with the analysis.

First, every data capture was from a different match, making it impossible to compare VARs and uncover behavioral patterns. And because the VARs sit a long way back from the screen for most of the time (outside the tracking range of the Tobii Pro Fusion), we didn’t collect enough data. So, I’ve decided to run another experiment in a training environment that will allow me to solve these two issues. I am thinking about setting up the trial on a single screen so that I can capture the trigger event — what the VAR saw on the main screen that caused them to look at the replay. And in a training environment, I can ask the VARs to sit within the tracking box so that I can get enough data. And the last thing I want to try using recordings of live matches so that I can compare VAR behavior, safe in the knowledge that they were all looking at the same events.

Q: That sounds great, Tobias. I am looking forward to seeing what findings you might uncover. But for now, I have one final question. What do the referees think of VAR and its introduction to football?

A: The feeling I get from the referees is that they are very happy with the introduction of VAR and the additional layer of safety it brings. If they make a mistake, they now have the chance to correct it. Lowering the impact of referee decisions and making football fairer. Referees and VARs will never be perfect, but we do our best to make them as good as possible. This project is one step toward expanding their training possibilities. I want to thank you, Michael, for all the support in this non-standard setup. I would also like to thank the University of Tübingen for supporting us with their expertise in analyzing the eye tracking data.

A final word from Michael

For my part in this project, I felt very privileged to be allowed into the Bundesliga’s VAR operations room, to be able to take pictures, and to experience match day from behind the scenes. That was something special. And I agree with Tobias, it was fantastic to see our technology in action, how it visualizes what the VAR pays attention to, and the role it plays in understanding human behavior. I would also like to thank The University of Tübingen for their help in analyzing the data we collected. And to Tobias for having the idea in the first place.

Tobias Bauch

On LinkedIn