Regina Joseph is a Superforecaster, a behavioral data science researcher, and principal in 2 R&D consultancies, Pytho LLC and Sibylink, specializing in decision and information science R&D and product development. She holds a patent for her structured analytic system called Neuer and has a patent pending on Human Forest, a new prediction technology. She has come in first place, as either a Superforecaster or research team performer, in three consecutive competitive Intelligence Advanced Research Projects Activity (IARPA) programs since 2011.
Since 2019 alone, she and her Pytho co-founder have received two National Science Foundation awards to conduct a groundbreaking study on human versus artificial intelligence in predicting biomedical advances. Her prediction training programs are used by clients like the Ministry of Foreign Affairs and Ministry of Public Health in the Netherlands, among others. Her experience places her at the forefront of the science and practice of forecasting and decision-making.
Last week, Regina was kind enough to sit down with Global Guessing to discuss a wide array of subjects ranging from forecasting, the differences between quantified and old-school forecasting approaches, effective cognitive training approaches, dealing with the gender issues in the forecasting space, and her thoughts on new, social, forecasting platforms. Enjoy! And read to the end of the interview for a surprise offer!
Andrew Eaddy (AE): As someone who is at the center of the quantified and super-forecasting world, could you describe what separates quantified forecasting from other forms of forecasting or analysis? How does the forecasting you do differ from the Ian Bremmer’s of the world?
Regina Joseph: I think the work and results of the Intelligence Advanced Research Projects Activity (IARPA) Ace Program pushed Ian Bremmer’s work towards quantified forecasting. His prior work beforehand was more on the qualitative side or had a qualitative focus, and the distinct issue with that kind of work–work that many were doing at the time–is how do you establish consistency?
If we are talking about predictions in the political science or international relations spaces, how do you determine if a pundit is really good? As Professor Tetlock writes about in Superforecasting, you often see that experts have the least accuracy in their fields of subject matter expertise and that they are not the best forecasters. They are usually not much better than random chance. And if they are not much better than random chance, that begs the question: Who is better than random chance?
We know who some of them are from IARPA tournaments, and we know they are better than random chance because they make hundreds of predictions every single year and we keep track of their scores.
Clay Graubard (CG): How would you then compare the work that you and other super forecasters do with the Good Judgment Project and elsewhere to the type of forecasting and predictions that say FiveThirtyEight does or The Economist does?
Regina: The main difference is that with the Good Judgement Project or what was done with the IARPA research programs, were that those were experimental programs with specific objectives. So you are talking about different purposes. FiveThirtyEight, or any other similar commercial business, is not being put to the test. They are not being asked to make quantified predictions, on hundreds of questions, on many different subjects, all while being tracked for their accuracy. Nobody is being exposed to the same level of rigor over the same period of time that superforecasters are in an experimental context.
My research partner and I have a company called Pytho and on one of our projects we are co-Principal Investigators on what is called the Human Forest research program in collaboration with ASU which was just awarded two National Science Foundation grants and we were invited for a third.
With the Human Forecast project we are pitting human superforecasters against artificial intelligence models in an attempt to predict the clinical trial success. Our goal is to highlight the power of collective intelligence, which when harnessed by the right technology, can outperform the most sophisticated models. So we are spearheading some reference class forecasting on time specific and non-time specific forecasts on COVID vaccines and treatments.
Yes, everybody is doing a type of forecasting, but the level of rigor, the level of complexity, the amount of effort, and specifically who is being tracked, who is keeping score, and how score is being kept are all completely different.
CG: Superforecasting is still in its nascency - it has only existed for around 10 years. Looking to the future of forecasting in general, what expectations do you have regarding the human-machine divide? Could you broadly sketch what your philosophy is on the hybridization between the two approaches to forecasts?
Regina: Myself and my research partner have been working on this for the last 10 years and we believe that there's a five step process which we call the supply chain of prediction.
The first is identifying and targeting skilled forecasters.
The second is establishing rigorous questions. Question forecasting is a type of meta forecasting, right? One of the things that I was involved in at ACE, besides being a superforecaster, was generating questions on the research side of the team. I personally felt the questions for the first two years were too easy and other superforecasters also felt that the questions could have been more interesting and challenging. You have to be a good forecaster to write good questions.
The third step is cognitive training. There are very specific forms of training that we know and have tested that specifically improve a person's ability to generate accuracy in their forecasts, such as simply being taught how to find a base rate.
Learning how to develop better interfaces, better UI / UX strategies is part of the fourth step, which is the elicitation platform. How you extract forecasts from people matters and that is where the machine-human hybridization really comes into play. The feedback loop between the data, the information science component which extracts insight from the data, and the translation of that information into statistical concepts which can be explained, processed, and interpreted by a majority of people.
The fifth and final step is statistical aggregation. When assessing people’s forecasts, there are a bunch of things that you have to take into account during that period of time, especially if you are tracking people’s forecasts over a long period of time. So you need to be able to apply a scoring function to determine how you are taking those forecasts into account and you also have to weight them.
CG: Being the second step, you clearly see training as an important factor in the development and practice of forecasting as a science and practice. Both your companies, Sibylink and Pytho, offer training for forecasting and I was wondering what was your aim with these programs?
Regina: The nature of the work I do is getting people to understand complex ideas easily. That translational component, distilling complexity into the graspable, is a large facet of my work and is what separates me and distinguishes our brand. We do it with style! We think a lot about how you add the entertainment side of it so that you really can crowdsource.
We also really think about how we can facilitate reaching out to communities that don't normally have access to this material, like women and especially women of color. I think that there are reasons why I'm one of the few female superforecasters.
I think that has a lot to do with the double shift many women face: the time barrier of having to earn the bacon and fry it for the family too makes finding disposable time to take part in research experiments prohibitive.
There are also still few women conducting research in forecasting–I still often find myself the only woman in the room. As a Superforecaster I do double duty both as researcher and guinea pig depending on the research program. So I would love to see a near future of more women to work with, but I also see the limits of that possibility.
The research environment is typically male-dominated. I've definitely had my #MeToo experiences in them. So I completely understand why there is a statistic that most women in science only last for about 10 years before they drop out. That's a huge problem that I certainly want to solve.
That starts with me being able to be visible and demonstrate to communities that this is something that's not only a viable career, but also something that they need for their own survival. There's a huge gap in news awareness in women. And that is a focus of my research as well, the nature of news, awareness and informational and situational awareness from a national security perspective around women and other groups, people of color. So that's also something that I'm trying to address with my training programs.
AE: Both Clay and I have explored some of the crowdsource-based prediction platforms that are getting a lot of new, aspiring, forecasters interested in the science of predicting. Do you think those are a positive influence on the space as a whole?
Regina: They can be, but what worries me also is that they have their own bubbles. One of the things as researchers that we're really concerned about is the nature of these types of bubbles that can result, which have attendant biases attached to them. Around the use of forecasting, it can be very easy to manipulate people with numbers.
Eitan Hersh has written about the gendered nature of politics, that there's an individualistic approach to politics for men. It's kind of a competitive sport. Forecasting operates in much the same way. It can be seen by men as a competitive sport, and that can bring into it certain levels of overconfidence that I think can also be off-putting to other communities that would otherwise be interested in forecasting. If we want to see forecasting move in a direction where it can be used, then being conscious of bubbles is key. Thinking about how you shape diversity in forecasting as a cultivated skill as well as its deployment.
I'm still usually the only woman in the room. Figuring out how that changes is difficult. It has to happen with greater visibility and greater alliances. That's the way I see how all of this would shake out in the future. I think that those issues have to change and they have to change fairly rapidly. And I'm hopeful with the change in government, but I'm also concerned that if it’s still so rare to see women doing this then we have to ask ourselves why.
Our work brings the hopeful message that what we're doing at Pytho is testing the notion that if you train enough people the right way, crowdsource them, you can actually generate predictions that are more accurate than machine model forecasts. Right? So for everybody that's worried about whether or not machines are going to take over their meaning in life we have a hopeful message to offer, which is it doesn't have to be that way. The more that we can get people to understand what we've been working on in hybridization, the more their fear of replacement would be reduced.
There are certain things that machine learning is great at, but there are many other complex things that humans are way better at or faster at. But you still have to train people properly. You have to give them the right tools. There have to be improvements to how we currently think about forecasting, how we incorporate it, and what kinds of communities are supporting it.
AE: Thank you. Your answers have been great. We're actually planning on ending all of these interviews by asking people for two rapid fire predictions.
CG: You'll have ten seconds to reach a probability.
Regina: I would say there is a 75%—
CG: We actually have questions.
Regina: Oh, OK. I thought you were asking me to generate a question, which is funny because I have to do that on a daily basis myself.
CG: All right. The first one, slightly serious: What is the likelihood that Putin annexes more territory in Eastern Europe in the next five years?
Regina: In the next five years, I would say that I'd put forty-five percent on it.
CG: All right, question number two, ten seconds, but actually you can reclaim the one second you had left from question one so you have 11 for this one: What is the likelihood that we credibly detect alien life in the universe in the next ten years?
Regina: Oh, in the next ten years. Well, when you say alien life, I need clearer resolution criteria. What do you mean by alien life?
Clay Graubard: Biological like cellular life, proto-organisms.
Regina Joseph: I put that at 40 percent.
Andrew Eaddy: Alright well thank you again, we have really enjoyed talking with you today!
Pytho's Upcoming Training Bootcamp: ARETE
ARETE Prediction Bootcamp prepares you to make better life and leadership decisions by providing a clear and systematic process. To transform your future, ARETE cultivates the critical thinking, foresight and resilience skills you need to succeed in the face of global disruption. Built on a scientifically-proven system that has trained diplomats, intelligence community & military analysts and government policy specialists in the US and Europe since 2012, ARETE's unique mentor-based program is now available to the general public.
By the end of this course, you will improve your predictive reasoning skill and receive a report detailing measurements of your performance; you will know how to break problems down more effectively in groups; you will understand how to lead by anticipating change; and boost your preparedness for making decisions under uncertainty in key areas, like negotiation & finance.
The next ARETE Prediction Bootcamp will begin on February 12th, and will be a ~2-hour weekly commitment over a six-week period. We encourage any of our readers who want to hone their forecasting skills to consider participating in this program!
Learn more about the ARETE Prediction Bootcamp and use the QR code below for 25% off:
Find Regina Joseph
Follow Regina Joseph (@Superforecastr) on Twitter
