About the predictor
Atlanta Journal-Constitution data visualization specialist John Perry assembled information on more than 14,000 bills introduced since 2001, each labeled with votes, sponsorship and text summaries, to create the Georgia Legislative Navigator for the AJC’s website. Then he and data specialist Jeff Ernsthausen recognized this would create a rare opportunity. That enormously detailed data could be used to, in essence, calculate the odds that a bill would pass. That could shine a light on the often murky legislative process and the factors that hurt or help bills.
So Ernsthausen consulted with academic experts in the quantitative methods, starting with Professor Charles Bullock, chair of the Political Science Department at the University of Georgia and a close observer of the Georgia Legislature. To identify the factors that might influence a bill’s outcome, he recommended a technique, logistic regression, originally developed for predicting whether individuals would develop certain diseases based on key risk factors.
A book by Professor Frank Harrell, chairman of the Department of Biostatistics at the Vanderbilt University School of Medicine became our statistical bible, and Harrell also provided advice and reviewed the AJC’s results. Gary King, director of the Institute for Quantitative Social Science at Harvard University, also provided some guidance and Professor John Stasko of the Georgia Institute of Technology provided advice and software used to identify key terms in each bill’s summary text.
The model is based on many obvious factors influencing a bill’s fate – the party of the bill’s sponsor, the number of co-sponsors, and whether certain leadership figures were a sponsor — and some not so obvious ones. In the latter category were such things as how close to the end of the session a bill was submitted and some of the wording in the bill’s summary. We looked at whether the summary included certain social issues terms that refer to abortion, gun control, prayer in school, alcohol or controlled substances. And we included a variable for terms like “city of” and “county of,” since so many bills that move through the Legislature are local bills with a higher overall passage rate.
Ernsthausen tested the AJC model by comparing its predictions with the actual outcomes of bills in earlier sessions. The result: This beta model does a reasonably good job of predicting the odds of passage for the bulk of bills that pass through the Legislature each year.
» Online: Learn more about what the predictor does and does not do
Few outside the room knew much about it, but the crowd packed in to see a Senate committee debate SB 270 was passionate about the proposal.
It would create a new city in north DeKalb County called Lakeside, and supporters and opponents sought desperately to sway members of the Senate State and Local Government Operations Committee to their side. What they did not know was that much of what makes a bill pass or fail may have little to do with their passion and much more to do with the information encoded in its legislative DNA.
Using a sophisticated probability model, The Atlanta Journal-Constitution has developed a prototype app that predicts the proability whether bills currently before the Georgia General Assembly are destined to become law or more likely to become trashcan fodder.
The app does not assess whether the bill would make a good law. For example, it does not weigh whether there needs to be a new city called Lakeside or whether the Lakeside bill is preferable to competing bills creating other cities in the same area. Instead, it looks at the characteristics of bills that passed or failed in past sessions, and uses those to predict the chances of passage.
Some factors are common sense, such as whether the bill has the backing of the majority party or whether its sponsor is in leadership. But the new tool is the first of its kind in Georgia to estimate the probability of a bill passing.
“What you’re doing is innovative,” Charles Bullock, chair of the Political Science Department at the University of Georgia, who reviewed the AJC’s results, said of the project. “For the interested reader, it would make what happens [in the Legislature] more understandable. For a lobbyist, it might make life much more predictable.”
The app is part of the AJC's Georgia Legislative Navigator on myajc.com, where readers can track bills, contact lawmakers and keep up with the latest news from the state Capitol.
The app gives SB 270 an 89 percent chance of passage, in large part because it is sponsored by Republican Sen. Fran Millar, a leadership figure in the majority party, and it deals with a local issue. Local bills tend to pass at a higher rate than bills that affect the entire state.
It ranks higher than competing cityhood bills in DeKalb because those bills are all sponsored by Democrats and have no Republican co-sponsors.
The bill passed the Senate committee, with Chairman William Ligon, R-Brunswick, casting the tie-breaking vote.
Moments after he helped pass the bill out of committee, Sen. Mike Crane, R-Newnan, said he found the AJC app’s prediction interesting.
Then he asked what all lawmakers want to know: “What does it say about anything with my name on it?”
The AJC app uses a technique developed by statisticians in the medical field to predict patients’ risk of developing certain diseases. In developing it, the AJC borrowed heavily from the work of Frank Harrell, chair of the Biostatistics Department at the Vanderbilt University School of Medicine, who also reviewed the AJC’s results.
“This is my first exposure to data from politics,” said Harrell, who has spent nearly 35 years working these kinds of statistical models. “It turned out to be even more useful than I thought you were going to find it to be.”
The app gives probability of passage, rather than up-or-down predictions, meaning some bills with low odds will pass. For instance, in a good model, a subset of 100 bills each with a 10 percent chance of passage would likely produce around 10 bills that pass. It’s the law of averages.
“Virtually no model in the social sciences always predicts accurately,” Bullock noted. “The exceptions don’t necessarily disprove the overall model.”
House Bill 21 is one bill that overcame long mathematical odds. It had one sponsor – a Democrat – and affected Georgia law statewide, both strikes against it. As such, the AJC model gave it a 3 percent chance of passage.
But the bill, which makes post-adoption contracts for child visitation or contact with birth relatives legally binding, passed both chambers last year and became law July 1.
“Your model probably doesn’t measure the fact that I’m experienced and have a lot of expertise in maneuvering,” said Rep. Mary Margaret Oliver, D-Decatur, the bill’s sponsor.
Oliver said adoption is a very personal topic and she did not go looking for co-sponsors. Instead, she held lots of personal conversations with lawmakers, educating them about the issue.
“Personal relationships will trump your formula any day of the week,” she said. “Work like crazy is generally a good formula for me.”
Jet Toney, a longtime Capitol lobbyist, said the passion of the lawmaker whose name comes first on the bill is a key component in getting a bill to move. If the primary sponsor desperately wants the bill to pass, it can give a bill legs, he said.
The AJC app is a prototype that will continue to evolve, incorporating new factors that affect a bill’s chances as well as insight the paper receives from experts on the process and the statistics community. But statistical models may never be able to embrace intangibles, like passion.
About the Author
