본문 내용으로 건더뛰기

KDI 경제교육·정보센터

ENG
  • 경제배움
  • Economic

    Information

    and Education

    Center

최신자료
Exposing Severe Methodological Gaps: A Critique of the Urban Institute‘s Panel Study on Land Use Reforms
AEI
2024.07.09
The Urban Institute’s study by Stacy et al. (2021) seeks to analyze land use reforms across U.S. cities and time to estimate the impact on housing costs and supply. The authors use a machine-learning algorithm to identify and analyze news articles on land use changes and integrate this information with postal and census data.[2] However, our review of this approach reveals significant methodological and data interpretation flaws that undermine the study’s conclusions.
Central to the study’s methodology is the use of machine learning algorithms to classify news articles related to land use changes. According to the Urban Institute study, this approach identifies 180 major land use changes from a set of 240 articles. Yet, a careful examination of a 20 percent random sample of these changes reveals significant classification errors. Only 14 percent of the sample were accurately classified as major changes, while 86 percent were misclassified. In 33 percent of the cases policy changes were likely minor― contradicting the study’s focus on significant reforms. In 11 percent of the cases, policy changes were likely ineffectual due to restrictive caveats that could impede construction. In 22 percent of cases, we could not confirm the policy direction from the available information. In the remaining cases the algorithm incorrectly classified the nature of the policy changes, tagged unrelated articles or was based on op-eds. These findings demonstrate an overreliance on machine learning and media sources, which often lack the necessary detail for accurate classification and understanding of policy changes, thus obscuring the true impacts of such reforms.