Skip navigation

Data Accuracy and Modeling Work Plan


SANDAG has launched an initiative to improve data and modeling accuracy and transparency.

Overview

To correct and prevent errors, SANDAG has embarked on an initiative to strengthen the transparency, accountability, and accuracy of the agency’s modeling and forecasting processes. SANDAG technical staff has organized into several teams to systematically tackle various reforms and will soon bring in outside experts to review its processes and develop enhancements for consideration by the Board of Directors.

A seven-part work plan, unanimously approved by the Board of Directors on February 24, 2017, will address all of the agency’s data programs, including its Demographic and Economic Forecasting Model (DEFM). DEFM is currently under staff review after a data loading error was found to have affected some of the model’s outputs, including future taxable retail sales estimates that are used in forecasting the financial capacity of the TransNet Extension program. The data, which turned out later to have been flawed, also was used in revenue calculations for a half-cent sales tax measure that went before voters regionwide in November 2016. The measure known as Measure A did not reach the necessary two-thirds vote needed to pass.

Independent Examination

To ensure public trust, the SANDAG Board of Directors at its February 24, 2017, meeting voted to undertake an independent examination of who knew what and when leading up to the Measure A election.

The Board referred the matter to the Executive Committee, which formed a subcommittee during its March 10, 2017, meeting tasked with evaluating potential parties to conduct the examination and make recommendations to the Board.

On March 14, 2017, the subcommittee, consisting of Vice Chair Terry Sinnott, City of San Diego Council President Myrtle Cole, and Poway Mayor Steve Vaus, released a Request for Proposals to solicit services from qualified law firms with documented experience and expertise to perform an independent examination. A press release also was sent out announcing the request for proposals. Seven firms responded to the request for proposals. The subcommittee reviewed and scored the proposals.

On April 14, 2017, the subcommittee recommended Newport Beach-based Hueston Hennigan LLP to the Executive Committee to conduct the examination. On the same day, the SANDAG Board unanimously approved the hiring of the firm. Per the request of the Board, Hueston Hennigan committed to completing the examination on an expedited basis.

On May 12, 2017, the SANDAG Executive Committee took action to reconstitute the Measure A revenue forecast examination subcommittee, replacing Council President Cole (whose continued service conflicted with her busy schedule) with Imperial Beach Mayor Serge Dedina, to meet with and provide direction to Hueston Hennigan LLP on an as-needed basis.

On August 4, 2017, Hueston Hennigan presented its independent examination of Measure A revenue estimate communications findings at a special meeting of the SANDAG Board of Directors. The independent examination final report found that SANDAG did not intentionally mislead the public or the Board regarding its forecast. However, the report did identify various issues that needed to be addressed, and it included a series of recommendations.

At the August 4 meeting, the Board tasked the subcommittee with evaluating the recommendations and bringing back recommended actions. Those recommendations will be considered at future meetings.

The subcommittee provided an update to the Board at its September 8, 2017, meeting. As a result of that update, the Board voted to (1) discontinue efforts to evaluate the performance of the former Executive Director due to his departure from the agency; (2) direct the subcommittee to meet with senior personnel and report back to the Board in closed session with an advisory opinion regarding any recommended next steps; and (3) provide direction on next steps related to a performance auditor.

At its October 27, 2017, meeting, the Board received an independent forensic analysis report that showed no documents were inappropriately erased by SANDAG staff from the "Hana" server "Tools" folder used to store working files as staff searched for the cause of errors in the agency's forecasts of future revenues.

SANDAG Efforts to Improve Data Accuracy and Integrity

As a taxpayer-funded agency, SANDAG takes the integrity and reliability of its data very seriously. The agency has already taken the following steps:

  • Generated an independent forecast for the existing TransNet sales tax program
  • Consolidated the chief economist position with the technical services director position to integrate oversight of forecasting processes
  • Put in place a new code base, new algorithms, and new source data for the DEFM computer model

The seven-part work plan includes the following elements:

  • Conduct Detailed Review: Review the nature of the error in DEFM and its root cause. Review and validate input data, transformations, and equations in the Series 13
    forecasting model (the most recent series completed using DEFM) to ensure data accuracy and integrity of the model results. Present this information to the Board of Directors.

  • Conduct Dependency Analysis: Identify key SANDAG reports and deliverables that used data from Series 13. Evaluate the significance of the impacts from any potential forecasting errors and the potential effects on findings and policy recommendations. Address issues identified in the analysis.

  • Map Modeling Process Flow: In preparation for future forecasts, map all data flow from source through databases, models, and outputs to provide transparency and identify areas for improved quality assurance processes. Complete online documentation and visual mapping of interactions in the model, showing all data sources, processes, interactions, and flows.

  • Improve Data Governance: As a first step toward formal data governance, conduct interviews and document the customer-supplier relationship between SANDAG staff and the SANDAG Technical Services Department. This effort will lead to a better understanding of the type of data that agency staff request from Technical Services and a better understanding of how the data are requested, stored, used, and versioned. This information will be used to develop a data warehouse, standardize data extraction routines, and ensure consistency of data.

  • Review and Oversight: Validate the new SANDAG population, housing, and economic forecasting model using an independent expert review committee, including convening a panel
    of experts in economics, demographics, and land use to review the methods, data sources, and assumptions of the new SANDAG forecasting model. The panel will evaluate the efficacy and sufficiency of the proposed Series 14 forecasting model to adequately forecast population, housing, and economic variables for SANDAG planning purposes.

  • Enhance Transparency: Develop a set of agency methods and standards to ensure data and analytic transparency, including establishing check points where full disclosure and analysis are provided to ensure that others can see how models were developed, how data was processed, and what assumptions were made along the way.

  • Develop and Formalize Processes: Understand how staff roles, work flows, and technology (e.g., models, databases, software) contribute to producing key agency deliverables. This information will be used to realign the Technical Services Department, as well as add professional quality assurance staff and a dedicated database administrator. This effort will reduce single points of failure, and increase accountability, visibility, and efficiency. Conduct research and prepare an assessment of the current state of software and database platforms to facilitate a plan to implement industry best practices as they relate to data structures, data quality, database design and development, and database governance.

Keeping the Board and Public Informed

SANDAG staff will present ongoing progress reports to the agency Board of Directors and the TransNet Independent Taxpayer Oversight Committee to keep them and the public informed about the implementation of the work plan to improve data accuracy.

This web page will be updated periodically to provide additional information.

What is DEFM?

SANDAG uses a variety of forecasting tools to make estimates for planning purposes. One of them is known as the Demographic and Economic Forecasting Model (DEFM). Originally created in the 1970s, the model has continually undergone minor updates since then. Today, DEFM incorporates data from nearly 700 variables to make forecasts as far out as 2050.

In addition to being used for predicting demographic trends, DEFM also provides data for economic forecasts. One of the outputs from the model – taxable retail sales – is used by SANDAG staff as an input to estimate future sales tax revenues.

Fundamentally, DEFM is a synthesis of two widely used demographic and economic techniques. On the demographic side, the cohort method considers such factors as birth and death rates, and the age, gender, and ethnic distribution of the resident population to arrive at forecasts of demographic variables. On the economic side, time-series/regression methods are used to estimate economic relationships. The resulting econometric equations provide forecasts of employment, income, and other economic variables based on assumptions about national, local/state growth patterns, and local inter-industry relationships.

Additional Resources

Related Resources