In its development of its 2010 Review methodologies for assessing State fiscal capacities, the Commonwealth Grants Commission (CGC) sought a consultancy study on the effects of government policy versus non-government policy on the enrolment of students.
The consultant used econometric methods to estimate those effects. The report is available from the CGC website: http://cgc.gov.au/method_review2/2010_review_documents2/2010_review_consultancy_report/2009_-_modeling_of_post-compulsory_enrolments
In its report the consultant identified and used a small number of non-policy variables and then used State dummy variables to estimate the enrolments, by a classification of statistical areas.
The consultant derived estimates and drew conclusions from those estimates:
It attributed the State dummies as to the policy effects
it claimed that the policy effects were 20% and non-policy effects to be 80%, by dividing the explained variances into that explained by the selected non-policy variables and that by the residual State dummies
There are some issues with that approach. Firstly, as with any econometric estimates, there are unexplained variances and explained variances. There is an issue of what the unexplained variances should be attributed to, that is, effects of policy or non-policy? Or both but by what proportion?
Secondly, even within the explained part, the use of State dummy variables means they are the residuals in the sense that they pick up the effects of the differences between the States after those explained by the identified and selective non-policy variables.
Intuitively, one would understand that in any econometric estimations, the variables used are only a part of the actual ones that affect the issue at question. Inevitably, there are some effects of those real variables or their effects are left out due to various reasons.
Under the current scenario, the residual State dummies actually include the effects of those variables left out of the estimation. So it is categorily incorrect to attribute the effects of the dummies to the effects of State policy variables.
In addition, the consultant acknowledge that the Census data used had an issue of under count. It assumed that the effects of under count to be the same across all States.
It is highly likely that the under count in the Census data may have a strong bias, in that it was more serious in remote areas as compared to city or urban areas. This means that the States with larger shares of remote population are likely to have higher enrolment in the Census than their actual enrolment, and vice versa.
The assumption of equal effects across the States was highly problematic and results derived under such a conclusion was wrong.
This was especially the case in the Asutralian context, where two small States have drastically and diagomatically different shares of remote population.
The ACT has almost 100% urban population, because it is largely a city. The Northern Territory, on the other hand, has the highest remote and very remote population because of its very large share of Indigenous population in its total population.
Showing posts with label econometrics. Show all posts
Showing posts with label econometrics. Show all posts
2009-05-08
A small contribution to the practical application of econometrics
My colleagues and I encountered an interesting econometric problem in an exercise in using the results from econometrical estimation during a project. For that project, we had to estimate a location effect for each of the eight regions and the results were used to determine the allocation of funds among eight organisations located in eight different regions. Due to external factors to those organisations beyond their individual control, their operations require different funding to take account the impact of their locations on their operation. So it was not just an academic exercise of econometrics, but also had a real impact on the allocation in the real world.
As with any econometric estimation, one may get the resultant estimates of which some are statistically significant and others insignificant. This was exactly the case with our project. We used dummy variables to capture the location effects for the eight different organisations. Organisation 8 was used as the reference or default, so its location effect was 0, this is because there can only normally be dummies for n-1 of the n organisations. It was deemed to be highly significant at the value of 0. Table 1 shows the results from the econometric estimation:
Table 1 Estimation results
We had further transformations done on these results to translate these estimates to the location factors for each organisation to use in the final allocation of funding. However, they are not essential to the focus of this article here and I will not discuss them and only focus on how we used these results as the inputs into those transformations.
Because there would be a real impact on the allocation of funding, we were faced with the question: should we use the estimated values for these organisation irrespective their statistical significance, or 0 value for those organisations whose estimates are statistically insignificant, or any other value and what they could be? What values to use would affect the funding, or the shares of the total of those eight organisations. We had to make a choice with consequential effects on the allocation outcomes.
There were two different arguments, one for using the actual estimated values and the other for using 0 value for those organisations concerned. The former argued that they were the actual estimates that produced best account of location effects, while the latter said they were statistically insignificant and that meant they were not statistically different from 0 value.
Both arguments had some rationale, depending how one looks at the issue. However, they are two extreme cases/values of possibly many values to choose, because one could argue that one could use a combination of the two, that is, any value between the two. The third approach could avoid the extreme values of using either the actual estimates or 0. But the central question remained: what value to use that would produce the most desirable or best outcome of the fund allocation?
We had pondered that question for a very long time. Finally, we came to the conclusion that the use of any combination of the two would be arbitrary, except perhaps one set of values. This set of values was informed by the data we used. They were determined by using the information on the level of statistical significance (p values) for the estimates. We used that information as the basis to discount the actual estimates and use the discounted estimates as the final values as the inputs in the final transformation processes for deriving the location factor to be used in the funding allocation.
More specifically, we combined the actual estimated values and 0 using the p values as the weights. For example, for organisation 6, we multiplied the actual estimate, 0.032, by 0.552 - the difference between 1 and its p-value of 0.448. This gave us a value of 0.017664. That was equivalent to the result of this following calculation: 0.448 * 0 + (1 – 0.448) * 0.032 = 0.017664, where the weights were 0.448 and 1 - 0.448 = 0.552.
We considered this as the most desirable choice for organisation 6. It was not arbitrarily determine by anyone’s arbitrary intervention, but based on information derived from the data and the estimation. It was a better result than either of the two extreme values. It was defensible statistically.
This was because even the conventional critical values in determining the statistical significance of the estimates were somehow arbitrary. For example, if one takes the critical value of significance level was 2 from the t-statistic, then someone could ask, what about any values that were smaller than but infinitely close to 2? Why should those infinitely small differences should make very large differences in the implied parameters to use, that is, between zero and the actual estimates?
Probably no one can answer that question satisfactorily, except to say that the choice of the critical values for judging statistical significance is largely a matter of convenience. In other words, people have to use a value and a infinite range of values could be used, but one has to make a choice, so has to pick a convenient number that can make the trick.
To conclude, although we were forced to make a choice for the values to use and had been agonised for a very long time, we finally did made an interesting discovery and came up with quite satisfactory results. Our approach may have implications for the practical use of econometric estimates in many situations. We think it is a statistically sound approach. It is simple but significant for practical application of econometrics.
As with any econometric estimation, one may get the resultant estimates of which some are statistically significant and others insignificant. This was exactly the case with our project. We used dummy variables to capture the location effects for the eight different organisations. Organisation 8 was used as the reference or default, so its location effect was 0, this is because there can only normally be dummies for n-1 of the n organisations. It was deemed to be highly significant at the value of 0. Table 1 shows the results from the econometric estimation:
Table 1 Estimation results
We had further transformations done on these results to translate these estimates to the location factors for each organisation to use in the final allocation of funding. However, they are not essential to the focus of this article here and I will not discuss them and only focus on how we used these results as the inputs into those transformations.
Because there would be a real impact on the allocation of funding, we were faced with the question: should we use the estimated values for these organisation irrespective their statistical significance, or 0 value for those organisations whose estimates are statistically insignificant, or any other value and what they could be? What values to use would affect the funding, or the shares of the total of those eight organisations. We had to make a choice with consequential effects on the allocation outcomes.
There were two different arguments, one for using the actual estimated values and the other for using 0 value for those organisations concerned. The former argued that they were the actual estimates that produced best account of location effects, while the latter said they were statistically insignificant and that meant they were not statistically different from 0 value.
Both arguments had some rationale, depending how one looks at the issue. However, they are two extreme cases/values of possibly many values to choose, because one could argue that one could use a combination of the two, that is, any value between the two. The third approach could avoid the extreme values of using either the actual estimates or 0. But the central question remained: what value to use that would produce the most desirable or best outcome of the fund allocation?
We had pondered that question for a very long time. Finally, we came to the conclusion that the use of any combination of the two would be arbitrary, except perhaps one set of values. This set of values was informed by the data we used. They were determined by using the information on the level of statistical significance (p values) for the estimates. We used that information as the basis to discount the actual estimates and use the discounted estimates as the final values as the inputs in the final transformation processes for deriving the location factor to be used in the funding allocation.
More specifically, we combined the actual estimated values and 0 using the p values as the weights. For example, for organisation 6, we multiplied the actual estimate, 0.032, by 0.552 - the difference between 1 and its p-value of 0.448. This gave us a value of 0.017664. That was equivalent to the result of this following calculation: 0.448 * 0 + (1 – 0.448) * 0.032 = 0.017664, where the weights were 0.448 and 1 - 0.448 = 0.552.
We considered this as the most desirable choice for organisation 6. It was not arbitrarily determine by anyone’s arbitrary intervention, but based on information derived from the data and the estimation. It was a better result than either of the two extreme values. It was defensible statistically.
This was because even the conventional critical values in determining the statistical significance of the estimates were somehow arbitrary. For example, if one takes the critical value of significance level was 2 from the t-statistic, then someone could ask, what about any values that were smaller than but infinitely close to 2? Why should those infinitely small differences should make very large differences in the implied parameters to use, that is, between zero and the actual estimates?
Probably no one can answer that question satisfactorily, except to say that the choice of the critical values for judging statistical significance is largely a matter of convenience. In other words, people have to use a value and a infinite range of values could be used, but one has to make a choice, so has to pick a convenient number that can make the trick.
To conclude, although we were forced to make a choice for the values to use and had been agonised for a very long time, we finally did made an interesting discovery and came up with quite satisfactory results. Our approach may have implications for the practical use of econometric estimates in many situations. We think it is a statistically sound approach. It is simple but significant for practical application of econometrics.
Subscribe to:
Posts (Atom)
Posts
