Impact Evaluation of Raskin Program using Matching Method : Case of IFLS 5

Beras untuk Keluarga Miskin (RASKIN) program has been applied since 1998 and has been renamed as Beras Sejahtera (RASTRA) in early 2017, but their effectiveness is still debatable. This study tries to evaluate the impact of RASKIN program on household income. Using data from 3,745 households in Indonesia Family Life Survey (IFLS) 5 that has been estimated using propensity score matching, this study has identified precisely that RASKIN program has a negative and significant effect on household income. This happens because the benefits that reveived by Rumah Tangga Sasaran (RTS) are very small. The small benefit is affected by the amount of rice received, frequency and price that have been paid to get RASKIN is not in accordance with the guidelines. The result of this study is along with previous studies, where the amount and price of rice that distributed through RASKIN progr am is not exactly correct. Therefore, there must be a change in program format, not just renaming from RASKIN to RASTRA only.


INTRODUCTION
Indonesia should not worry too much about subsidies or protection of rice production in other countries, price volatility in the world rice market, the possibility of large exporters forming a cartel, or international movement through trade liberalization of rice (Dawe, 2008).Therefore, it is appropriate for the Indonesian government to focus more on policies to stabilize domestic prices and supply of rice.If these two things can be stabilized by the state, then the poverty rate and the number of poor households can be reduced.This happens because almost all of 230 million people in Indonesia consume rice and rice are the staple food for them (McCulloch & Timmer, 2008).
Figure 1 below shows the growth of the number and percentage of poor people in Indonesia from 1999 to 2015.From this figure it can be seen that the number and percentage of people in recent years tend to decrease.In 1999 after crisis, the number of poor people reached 47.97 million or 23.43 percent of the Indonesian total population.The number continues to decline until 2015, which reached 28.59 million people or about 11.22 percent of the total population.The number and percentage of poor people had increased in 2006 due to agricultural commodity prices increase in global The end of the oil boom in the early 1980s brought about a major change in rice policy.The government is no longer flooded with oil revenues previously used to finance the rice sector markets (Dawe, 2008).After the crisis, the income gap.
Authority of BULOG's import monopoly was removed and since 1999, price stabilization has been achieved through free trade.between households of agricultural commodity producers and non-producers decreased (Yamauchi & Devina, 2012).This decline has an impact on the declining number and percentage of poor people in recent years.
According to these facts, the government's role in stabilizing domestic price of rice is very important.So far, the Government of Indonesia has made various efforts to stabilize the price of rice.Rice policy in Indonesia can be divided into three phases since the early 1970s.In the first phase, the New Order government invested heavily on input subsidies, irrigation infrastructure, research and development on system, resulting in rapid growth of rice production in every year.One of the main reasons is that Soeharto's government is so focused on agriculture and industrialization that mantaining food price stability is a must (McCulloch & Timmer, 2008).Since the early 1980s, the performance of the world rice market has begun to be convincing, thus supporting the Government wants of price stabilization (Dawe, 2008).
Price stabilization was achieved during this period until the late 1990s by providing mandate of price stabilization, import and resource monopoly to BULOG.This combination of policies was able to improve food security and price stability, but increased corruption in BULOG after the early 1990s, due to the full authority given to achieve the mission.The end of the oil boom in the early 1980s brought about a major change in rice policy.The government is no longer flooded with oil revenues previously used to finance the rice sector.As a result, the growth of rice production continued to slow down until the economic crisis in 1998.The authority of BULOG's import monopoly was removed and since 1999, price stabilization has been achieved through free trade.In fact, prices were more stable during the free trade period of 1999-2004 than wen import authority was held by BULOG.However, crisis and the decentralization to the district/city government led to a significant reduction in the supply of inputs and services by the government to produce rice.
The third phase, which began in 2004, is marked by the imposition of a rice import ban that removes the price stabilization mechanism, whether through free trade of BULOG's import monopoly.As a result, during each hungry season before the March-April harvest, prices are rising sharply as the supply of rice decreases.The government has attempted to reduce the price increase through ad hoc imports, distribution of subsidized rice to communities conducted by BULOG, or through the expansion of Rice for Poor Household (Beras untuk Rumah Tangga Miskin, RASKIN) program (McCulloch, 2008;McCulloch & Timmer, 2008).
The RASKIN program, which began in 1998, is aimed at improving the food security of poor households.Initially, the program was called Special Market Operation (OPK), but since 2002 the program was officially changed to RASKIN.From figure 1, it can be hypothesed that the RASKIN program contributes to the reduced number and percentage of the poor people.
According to RASKIN General Guidelines, RASKIN's success can be measured based on the achievement of the 6T indicator, which is right on target, exact in quanity, exact on price, exact on time, proper quality and proper administration.The RASKIN program is said to be on target if RASKIN is only given to poor families enrolled in the List of Beneficiaries (DPM-1), exactly on amount if the RASKIN amount purchased by the beneficiaries equals the RASKIN amount of 20 kg/head of the family/month, and exact on price if the price paid by the beneficiaries is 1,000 Rupiah/kg at the distribution point (Jamhari, 2012;Emalia, 2013).
The study by Jamhari (2012) and Emalia (2013) evaluates the effectiveness of RASKIN program with reference to the 6T indicator.Jamhari (2012) finds that the RASKIN distribution in rural and urban Indonesia has not been exact on quantities and prices.Meanwhile, the research of Emalia (2013) found that the distribution of RASKIN in Bandar Lampung has not been exact in quantity.
According to RASKIN General Guidelines 2001-2005, the beneficiaries is determined through Village Consultation (Mudes) with reference to the Pre-Prosperous (KPS) and Prosperous 1 Family (KS-1) data published by BKKBN.However, the RASKIN General Guidelines 2006-2007 doesn't mention that Mudes should refer to beneficiaries data published by BPS.In fact, in the Beneficiary Assignment section it is not stated that beneficiaries should be poor households.Nevertheless, Mudes has not been fully implemented in every village, so its implementation is not yet optimal.Some have set targets through Mudes, some use national reference data (BKKBN or BPS) as the basis, and some are determined by the Head of RT/RW or Village Head.In fact, in many cases, RASKIN is evently distributed to every household for social, cohesiveness, mutual cooperation and so on (Hastuti et al., 2008;Jamhari, 2012;Emalia, 2013) Because it is funded with APBN, it is appropriate for the RASKIN program to be audited as an effort to maintain accountability and transparency.There are two state institutions that have audited this program.In 2004, the audit was conducted by the Food and Extension Agency (BPKP) and in the following year the task was taken over by the Supreme Audit Agency (BPK).Although both are auditing the RASKIN program, the audited aspects are different.The result of BPKP audit in 2004 showed that in 2003 the implementation of the RASKIN program at the national level reached 78.2 percent (of ideal 100 percent).In terms of rice distribution to beneficiaries, RASKIN program performance reached 61.42 percent and only 68.5 percent right on price.
The result of BPKP audit shows that the performance of RASKIN program has not been optimal.This is also supported by Hastuti et al. (2008) who found that in general the RASKIN program was not very effective.The number of problems that arise related to the distribution of rice from the main distribution point to the beneficiaries and the issues is always same from year to year is the main reason.
The low effectiveness of the program is also evident from the poor socialization and transparency of the program; the objectives, quantity, and frequency of rice received by the beneficiaries are incorrect; high cost program management; ineffective monitoring and evaluation; and ineffective complaints mechanism.This is in line with Hutagaol and Asmara (2008) study which found that the RASKIN distribution in West Java has not been effective due to inaccurate quantities and prices.
Similarly, Purwanti (2010) and Siddik (2009) found that the RASKIN distribution, quantity and price has not been well targeted in Magelang and Sleman districts 1 .Nevertheless, the Government of Indonesia continues to run the program.
Although at the beginning of 2017 the RASKIN program has changed its name to Beras Sejahtera (RASTRA), but its distribution mechanism is still the same.Seeing this, many questions arises: how does the evaluation of RASKIN program been so far ?Does the RASKIN program have an impact on poor household income so that the program continues with a different name ?Our study tries to anwer these questions.
So far, few studies have evaluated the impact of RASKIN with matching method.Therefore, using data of 3,745 household surveyed in 2014 by Indonesia Family Life Survey (IFLS), this study attempts to evaluate the impact of RASKIN by using matching method.This study uses the method of Propensity Score Matching to evaluate the impact of RASKIN program on household income.This method refers to some previous studies such as Gilligan & Hoddinott (2007) and Abebaw et al. (2010).
Gilligan & Hoddinott (2007) compared the impact of the Employment Guarantee Scheme (EGS) and Free Food Distribution (FFD) programs in rural households in Ethiopia.The EGS program took place between 1999 and 2002.Meanwhile, from 2002 to 2004 the FFD program was implemented.Thus, 2002 is the turn of the two programs.Despite household targets in both programs are similar, but the timing and amount of transfers are different.With the difference-in-differences propensity score matching, their study compares the average treatment effect between EGS and FFD program participants with non-participants.Gilligan & Hoddinott (2007) found that there was a considerable average treatment effect on EGS participants for total growth of consumption and food consumption in 18 months after 2002.EGS participants had lower livestock livelihood growth in the 1999-2004 period and the impact was greatest in relatively more prosperous households.In addition, the FFD program has a smaller impact than the EGS.Abebaw et al. (2010) used the PSM method to control initial differences between Integrated Food Security Program (IFSP) and non-IFSP groups in the Ibnat district of Amhara, Ethiopia in 2007.Abebaw et al. (2010) identified 99 targeted IFSP households and 79 non-IFSP households from a sample of 200 households (each group consist of 100 households).Both groups have similar characteristics before intervention, except on the program.As a result, the IFSP household group food intake was 30 percent higher than those belonging to non-IFS household group.In addition, the benefits of food intake are greater in households with fewer members, larger land ownership and female headed households.

RESEARCH METHODS
This study tries to see whether the RASKIN program is really received by the poorest household.Too see it, this study uses the classification of poor people according to BPS.There are at least 14 criteria of the poor according to BPS (see Table 1).These criteria can be used as control variables to identify the impact of RASKIN.Of the fourteen criteria, we only succeeded in identifying six variables from IFLS 5 data.The criteria of fuel for daily cooking are actually in the IFLS data, but after testing to meet the conditional mean independence assumption the result does not meet the rules of balancing property.This study uses quasiexperiment method because if using randomized experiment takes a lot of time and cost.Quasi-experiment method used in this study is matching method, more precisely propensity score matching.The steps of propensity score matching method are (1) estimate the propensity score; (2) choose matching algorithm; (3) ensure the data has common support; (4) ensure the matching quality; and (5) estimate the standard errors and sensitivity analysis.This method was choosen because we wanted to see how big the income difference between households receiving RASKIN and households that should receive RASKIN but in fact did not receive.
In estimating the propensity score, this study uses logit model to see the effect of control variable on treatment variable.The treatment variable is Raskin, which is a dummy variable that indicates if the value is 1, then a household receives RASKIN and 0 if it does not reveive.These variable will be tested with outcome variable, which is household income (Rp).To see the effect of treatment variable on outcome variable, multiple control variables are needed to ensure that the characteristics of the observations are similar in nature.The control variables are: (1) Education level of head of the family, dummy variable (1: no school/no primary school, 0: finish primary school and/or next); (2) Living in Java, dummy variable (1: living in Java, 0: not living in Java); (3) Household income, dummy variable (1: below 600,000 Rupiah per month, 0: above 600,000 Rupiah per month); (4) Electricity use, dummy variable (1: not use electricity, 0: use); (5) Source of drinking water, dummy variable (1: wells, rivers, rainwater etc., 0: pump wells); (6) Ownership of private toilet, dummy variable (1: not have private toilet, 0: have); (7) Ownership of vehichles, dummy variable (1: not have vehicles, 0: have); and (8) Ownership of savings, dummy variable (1: not have saving accounts, 0: have).
RASKIN essentially reduces food consumption expenditure, so beneficiaries can have additional revenue that can be used for other expenses, such as tuiton fees (Hastuti et al., 2008).This is what underlies the selection of outcome variables.Meanwhile, the variables in family education, household income, electricity usage, drinking water source, private toilet ownership, and saving accounts ownership are 6 proxies from 14 criteria of poor people according to BPS.Selection of such variables as in Sulistyaningrum (2016) study and have a role of controlling household characteristics that accept RASKIN and not.The selection of control variables is to estimate the propensity score using the Leave-One-Out Cross Validation method as proposed by Heckman et al. (1998), Heckman & Smith (1999) and Black & Smith (2004) 2 thus fulfilling the conditional mean independence assumption.Households that became our observation in this study is 3,745.Of these, the households that receive RASKIN were 1,152 or 30.76 percent and those who did not receive were 2,593 or 69.24 percent.Descriptive statistics of each variable can be seen in table 2 above.This table describes the units, average, number of observations, standard deviation, minimum, and maximum value of each variable.The average of household income observed in this study is 19,132,868.81Rupiah per year, with minimum value 0 and maximum value 720,000,000 Rupiah per year.Table 3 shows the comparison of treatment and control group characteristics in this study.Treatment group members with incomes under 600,000 Rupiah per month were 503 households, while those with income above 600,000 Rupiah per month were 649 households.Control group members with incomes below 600,000 Rupiah per month were 909 households, while those with income above 600,000 Rupiah per month were 1,684 households.
Treatment group members that did not finish primary school or no school were 213 households, while those who finished from elementary school or education level above it were 939 households.
Control group members that did not finish primary school or no school were 138 households, while those who finished from elementary school or higher education level were 2,455 households.The highest education of the head of family: no school/did not finish primary school 14 Don't have savings/goods that are easily sold with a minimum of 500,000 Rupiah such as motorcyle credit/non-credit, gold, livestock, motor boat, or other capital goods.

Table 2. Descriptive Statistics Of Research Variables
Source : Data Processed RASKIN essentially reduces food consumption expenditure, so beneficiaries can have additional revenue that can be used for other expenses, such as tuiton fees (Hastuti et al., 2008).
This is what underlies the selection of outcome variables.Meanwhile, the variables in family education, household income, electricity usage, drinking water source, private toilet ownership, and saving accounts ownership are 6 proxies from 14 criteria of poor people according to BPS.
Selection of such variables as in Sulistyaningrum ( 2016) study and have a role of controlling household characteristics that accept RASKIN and not.The selection of control variables is to estimate the propensity Score using the Leave-One-Out Cross Validation method as proposed by Heckman et al. (1998), Heckman & Smith (1999) and Black & Smith (2004) 3 thus fulfilling the conditional mean independence assumption.
Households that became our observation in this study is 3,745.Of these, the households that receive RASKIN were 1,152 or 30.76 percent and those who did not receive were 2,593 or 69.24 percent.
Descriptive statistics of each variable can be seen in table 2 above.This table describes the units, average, number of observations, standard deviation, minimum, and maximum value of each variable.The average of household income observed in this study is 19,132,868.81Rupiah per year, with minimum value 0 and maximum value 720,000,000 Rupiah per year.
Table 3 shows the comparison of treatment and control group characteristics in Treatment group members that did not use electricity as home lighting were 15 households, while those who use electricty as home lighting were 1,113 households.Control group members that did no't use electricity as home lighting were 19 households, while those who use electricity as home lighting were 2,594 households.Treatment group members that do not have savings were 912 households, while those who have savings were 240 households.Control group members that do not have savings were 1,607 households, while those who have savings were 986 households.Treatment group members who live in Java were 620 households, while those who not live in Java were 532 households.Control group members who live in Java were 1,378 households, while those who live not in Java were 1,215 households.
To ensure that these control variables do not contain symptoms of multicolinearity, a correlation test is necessary.If the correlation coefficient is close to 0, then a control variable does not have strong relationship with other control variables.Vice versa, if the correlation coefficient are close to 1, then a control variable has a strong relationship with other control variables.The results of the correlation test can be seen in table 4 below.With a rule of thumb of 0.8, then all control variable are not highly correlated.That is, there are no symptoms of multicolinearity in the control variables used in this study.households.Control group members that do not have savings were 1,607 households, while those who have savings were 986 households.
Matching method pairs program participants with control group members with similar attributes and estimate the impact of treatment by reducing the average outcome of matched comparison group members with the average outcome matched participants.Matching method also allows us to substantially reduce the bias, but not necessarily eliminate it (Heckman et al., 1997).
Rosenbaum & Rubin in Caliendo & Kopenig (2008) suggested the use of balancing scores b(X), where the covariate function X (respondent characteristics before treatment) so that the conditional distribution of X with b(X) is independent of the effect of treatment.One possibility of a balancing score is the propensity score, which is the probability of participating in a program with certain X characteristics.A matching procedure based on a balancing score is called Propensity Score Matching (PSM).There are three important issues in the implementation of PSM, namely: (1) whether the matching with replacement or not; (2) how many control group are paired with treatment group; and (3) what matching algorithm is used (Dehejia & Wahba, 2002) Heckman et al. (1997) suggests some essential pre-conditions for obtaining reliable and low bias estimates of PSM.These preconditions include: (1) data are collected on the basis of the same questionnaire for each group over the same time period; (2) treatment and control observations have identical socio-economic, demographic and agro-ecological characteristics; and (3) variables relevant to treatment and outcome are included in the propensity score function.This study qualifies condition (1) because the data used is sourced from IFLS 5. IFLS 5 is household data in Indonesia at the year of 2014.In addition, this study also uses as many control variables as possible to describe the characteristics of households observed so that the condition (2) are fulfilled.Furthermore, these variables are estimated in the propensity score function along with the treatment and outcome variables.
As in Heckman et al. (1997) and Smith & Todd (2001, 2005), we consider . We also assume D as the treatment variable equal to 1 if the household receives RASKIN and 0 which doesn't receive.This study will estimate the average impact of RASKIN on households receiving it (ATT): where X is a vector of the control variable.We suppose as the probability of respondents getting RASKIN.
Propensity score matching establishes statistical comparison groups by combining observations of RASKIN recipients to non-RASKIN recipients with equal P(X) values.The validity of this approach depends on the following two assumptions:  2) assumes conditional mean independence, which is conditional on non-RASKIN recipients having the same average outcomes as RASKIN recipients if they do not receive RASKIN.Rosenbaum & Rubin (1983) shows that if outcome is independent of the program recipients after conditioning the vector X, then the outcome will be independent of the program recipients after conditioning P(X) only.If equation ( 2) and (3) are true, then PSM is a valid method to estimate generating unbiased estimates of ATT.In addition, PSM is also said to be a valid method if the assumption of common support is met.Common support means that there is an overlap region between the density distribution of treatment and control group score.Each individu in the treatment and control group should have a similar probability.After the two assumptions are met, then PSM estimation can be done.There are five stages to estimate PSM, namely: When we want to estimate the propensity score, there are two choices that we must deal with.First, the model we use to estimate, then the variables we use in the model.In principle, all discrete choice models can be used.For the binary treatment case, where we want to estimate the probability of participants with non-participants, logit and probit models will produce the same estimate.However, the choice of the model doesn't matter to much, although more logit distributions have density mass in the bounds (Caliendo & Kopenig, 2008).Therefore, this study uses a logit model to estimate the propensity score.
The value of propensity score can be seen in table 5 below.This table shows that the logit model we use is good enough because many control variables significantly affect the treatment variables.The household income (hhincome), head of family's education (hhedu), ownership of private toilet (toilet), ownership of vehicle (vehicle), source of drinking water (water) and ownership of savings (savings) affect the chances of a household receiving RASKIN.Only ownership of vehicle that have a negative impact on the chances of a household receiving RASKIN, while other variables have a positive effect.
If the income of a head of family is lower than 600,000 Rupiah per month, then the chances of receiving RASKIN are greater.If a head of family is not attended school or doesn't finish primary school, then the chances of his/her household receiving RASKIN is greater.If a household doesn't have a private toilet at home, then the chances of receiving RASKIN are greater.If the source of drinking water of a household comes from unprotected wells/rivers/rainwaters, then the chances of receiving RASKIN are greater.If a household doesn't have savings, then the chances of receiving RASKIN are greater.Meanwhile, if a household has a vehichle, then the chances of receiving RASKIN are greater.
Electricity usage (electricity) and living in Java (java) have no significant effect on the chance of a household to receive RASKIN.This means that if a household has use electricity for lighting his/her home, his/her chances of receiving RASKIN are the same as a household with no electricity.Similarly, if a household lives in Java, the chances of receiving RASKIN are the same as households living outside Java.This indicates that poor households that should get RASKIN don't see by residence and electricity usage.From this figure it can be seen that households who receive RASKIN have similar characteristics before matching.This can be seen from the many overlapping propensity score.That is, the propensity score fulfills the conditional mean independence assumption based on the result.Before matching, there are little differences in household characteristics That we observed, but after matching the characteristics between treatment and control group should be significantly different.Table 7 below shows the result of t-test after matching.This table presents the average treatment and control group for each control variable are paired with the outcome variable.
The difference is seen from the percentage bias and its significance is seen in the t-test column.From this column, it can be seen that none of the control variables significantly affect the outcome variables.That is, after matching none of the characteristics treatment and control group were similar.Pseudo R squared illustrates how well the control variables explain the probability of participation (Caliendo & Kopenig, 2008) The results show that the data are not sensitive again at gamma 1,95 (see Table 8).This indicates that the data has been free from hidden bias.That is, our model is not sensitive to other factors outside the model.In other words, if there is a change in the value of the variables outside of the model, then the results of this study remain valid.

CONCLUSION
This study aims to evaluate the impact of RASKIN program on household income.These objectives were succesfully met, as we found that RASKIN program has a negative and significant effect on household income.That is, if a targeted household receives RASKIN, then the average household income is actually decrease.This is because the benefits received by the targeted households from income transfers are relatively small.The small amount of income transfer is due to the amount of rice received by the beneficiaries and the frequency of RASKIN they receive is less than the specified amount.In addition, the beneficiaries sometimes pay more than the normative price (Hastuti et al., 2008).With such facts, it is only natural that RASKIN doesn't raise the beneficiaries income.In other words, it can be said that RASKIN program has not been effective in increasing the targeted households income.Therefore, there must be a change in program format, not just the changing name from RASKIN to RASTRA.

Figure 1 .
Figure 1.Number and Percentage of Poor People in Indonesia, 1999-2015 Source: Calculation and Analysis of Macro Poverty Indonesia Year 2015

Figure 3 .
Figure 3. propensity score distribution and common support for estimating propensity score Source: Data Processed.

Table 3 .
Treatment and Control Group Characteristics