Background
Balochistan is the largest province of Pakistan by area, and the least developed. It suffers from several political, tribal and border conflicts. The distances to health facilities for the catchment population are long with limited accessibility. Immunization is one of the most cost-effective interventions to prevent deaths from vaccine preventable diseases (VPDs), especially in children. While Pakistan has an overall routine immunization coverage of 66 per cent for fully immunized children (FIC), coverage in the province of Balochistan is much lower at 29 per cent. This study aimed to assess the feasibility of introducing mHealth intervention using an artificial intelligence (AI) platform based on SMS (short-message service) and Interactive Voice Response (IVRs) to remind and persuade parents to get their children vaccinated.

Methods
We employed a mixed study design using both quantitative and qualitative approaches. Baseline data were collected from 1,600 eligible mothers/parents within the catchment areas of 75 basic health units (BHUs) in Quetta (provincial capital of Balochistan province), and the automated platform was instituted with SMS and IVRs from EPI (Expanded Programme on Immunization) Quetta. Daily reminders and IVRs were sent to the cell numbers in the database/records. Responses were noted on the AI platform. After a period of about two months, an end line survey of 1,203 participants was undertaken with a loss of 397 (25%) participants due to seasonal migration to warmer cities in Sindh province. For the qualitative part, three key informant interviews (KII) were conducted; two KIIs with Lady Health Supervisors and one KII with the WHO staff responsible for routine immunization. There were also three focus group discussions (FGDs).

Results
We found that the intervention was feasible as well as acceptable both at the community and programme management levels. The baseline indicators of immunization coverage improved significantly on end line survey (95% confidence interval, CI = 0.208-0.269; P<0.001).

Conclusions
Our study demonstrates the potential for mHealth and AI to improve childhood immunization and addresses equity in the least developed areas of this country. The replication of the strategy in subnational immunization programmes could decrease morbidity and mortality due to VPDs.

The Expanded Programme on Immunization in the Low- and Middle-Income Countries (LMICs), including Pakistan, aim to achieve Fully Immunized Child (FIC) through having 85% of the target population of the one who had received one dose of BCG, of Penta (including deptheria-tetanus- pertusis in addition to Influenza b and Hepatitis B vaccines) I,II,III; OPV I,II,III; and measles-1 by the age of one year.1,2 Electronic Health (eHealth) refers to the practice and strategy to support health care through communication and in- formation technologies. World Health Organization (WHO) considers eHealth initiative as a tool to strengthen health systems, improve health services delivery, and ensure ac- cess to those services.1 Mobile health (mHealth) is a subset, and it applies when mobile phones or any portable elec- tronic devices with software applications are used for health interventions. The mHealth interventions have been applied in several LMICs to ensure access to health services and to improve health status through providing health re- lated information, learning resources, reminders both to health workers, patients and communities.3–8 Use of mHealth in improving vaccination coverage has been in evolution, and there have some efforts in piloting or replicating evidence from within country’s or regional experi- ences even in Pakistan and neighbouring countries.9,10

Immunization is one of the most cost-effective inter- ventions to prevent deaths from vaccine preventable dis- eases (VPDs), especially in children.11 While Pakistan has an overall routine immunization coverage of 66 per cent for fully immunized children, coverage in the province of Balochistan is much lower at 29 per cent.12 This is due to several barriers including challenging geography, low liter- acy and a decreased demand for immunization.13,14 Con- ventional methods, such as passive follow-up or expecting parents that they would show-up by themselves for chil- dren’s vaccination, have failed to improve this situation. Balochistan is the largest province of Pakistan by area and the least developed. At the same time, it suffers from several political, tribal and border conflicts. The distances to health facilities for the catchment population are long with limited accessibility, and the facilities have little outreach. The fe- male literacy rate and knowledge about immunization is low in Balochistan (e.g., literacy just about 8 per cent),13 which may have made it difficult to for women in the province to develop a thorough understanding of immu- nization and the importance of herd immunity in communi- ties.15 Moreover, low awareness levels amongst healthcare providers, concern about vaccine safety and beliefs in lo- cal remedies are major demand-side barriers to immuniza- tion.15 Lack of timely information and low confidence in the quality of services are additional demand-side barriers.12,16 Outreach services are mainly offered by Lady Health Work- ers (LHWs) and there are only a limited number of vaccina- tors for immunizing pregnant mothers and children.

The mHealth interventions are not limited to vaccina- tion only. The technology has been effectively used reduc- ing perinatal mortality rate, increasing exclusive breast feeding, for nutrition, health behaviours, patient medica- tion compliance, maternal education and heath behaviour, and in strengthening health systems. Most of those inter- ventions were implemented and tested among the LMICs of Africa and Asia, including Asian Subcontinent.5,7,17–19

Literature shows that several supply and demand side interventions help to improve immunization cover- age.2,14,15,20–25 Certain demand side interventions such as immunization campaigns, monetary incentives, informa- tion transfers have been successful to increase the demand for vaccines and to improve vaccination coverage among children in several LMICs15,26,27 Various mHealth related interventions have been implemented to increase the up- take of vaccination by increasing awareness and providing relevant information about the time and places to get vac- cines through reminders.1,3–8,17,19,24,28,29 Although mHealth interventions have potential to improve vaccina- tion coverage by increasing demand, the evidence for such interventions is not adequate in Pakistan, and there is a lack of local robust research which could produce good ev- idence and inform the policy makers about the innovative

solutions to improve childhood immunization uptake by the community. Considering this gap, this implementation re- search aimed to assess the effectiveness of introducing an mHealth intervention using an Artificial Intelligence (AI) platform based on Short-message service (SMS) and Inte- grated Voice Recordings (IVR) to remind and persuade par- ents to get their children vaccinated. Successful implemen- tation of the innovation was hypothesized to 1) reduce delay in initiation of vaccination; 2) reduce the delay in sub- sequent vaccinations and 3) increase fully immunized child (FIC) achievement. This implementation research, partic- ularly, looked at the implementation of Artificial Intelli- gence-based mHealth technology (SMS and IVR) in immu- nization in Quetta City in Balochistan province. It assessed acceptability and compliance with the new technology; and determined the effectiveness of the intervention on immu- nization coverage and dropout rates.

METHODS
It was a mixed method study with intervention. Data were collected in Quetta City through a mixture of qualitative and quantitative approaches. Baseline data were collected from 1,600 eligible parents within the catchment areas of 75 basic health units (BHUs) in Quetta. We used Lot Quality Assurance Sampling methods (LAQS) used for maternal and child health surveys for random selection of 1600 eligible (having a cell phone) participants (parents) for quantitative part of the survey. The same technique was used in both the baseline and the end line parts of the study. The primary outcome to be measured was any percent increase in vacci- nation in the target population.

 

INTERVENTION
An automated platform was instituted with SMS and IVRs initiated by the Expanded Programme on Immunization (EPI) Quetta. The messages were developed in Urdu lan- guage (universally understood in Balochistan) by the ex- perts in their field. They were also pilot-tested to under- stand if they were easily understood by the receiver. A unique number (6007) was used to send out the IVR and SMSs to the Cell phones of the parents. There were three IVRs developed: (1) IVR to ask parents of they had got their child vaccinated for the due vaccine in the immunization schedule, (2) a public service message to persuade vaccina- tion, and (3) an IVR to the vaccinator reminding him/her of dropouts in their area of responsibility. Daily reminders and IVRs were sent to the Cell phone numbers of all the mothers/fathers listed in the database/records. Responses were noted on the AI platform. Asking if they had had vac- cinated their children, on options of “yes” or “no”, the AI would generate appropriate response i.e., thank you for a “yes”, and sending the corresponding IVR to a vaccinator to approach them in case of a “no”. After three days, the same parent was again approached by the IVR for a “yes” or “no” response. Baseline data of 1600 eligible mothers/par- ents within the catchment areas of 75 BHUs of Quetta were collected, and the AI platform was instituted with SMS and IVR from “EPI Quetta” and 6007 starting November 2017 as there were technical difficulties in initiating the system.

 

Table 1. Themes, sub-themes and categories emerging from the FGDs

Themes	Sub-themes	Categories
1.  Vaccination	•   What it means to you? •   Socio-cultural barriers •   Community Health Workers •   Role of parents	•   Reasons •   Access •   Awareness •   Delays •   Education
2.  Refusals to vaccination	•   Reasons •   How to convince?	•   Vulnerability to diseases •   Awareness •   Physical accessibility to centres
3.  mHealth for vaccination	•   SMS reminders •   Calls	•   Benefits •   Harms •   Usefulness •   Barriers
4.  Source of information	•   Media	•   Local language •   Awareness Other sources

The daily reminders and IVR were sent to the Cell numbers in the records. Responses were noted on the AI platform. After a period of about two months, an end line survey of 1203 participants was performed with a loss of 397 partic- ipants due to their seasonal migration to warmer cities of Sindh.

 

QUALITATIVE DATA COLLECTION AND ANALYSIS
After a period of about two months, an end line survey of 1,203 participants was undertaken with a loss of 397 partic- ipants due to seasonal migration to warmer cities in Sindh province. In addition to the quantitative part, qualitative data collection as also conducted. Our objectives, to con- duct the KIIs and the FGDs, were to explore barriers to the immunization and usefulness of mHealth, respectively. The question and guidelines for qualitative data collection were prepared, and piloted with three homogenous groups in an area not included in the survey. The questions were refined and adjusted accordingly. For the qualitative part, qualitative data were collected by data collectors who are experienced in conducting KIIs and FGDs and three key informant interviews (KII) were also conducted; including two KIIs with Lady Health Supervisors (LHSs) and one KII with the WHO staff responsible for routine immunization, were selected purposively to explore the barriers to immu- nization and usefulness of the mHealth initiative. In addi- tion, four focus group discussions (FGDs), containing 6-10 mothers with same literacy level, were conducted to com- plement the data coming from Key Informant Interviews (KIIs). Out of four, three FGDs were conducted with care- givers (mothers with a child) and one focus group with re- spondents from an ethnic minority from within a highly educated community. The FGDs and KIIs were conducted in national and local languages easily comprehended by the participants. Respondents were asked about their per- ception about immunization programme and their experi- ences regarding mHealth intervention. For both the KIIs and FGDs, field-tested guidelines were used. The interviews were recorded with prior permission, and themes were developed for further analysis. Qualitative data were analysed by two authors and thematic analysis was done by devel- oping codes and relevant themes. The main purpose of the FGDs was to explore the perception and practices of moth- ers regarding vaccination, usefulness of mHealth, SMS/call reminders, and barriers to timely vaccination. Analysis of the focus group discussions data yielded four main themes (Table 1).

 

QUANTITATIVE DATA COLLECTION AND ANALYSIS
The questionnaire for quantitative data collection had five parts: consent form, household information, sociodemo- graphic characteristics, immunization coverage form for in- fant immunization, and perceptions about childhood vac- cination (based on Health Belief Model). The questionnaire was piloted for reliability and subsequent modification in the questions, accordingly. Quantitative data were collected by trained data collectors using notes from interviews and e-recordings with prior consent from the participants. In case if later it was realized that some responses were miss- ing, the data collector tried approaching the respondent again. If a household refused to take part in providing data, or data remained incomplete, another household was ran- domly selected for data collection. The entire data collec- tion process was supervised by two of the authors. Quan- titative data were analysed to determine descriptive results (frequency distribution), for socio-demographic variables and for demand variables. Inferential statistics (t-test) was used to find any change from baseline to end line results. We also analysed for the prospective data of the unimmu- nized/immunized children through the intervening AI plat- form (IVR messages sent to the parents’ Cell phones and responses recorded). Data were collected, cleaned, coded and entered. Double entry in SPSS ensured correctness. All quantitative data and qualitative data (recording, notes) were kept confidential and in the possession of the principal investigator.

ETHICS CONSIDERATIONS
As this study involved personal information, we abided by the universal standards of ethical research. The Institu- tional Review Board (IRB) of the Health Services Academy was approached for the ethical permission for this research, which was obtained. We obtained written informed consent from all the participants and all records were kept confiden- tial. There was no risk of harm to participants who partici- pated in this research.

RESULTS
We randomly selected 1,600 children who were eligible ac- cording to the criteria among which 1,539 (96.2 per cent) were less than two years of age and 61 (3.8 per cent) were two or more years of age (see Table 2). Most of the respon- dents (both father and mothers of the children selected) were either illiterate or had studied up to the secondary level. About half of the respondents were employed (49 per cent), and about one third (37.9 per cent) owned a business. Some respondents (13.1 per cent) were unemployed. Re- garding the total household income of the respondents, most of them had an income of less than 30,000 Pakistani rupees (US$300) per month. Very few (8.8 per cent) had an income above 30,000 Pakistani rupees per month. Mobile handset ownership was high, as 67.8 per cent owned a basic handset and 30.8 per cent owned android mobile phones (Table 2).

AGE-WISE DISTRIBUTION FOR IMMUNIZATION
A majority of study respondents (91.6 per cent) had an im- munization card. Most of the respondents identified a BHU as the place for vaccinating their children. Bacillus Cal- mette-Guerin (BCG) vaccination coverage was highest (92.2 per cent) followed by oral polio vaccine (OPV) 0, 1, 2, 3 at 92 per cent, 86.1 per cent, 72.per cent, and 59.8 per cent re- spectively. More than half (59.1 per cent) of children had re- ceived an inactivated polio vaccine (IPV). Pentavalent 1, 2, 3 had been received by 86 per cent, 73 per cent and 60 per cent respectively. Pneumococal vaccination also followed the similar pattern of having Pneumo-1 being the most fre- quently received within all the three doses. However, at 9 months of age, only 25.2 per cent children were vaccinated for measles-1 and for measles-2, only 0.43 per cent of the children were vaccinated at the age of 15 months. A fully immunized child (FIC) is defined as the one who had re- ceived one dose of BCG, of Penta I,II,III; OPV I,II,III; and measles-1. For this, 398 (24.9 per cent) children were found to be fully immunized as per the definition stated earlier. Whereas, 1,202 (75.1 per cent) of the children were not fully immunized (Table 3).

Measles remains the least vaccinated disease in the study, with the first dose (measles-1) received by only 25.2 per cent and the second dose (measles-2) almost negligible, with only 0.5 per cent of the surveyed children vaccinated. For about 400 (25 per cent) of the children surveyed, the source of the vaccination for measles-1 was BHUs, but for measles-2, the source of vaccination was only 7 (0.4 per cent) for BHUs. No sources were reported for measles-1 and measles-2 among 1,196 (74.8 per cent) and 1,589 (99.3 per cent) of the children, respectively. Only 458 (28.6 per cent) of the children were found to be fully immunized before the age of one.

ARTIFICIAL INTELLIGENCE INTERVENTION REDUCES DELAY IN FOLLOWING THE VACCINATION SCHEDULE
Findings showed a percent increase in coverage for pentava- lent, OPV and pneumococcal vaccines for first, second and third doses/boosters from the baseline to the end line data. This reflects that the delay in following vaccinations was re- duced by the AI intervention (Figure 1).

REDUCTION IN SUBSEQUENT VACCINATION AND IMPROVEMENT IN FIC STATUS
Following use of the AI platform, there was an increase in achieving full immunization coverage for both measles-1 and measles-2. The increase in measles-2 vaccination rates were highly encouraging, with an improvement from 0.43 per cent to 22 per cent among the sampled population. De- lays in initial vaccinations and delays of subsequent vac- cinations were reduced. The AI platform reminded parents about vaccinations even if they had forgotten the vaccina- tion date or lost the vaccination card. It was also found that the AI platform facilitated sharing responsibility of vaccina- tions between both parents (Table 4).

FULLY IMMUNIZED CHILD (FIC) ACHIEVEMENT TREND
A fully immunized child is defined as the one who had re- ceived one dose of BCG, of Penta I,II,III; OPV I,II,III; and measles-1. There was a marked improvement in the FIC attainment following the intervention (Figure 1)(Table 4). Furthermore, second dose of measles was markedly higher after the intervention.

RESPONSE OF THE CAREGIVERS TO IVR
Only for the IVR purpose, out of 1,600 respondents of the baseline survey, 1,314 were found eligible for the interven- tion of IVR after removing 286 duplicates numbers. The 1,314, received messages from EPI Quetta. Out of the 1,314, only 572 (43.5% per cent) responded to the IVR (Figure 2), and (Figure 3).

Out of those who did not respond (742), 64 per cent did not pick up the call, 29 per cent did not choose any option and 7 per cent were busy tones (Figure 4). Some of the main reasons for lack of response, when explored during FGDs, were the low literacy rate of mothers; mobile phones being with their husbands, which rang when their husbands were at work; and not knowing how to respond further.

Out of those who responded, most (80 per cent) re- sponded mentioned that they had their child vaccinated on the due date (see Figure 1). This aspect was explored during key informant interviews and the FGDs, and it found that the timely awareness messages and IVR had persuaded par- ents to bring their children to the health centre for the due vaccinations.

 

Table 2. Socio-demographic characteristics of the study participants (n=1600) [baseline] 

Characteristics	Category	Number of participants (%)
Gender of children	Male	813 (50.8)
	Female	787 (49.2)
Father’s age	Less than ≤ 35 years	1,007 (62.9)
	More than 35 years	593 (37.1)
Mother’s age	Less than ≤ 30 years	1,080 (67.5)
	More than 30 years	520 (32.5)
Place of birth	Hospitals	1,082 (67.6)
	Private health facility	303 (18.9)
	Home delivered	215 (13.4)
Education	Illiterate	445 (27.8)
	Informal	96 (6.0)
	Primary (until grade V)	182 (11.4)
	Middle (until grade VII	198 (12.4)
	Secondary	450 (28.1)
	Graduate (college)	177 (11.1)
	Post-graduate (university)	52 (3.3)
Occupation status	Employed	784 (49.0)
	Unemployed	209 (13.1)
	Business	607 (37.9)
Total household income (PKR)*	Less than 15,000	872 (54.5)
	16,000 – 30,000	588 (36.8)
	31,000 – 45,000	83 (5.2)
	More than 45,000	52 (3.3)
	No income	05 (0.3)
Ethnicity	Baloch	522 (32.6)
	Pakhtoon / Pathan	366 (22.9)
	Punjabi	145 (9.1)
	Urdu speaking	98 (6.1)
	Sindhi	63 (3.9)
	Others	406 (25.4)
Vaccination status of the child (as told by the mother)	Non-immunized	759 (47.4)
	Partially	774 (48.4)
	Fully	67 (4.2)
Type of mobile owned	Basic handset	1,085 (67.8)
	Smart phone	492 (30.8)
	None	23 (1.4)
Immunization Card	Yes	1466 (91.6)
	No	134 (8.4)

* (1 PKR = 0.01 US Dollar)

RESULTS OF KEY INFORMANT INTERVIEWS AND FOCUS GROUP DISCUSSIONS

The interviews and FDGs in Balochistan identified barriers and enablers to immunization. One main reason mentioned by participants was the extremely low level of awareness of immunization. Other barriers to immunization mentioned were distance to the health centres, transportation, gender discrimination, and non-trained vaccinators that led to fears of about vaccine safety.

Table 3. Distribution of vaccination doses among children (N=1,600)

Characteristics	Vaccination	Number of children (%)
Age	Less than ≤ two years old	1,539 (96.2)
	More than two years old	61 (3.8)
At birth	BCG	1,475 (92.2)
	OPV-0	1,474 (92.1)
At 6 weeks	OPV-1	1,378 (86.1)
	Pentavalent-1	1,377 (86.1)
	Pneumococcal-1	1376 (86.0)
At 10 weeks	OPV-2	1,161 (72.6)
	Pentavalent-2	1,164 (72.8)
	Pneumococcal-2	1,157 (72.3)
At 14 weeks	OPV-3	956 (59.8)
	Pentavalent-3	956 (59.8)
	Pneumococcal-3	957 (59.8)
	IPV	946 (59.1)
At 9 months	Measles-1	403 (25.2)
At 15 months	Measles-2	8 (0.43)
Fully Immunized Child	One dose of BCG, of Penta I,II,III; OPV I,II,III; and measles-1	398 (24.9)

Table 4. Significant Improvement at the End line Survey in BCG, IPV and being FIC from the Baseline Survey

Vaccine	Baseline	End line	T-test	P-value	CI
BCG	1600	1203	8.608	0.00	0.106-0.169
IPV	1600	1203	18.863	0.00	0.543-0.668
Fully Immunized Child	1600	1203		0.00	0.202-0.275

Figure 1. Vaccination coverage pre- and post-intervention surveys

 One LHS said,

“parents find EPI centres far from their reach, they have problems with transportation, lose their vaccination cards, and are less educated, especially the mother. Even if they reach the EPI centre, most of the vaccinators are not well trained. Wrong injections do occur resulting in injury and disability to the child. Myths and religious beliefs are also among the barriers.” She further added “IVR would be better than SMS in motivating and educating commu- nity, and if it were in local languages, it would have been more acceptable. Nonetheless, the intervention has visible impact on an increase and timely vaccination of the chil- dren in communities, more being among the highly edu- cated subgroups of the populations”.

One of the WHO officers highlighted the importance of mHealth, and its demand creation and acceptability:

“We need to create the demand of mHealth in the govern- ment department s well as in the community. We have al- ready proposed “e-vac” system of the Punjab government but there are issues of not having enough resources. An af- fordable technology, for poor province such as Balochis- tan, might be promising.”

Regarding the SMS and calls, the majority of the partic- ipants found them useful reminders for the timing of vac- cination for their children. According to the mothers, regis- tered contact numbers of mobile phones were usually with the fathers and hence that way they got timely informed about vaccinating the child. In some instances, even if the father was working outside or was not available, he could later inform the mother or the family to get the child vac- cinated at the due date and time. This was considered a big advantage of using AI in getting children vaccinated. Some mothers suggested that the importance of getting vaccina- tions could be increased by making the vaccination card as document needed for eligibility for school admission or for applying for passports in the country.

Generally, participants viewed the use of AI as beneficial in ensuring that children get all the needed vaccines. Some of the mothers were annoyed with the repeated calls and suggested that calls should be on the date of immunization only. Participants suggested that the AI should be further developed with inclusion of child’s photo as a proof of iden- tity, followed by SMS confirmation and data updated in the electronic records.

DISCUSSION

ACCEPTABILITY OF THE AI INTERVENTION

The community, especially the mothers, appreciated the new approach of sending reminders for vaccinating their children. There was a high level of acceptance and desire to continue using the SMS and IVR services. One possible rea- son is that during the last decade ownership of a handset among the populations has increased enormously, partic- ularly among LMICs. This has provided an opportunity to the health professionals to reach out to communities, target groups and vulnerable populations with ease and effective- ness.30 In line with the results of this study, several other studies demonstrated that mHealth interventions have high acceptability by the caregivers as well as the health workers in multiple settings.5,7,17,29,31–33 The intervention was equally accepted by the caregivers with low as well as high literacy rate and among different socio-economic popula- tion.

Figure 2. Number eligible respondents for the IVR

Figure 3. Number of participants who responded to IVR following regular awareness messages

 

Figure 4. Distribution of reasons for not responding to IVR

 

EFFECTIVENESS OF AI INTERVENTION

 

REDUCED DELAY IN INITIATION OF VACCINATIONS
It is well-documented from a number of countries that de- lays in initiation of vaccinations is very common. The DPT3 vaccine has been found to be delayed by almost 11 weeks in LMICs. This delay is more pronounced for newly intro- duced vaccines.34 In our study, despite BCG and OPV0 usu- ally having good coverage (above 80 per cent) in most of the cases, it was encouraging that the intervention improved this to 99 per cent coverage. Evidence showed that the booster dose on the designated date was also improved, in- creasing coverage of second and third boosters of pentava- lent, OPV and pneumococcal vaccines. The results show that uptake of both the first and second doses of vaccines increased remarkably after the intervention, especially for the second dose of measles. This is significant because measles compliance remains compromised and delays lead to gaps in immunity and frequent outbreaks of the vaccine preventable diseases. Different approaches and integration of efforts and programmes with innovative solutions are frequently recommended in the literature for LMICs. Use of mHealth technology to initially bypass and then to sup- port the health systems in LMICs, has resulted in decreasing delays in service delivery. Many studies have documented shortening the delays in accessing healthcare and health services, both for curative and preventive medicine. Sys- tematic reviews conducted for effectiveness of mHealth in improving immunization in infants and for maternal health3,5,9,25,28,35

REDUCED DELAY IN FOLLOWING THE VACCINATION SCHEDULE
We found that most of the mothers did have the immuniza- tion cards with them but only a few had fully vaccinated children in baseline survey showing non-compliance with the printed immunization schedule. Missing the initial dose of an antigen typically results in missing booster or next doses of the same antigen, and its concomitantly admin- istered vaccines.36–38 Following the timelines for vaccina- tions is imperative to maintain good herd immunity in a population. Other studies from different settings showed that differences in achieving the next date of vaccination has implications for infant and childhood morbidity and mortality.34,39 There is a limit to delay (up to two weeks to a month) and beyond that immunity cannot be boosted ap- propriately, and a child becomes susceptible to disease.40 Parents’ education and awareness levels, distance from the facility, vaccine stocks, vaccinators’ education levels, and location of birth can potentially contribute to delays in timely immunization.2,38,41

Findings of the study explored whether delay for the fol- lowing vaccinations was shorten after introduction of the AI intervention. By comparing the vaccination percentages for each antigen between baseline data and the end line data, and understanding from the KIIs and the FGDs, it was evident that the delays for the following vaccination dates were reduced markedly. Reminders through SMS and calls helped to sensitize and inform parents about vaccination and vaccination schedule which ultimately increased their demand for, and uptake of vaccines. In many cases, fathers were the owner of the mobile phone and mothers did not have the access to it. However, despite being a patriarchal society, fathers showed a positive attitude and delivered the information related to vaccination to their wives and sup- ported them to uptake the required vaccines for their chil- dren. Similar experiences were documented in other LMICs which showed that SMS reminders have resulted in a de- crease in dropout rates and reduced delays in vaccination schedules.8 Ethnic communities from developed nations have also used simple tools, such as calendars, to improve the vaccination coverage effectively.38

FULLY IMMUNIZED CHILD ACHIEVEMENT TREND
Vaccinations on time lead to achievement of a fully immu- nized child (FIC) among children under two years of age.

Children from developed as well as from developing coun- tries fail to achieve this status due to many factors as ex- plained previously. Inadvertent delays between DPT1 and DPT3, and measles lead to typical lag and delay in achieving FIC status among children.2,38,40,41 Receiving measles vac- cinations along with BCG, OPV I, II, III, Penta I, II,I II, and Pneumo I, II, III are considered to constitute FIC as de- fined by WHO. The comparative results showed an upward trend in the uptake of all the antigens for FIC, especially for third doses and measles-1 and 2. Barriers to FIC include parental education, parental attitude and knowledge, fail- ure of the immunization system, and lack of proper com- munication and information. Several literature showed that timely information and communication can improve vacci- nation by one dose which are in line with the findings of this research.20,21 Awareness and reminders through mHealth initiatives help in sensitizing and raising awareness among the caregivers about vaccine, its availability and providing timely reminders on when they are due which positively influence in improving FIC coverage. However, few studies showed that mHealth interventions did not have much im- pact on the improvement of the FIC coverage. A study con- ducted in 2014 in Karachi showed a low response rate to SMS and reminders. However, the situation has improved over the period as the format and language of the messages were improved.26

INCREASED DEMAND TO EXPAND AI INTERVENTION

Given that about 100% of the world population are within mobile network several mHealth interventions have been designed to leverage mobile phone penetration in LMICs to improve healthcare access and dissemination of health- related information. Many of the interventions have been successful and demonstrated an improved vaccination cov- erage in rural hard-to-reach areas.6 Results of this study showed that health professionals and implementers sug- gested for scaling-up of the AI intervention. Participants in FGDs also highlighted the importance of expanding SMS and IVR based services to other areas in addition to immu- nization. It was because of the acceptability and feasibil- ity of the intervention as it shown it other studies.3–6,17,25 However, further improvements leading to completely com- puterized vaccination card system, LHWs and vaccinators having children’s vaccination cards on their handsets, and regular reminders to parents on the due date were some of the recommendations from the major stakeholders of im- munization programme.

STRENGTHS AND LIMITATIONS OF THE STUDY

The study population had both illiterate and educated mothers, which provided a broader understanding in the KIIs and FGDs. Mothers welcomed the SMS and IVR, and tried to comply accordingly. Data collection was as accurate as possible. A comparatively large sample size was also a strength.

There were challenges in the implementation of the in- tervention. The telecom service provider could not expand it to all cellular networks. Later in the project, a universal SMS portal was used for awareness outreach to all of the registered sample population. Targeting each child with specific SMS or IVR was not possible in the current set up of the AI, although the set up improved at the end of the pro- ject. There was also a loss of 28 children who died due to non-vaccine-related causes during the study. About 300 of more families left the intervention area during winter break to go to Karachi and other cities in Sindh where they usu- ally go to meet their relatives and enjoy a less cold weather. Therefore, at the end line we had a smaller sample size of 1,203 participants instead of the 1,600-sample size we had for the baseline. One more limitation of our study was an absence of a comparison group, which however, was not predetermined for the current study.

CONCLUSIONS

This study showed marked improvement in the up-take of vaccination for the children following the introduction of the AI-based intervention. Discussions with the mothers and other stakeholders showed the intervention to be widely accepted and an important tool in increasing de- mand for vaccination in the community. The intervention has the potential to become more technically sound and user friendly. The potential for using technology to improve coverage is promising. However, few suggestions to con- sider for : 1) AI-based intervention has the potential to improve immunization coverage and equity in immuniza- tion, and to generate demand and address vaccine-related vaccine hesitancy, the technology should be scaled up in Balochistan and subsequently across Pakistan; 2) As the AI- based intervention is implemented and scaled up, lessons learned should be well documented to allow other locations to benefit from the initiative; and 3) The technology should be developed based on working with the community to meet the needs expressed at the local level.

ACKNOWLEDGEMENTS
We acknowledge the Alliance for Health WHO, the UNICEF, the GAVI and the Health Services Academy Islamabad for providing grant to conduct this research. We also acknowl- edge Faraz Khalid, Alyssa Sharkey and Saadia Farrukh from the UNICEF for their continued support, assistance and feedback during the development of this article.

ETHICS AND CONSENT

Health Services Academy’s Institutional Review Board (IRB) approved the study.

FUNDING

Grant by Alliance for Health Policy and System Research, World Health Organization, Geneva, UNICEF and GAVI, the Vaccine Alliance.

AUTHORSHIP CONTRIBUTIONS

EAK and MIP and BS contributed to the conception, design and drafting of the work; EAK, AS and SA revised it critically for important intellectual content and approved the version to be published.

COMPETING INTERESTS

The authors have completed the Unified Competing Interest form at www.icmje.org/coi_disclosure.pdf (available on re- quest from the corresponding author) and declare no com- peting interests.

CORRESPONDENCE TO

Ejaz Ahmad Khan, MBBS, MPH, FFPH. Associate Professor Health Services Academy, Prime Minister’s National Health Complex, Park Road, Chak Shahzad, Islamabad, Pakistan. Email. ejaz@hsa.edu.pk

REFERENCES

1. World Health Organization. mHealth: New horizons for health through mobile technologies. Observatory. Published online 2011. doi:10.4258/hir.2 012.18.3.231

 

2. Mvula H, Heinsbroek E, Chihana M, et al. Predictors of uptake and timeliness of newly introduced pneumococcal and rotavirus vaccines, and of measles vaccine in rural Malawi: A population cohort PLoS One. Published online 2016. doi:1 0.1371/journal.pone.0154997

 

3. Oliver-Williams C, Brown E, Devereux S, Fairhead C, Holeman I. Using mobile phones to improve vaccination uptake in 21 low-and middle-income countries: Systematic review. JMIR mHealth uHealth. 2017;5(10). doi:10.2196/mhealth.7792

 

4. Lee SH, Nurmatov UB, Nwaru BI, Mukherjee M, Grant L, Pagliari C. Effectiveness of mHealth interventions for maternal, newborn and child health in low- and middle-income countries: Systematic review and meta-analysis. J Glob Health. 2016;6(1):10401. doi:10.7189/jogh.06.010401

 

5. Colaci D, Chaudhri S, Vasan A. mHealth Interventions in Low-Income Countries to Address Maternal Health: A Systematic Review. Ann Glob Heal. 2016;82(5):922-935. doi:10.1016/j.aogh.2016.09.001

 

6. Uddin MJ, Shamsuzzaman M, Horng L, et Use of mobile phones for improving vaccination coverage among children living in rural hard-to-reach areas and urban streets of Bangladesh. Vaccine. 2016;34(2):276-283. doi:10.1016/j.vaccine.2015.11.024

 

7. Lee SH, Nurmatov UB, Nwaru BI, Mukherjee M, Grant L, Pagliari C. Effectiveness of mHealth interventions for maternal, newborn and child health in low– and middle–income countries: Systematic review and meta–analysis. J Glob Health. 2016;6(1):10401. doi:10.7189/jogh.06.010401

 

8. Domek GJ, Contreras-Roldan IL, O’Leary ST, et SMS text message reminders to improve infant vaccination coverage in Guatemala: A pilot randomized controlled trial. Vaccine. 2016;34(21):2437-2443. doi:10.1016/j.vaccine.2016.03.065

 

9. Uddin MJ, Saha NC, Islam Z, et al. Improving low coverage of child immunization in rural hard-to- reach areas of Bangladesh: Findings from a project using multiple interventions. Vaccine. 2012;30(2):168-179.

10. Kim SS, Patel M, Hinman Use of m-Health in polio eradication and other immunization activities in developing countries. Vaccine.

2017;35(10):1373-1379. doi:10.1016/J.VACCINE.2017.01.058

 

11. Rappuoli R, Mandl CW, Black S, De Gregorio Vaccines for the twenty-first century society. Nat Rev Immunol. 2011;11(12):865-872. doi:10.1038/nri3085

 

12. National Institute of Population Studies (NIPS) [Pakistan] and ICF. Pakistan Demographic and Health Survey 2017-18.; 2019.

 

13. Kim J, Alderman H, Orazem P. Can cultural barriers be overcome in girls’ schooling? The community support program in rural Balochistan. World Bank Dev Res Gr. Published online 1998.

 

14. Bugvi AS, Rahat R, Zakar R, et al. Factors associated with non-utilization of child immunization in Pakistan: evidence from the Demographic and Health Survey 2006-07. BMC Public Health. 2014;14(1):232.

 

15. Gilmore B, McAuliffe E. Effectiveness of community health workers delivering preventive interventions for maternal and child health in low- and middle-income countries: A systematic review. BMC Public Health. Published online 2013. doi:10.118 6/1471-2458-13-847

 

16. Murtaza F, Mustafa T, Awan R. Determinants of nonimmunization of children under 5 years of age in J Fam Community Med. Published online 2016. doi:10.4103/2230-8229.172231

 

17. Higgs ES, Goldberg AB, Labrique AB, et al. Understanding the Role of mHealth and Other Media Interventions for Behavior Change to Enhance Child Survival and Development in Low- and Middle- Income Countries: An Evidence Review. J Health Commun. 2014;19(sup1):164-189. doi:10.1080/108107 2014.929763

 

18. Zhou H, Sun S, Luo R, et Impact of text message reminders on caregivers’ adherence to a home fortification program against child anemia in rural western China: A cluster-randomized controlled trial. Am J Public Health. 2016;106(7):1256-1262. do i:10.2105/AJPH.2016.303140

 

19. Labrique AB, Vasudevan L, Kochi E, Fabricant R, Mehl G. Mhealth innovations as health system strengthening tools: 12 common applications and a visual framework. Glob Heal Sci Pract. Published online 2013. doi:10.9745/GHSP-D-13-00031

 

20. Oyo-Ita A, Nwachukwu CE, Oringanje C, Meremikwu Cochrane Review: Interventions for improving coverage of child immunization in low- and middle-income countries. Evidence-Based Child Heal A Cochrane Rev J. 2012;7(3):959-1012. doi:10.100 2/ebch.1847

 

21. Rainey JJ, Watkins M, Ryman TK, Sandhu P, Bo A, Banerjee K. Reasons related to non-vaccination and under-vaccination of children in low and middle income countries: Findings from a systematic review of the published literature, 1999-2009. Vaccine. Published online doi:10.1016/j.vaccine.2011.0 8.096

 

22. Oyo-Ita A, Wiysonge CS, Oringanje C, Nwachukwu CE, Oduwole O, Meremikwu MM. Interventions for improving coverage of childhood immunisation in low- and middle-income countries. Cochrane Database Syst Rev. Published online 2016. d oi:10.1002/14651858.CD008145.pub3

 

23. Ozawa S, Yemeke TT, Thompson KM. Systematic review of the incremental costs of interventions that increase immunization Vaccine. Published online 2018. doi:10.1016/j.vaccine.2018.05.030

 

24. Johri M, Pérez MC, Arsenault C, et Strategies to increase the demand for childhood vaccination in

low- and middle-income countries: A systematic review and meta-analysis. Bull World Health Organ. Published online 2015. doi:10.2471/BLT.14.146951

 

25. Lassi ZS, Middleton PF, Crowther C, Bhutta ZA. Interventions to Improve Neonatal Health and Later Survival: An Overview of Systematic Reviews. EBioMedicine. 2015;2(8):985-1000. doi:10.1016/j.ebio 2015.05.023

 

26. Kazi A, Murtaza A, Khoja S, Zaidi A, Ali Monitoring polio supplementary immunization activities using an automated short text messaging system in Karachi, Pakistan. Bull World Health Organ. Published online 2014. doi:10.2471/blt.13.122564

 

27. Galadima AN, Zulkefli NAM, Said SM, Ahmad N. Factors influencing childhood immunisation uptake in Africa: a systematic BMC Public Health. 2021;21(1):1475. doi:10.1186/s12889-021-11466-5

 

28. Manakongtreecheep K. SMS-reminder for vaccination in Africa: research from published, unpublished and grey literature. Pan Afr Med J. 2017;27(Suppl 3):23. doi:10.11604/pamj.supp.2017.27.3.12115

 

29. C LSNUNBMMGLP. Effectiveness of mHealth interventions for maternal, newborn and child health in low- and middle-income countries: Systematic review and meta-analysis. TT -. J Glob Health. 2016;6(1):10401. doi:10.7189/jogh.06.010401

30. Crocker-Buque T, Mindra G, Duncan R, Mounier- Jack S. Immunization, urbanization and slums – a systematic review of factors and interventions. BMC Public Health. 2017;17(1):556. doi:10.1186/s12889-01 7-4473-7

 

31. Zhou H, Sun S, Luo R, et Impact of text message reminders on caregivers’ adherence to a home fortification program against child anemia in rural western China: A cluster-randomized controlled trial. Am J Public Health. 2016;106(7):1256-1262. do i:10.2105/AJPH.2016.303140

 

32. Tamrat T, Kachnowski S. Special delivery: An analysis of mhealth in maternal and newborn health programs and their outcomes around the Matern Child Health J. 2012;16(5):1092-1101. doi:10.1 007/s10995-011-0836-3

 

33. Hall CS, Fottrell E, Wilkinson S, Byass P. Assessing the impact of mHealth interventions in low- and middle-income countries – what has been shown to work? Glob Health Action. 2014;7(1). doi:10.3402/gh v7.25606

 

34. Clark A, Sanderson C. Timing of children’s vaccinations in 45 low-income and middle-income countries: an analysis of survey data. Lancet. 2009;373(9674):1543-1549. doi:10.1016/S0140-6736(09)60317-2

 

35. Saraiva FO, Minamisava R, Vieira MA da S, Bierrenbach AL, Andrade Vaccination Coverage and Compliance with Three Recommended Schedules of 10-Valent Pneumococcal Conjugate Vaccine during the First Year of Its Introduction in Brazil: A Cross- Sectional Study. PLoS One. 2015;10(6):e0128656. doi:10.1371/journal.pone.0128656

 

36. Bailly AC, Gras P, Lienhardt JF, et al. Timeliness of vaccination in infants followed by primary-care pediatricians in France. Hum Vaccines Immunother. Published online 2018. doi:10.1080/21645515.2017.14 09318

 

37. O’Donnell S, Dubé E, Tapiero B, Gagneur A, Doll MK, Quach C. Determinants of under-immunization and cumulative time spent under-immunized in a Quebec Vaccine. Published online 2017. doi:10.1016/j.vaccine.2017.08.072

 

38. Gibson DG, Ochieng B, Kagucia EW, et Individual level determinants for not receiving immunization, receiving immunization with delay, and being severely underimmunized among rural western Kenyan children. Vaccine.

2015;33(48):6778-6785. doi:10.1016/j.vaccine.2015.10.021

 

39. Buttery JP, Graham Immunisation timing: the protective layer in vaccine coverage. Lancet. 2009;373(9674):1499-1500. doi:10.1016/S0140-6736(09)60340-8

 

40. Gras P, Bailly AC, Lagree M, Dervaux B, Martinot A, Dubos F. What timing of vaccination is potentially dangerous for children younger than 2 years? Hum Vaccin Immunother. 2016;12(8):2046-2052. doi:10.1080/21645515.2016.1157239

 

41. Mohammadbeigi A, Mokhtari M, Zahraei SM, Eshrati B, Rejali M. Survival Analysis for Predictive Factors of Delay Vaccination in Iranian Children. Int J Prev Med. 2015;6:119. doi:10.4103/2008-170868