Track and Trace Alpha Field Test (Malawi) Report

 

Embed or link this publication

Description

Track and Trace Alpha Field Test (Malawi) Report

Popular Pages


p. 1

TRACK & TRACE ALPHA FIELD TEST, SEPT-OCT 2016 MALAWI End user “alpha phase” field testing in rural Malawi focusing primarily on software applications created by CSF and JSI to support the delivery process of textbooks to schools. Compiled by Sonny Lacey | October 11, 2016 sonny@coldwaterdesign.com Skype: sonny.lacey S

[close]

p. 2

TRACK & TRACE ALPHA FIELD TEST, SEPT-OCT 2016 MALAWI ALPHA TEST DEFINITION Software and system development can be broken into three general phases: alpha, beta and gold. These phases represent maturity of the product, scope of parties involved and depth of ancillary assets (such as training and support) that can be fielded. The alpha test is the first gate through which an application must pass where actual end users in their day-to-day environment are a part of evaluation. The alpha test, however, does not constitute a general availability of the application to a large population, nor does it assume that robust end user support systems are in place. An alpha test typically involves the application engineering team, since the product is not considered mature enough, and final user and system requirements are still being refined. An alpha test is essential prior to rolling out a software system, as is a beta test. The difference between alpha and beta testing is that the alpha test stresses the software, itself, while the beta test evaluates the impact on the ancillary human processes, business negotiation, training, end user recruitment and large-scale support. MALAWI ALPHA FIELD TEST METHODOLOGY The Malawi Alpha Field test involved a subset of the RTI/MERIT distribution of the Standard 1 English language textbook. This subset comprised a number of books that were to be delivered in the Dowa educational district, Chisepo zone. The schools chosen were the Chinziri, Chisepo, Mwaza and Thiwi Primary schools. All four schools had the following characteristics:  Rural  Not on the national electric power grid  Catchment areas including semi-literate parent groups  Part of a World Vision Area Development Program with strong community support  Within the distribution map of the first shipment of S1 English language textbooks (Al Ghurair print, RTI/MERIT procurement) PRODUCT PHASES: ALPHA = very limited user population, direct involvement from developers/softwarecentric BETA = larger user population and more use cases, limited involvement from developers/processcentric GOLD = ready for widespread use and distribution under all circumstances, robust support and training available 1

[close]

p. 3

 Within overlapping zones of cellular coverage (Airtel and TNM) as well as pockets of marginal cellular reception The primary use cases that were tested included:  Correct delivery to a school  Delayed, missed and wrong delivery to a school These two use cases formed the basis for the four canonical use cases that were developed for the primary software solution requirements (correct delivery, delivery stopped between two nodes, wrong destination delivery, missing delivery). The incorrect delivery that took place during the alpha test in essence “overlapped” the three negative use cases by demonstrating processes that are common to each case. Delayed, missed and wrong delivery cases all have a common element of the proper receiving school not taking reception of the books at a specific time. Points that are not common, however, were deemed as too difficult to test in the limited alpha setting (e.g. setting up multiple textbook transfer points to recreate a book “stuck” between two nodes). Local area map of a delivery route 2

[close]

p. 4

SCHEDULE OF TRAINING AND TESTING The alpha field test contained one week for training the highest-value stakeholder end user groups and one week demonstrating the use of the application on a small delivery of textbooks. TRAINING WEEK (SEPTEMBER 27 TO 30, 2016) Tuesday Wednesday Thursday Friday Planned Inspect S1 English book packaging, enter barcode data, train warehouse and courier personnel at Allied Freight: 8 people @ 2 hours Train teachers and community members at Dowa schools (Chinziri, Chisepo, Mwaza, Thiwi): 5 people each location @ 1 hour each location Train MOEST administrators on the use of system dashboard functions: 4 people @ 1 hour (eventuality and spillover day) Actual S1 English books not available (at MRA), trained warehouse and courier personnel at Allied Freight: 12 people @ 2.5 hours Trained teachers and community members at schools: 5-9 people at each location @ 1 hour each location; head teachers missing at two schools Trained MOEST ICT personnel on the use of system dashboard functions: 5 people @ 2 hours Reinforce/Trained Allied Freight couriers on use of system: 4 people @ 1 hour Reinforced/trained MOEST ICT personnel on the use of system dashboard functions: 5 people @ 1 hour Reinforce/Trained Allied Freight admins on use of system: 2 people @ 1 hour 3

[close]

p. 5

TESTING WEEK (OCTOBER 03 – 07, 2016) Monday Tuesday Wednesday Thursday Friday Planned Test delivery of book packages from Allied Freight warehouse to two schools (Chisepo, Mwaza); held back delivery to two schools (Chinziri, Thiwi) and monitored system for user input (when/where): 8 hours Test delayed delivery of book packages to Chinziri and Thiwi schools: 8 hours Observe MOEST administrators on use of dashboard reports: 4 people @ 2 hours ACR team debrief USAID mission (eventuality and spillover day) Actual Tested delivery of book packages from Allied Freight warehouse to two schools (Chisepo, Mwaza); held back delivery to two schools (Chinziri, Thiwi) and monitored systems: 9 hours Tested delayed delivery of book packages to Chinziri and Thiwi schools: 9 hours MOEST administrators called away for day-long meetings offsite Observed MOEST administrators on use of dashboard reports: 3 people @ 1.5 hours ACR team debriefed USAID mission ACR team interviewed Bollore Freight operations and administration team in Lilongwe headquarters CSF team questioned Airtel telco on toll-free/shortcode operations and cost 4

[close]

p. 6

LESSONS LEARNED FOR COMPLETING THE ALPHA TEST REQUIREMENTS In order to complete the alpha field test and receive prize monies, both development teams (CSF and JSI) had agreed to compile lists of items that were to be refined within fifteen days past the end of testing. While these lists are still forthcoming, both developers have agreed that the primary form of these refinements revolve around defect fixes and minor adjustments for user inputs and messages. The following items reflect what is generally thought to be within the realm of 2 weeks of development time. User Inputs SPELLING Several instances were observed where user inputs confused the system and resulted in non-delivery of status indicators or follow-up notifications. Some examples of spelling mistakes in these cases are as follows:  The user text input of “confirmeb” instead of “confirmed”  The user input of “recieved” instead of “received” STRINGS AND KEYS Additionally, the use of a space between a user input and an order number or school code hampered the system:  “delayed12345” instead of the system-recognizable form, “delayed 12345” The use of an alpha-numeric code for orders or class codes also proved to be troublesome for feature phone users in that text (SMS) entry for a singular letter character requires the user to hold a key down and secondarily select from a letter choice. While this is the same requirement for entering singular numeric digits, often times the numeral is the first choice and a feature phone user may enter it quickly:  “CFI1234” as opposed to an easier, all-numeric string such as “1234” SHORTENED AND ACTIVE VOICE Both development teams recognized the need for shortened user inputs that do not rely on verb tense as well as refining the inputs to be more semantically distinct:  “confirm” instead of “confirmed”  “partial” instead of “incorrect” (which is a superset of various states such as “partial,” “damaged,” “delayed,” etc.) System Outputs Both teams recognized the need to keep the system outputs (notification messages, delivery status messages and the like) succinct and meaningful. A long system output can easily go over the 160character limit in most SMS user interfaces. Likewise, a long message may require the user to scroll, which can be problematic in some lower-end feature phones. Keeping the main content of the message short and simple, while quite easy in theory and hindsight, is a challenge that will increase usability. 5

[close]

p. 7

LESSONS LEARNED AND MOVING TOWARD A BETA TEST Both systems performed admirably for the alpha test, and the lessons learned were heavily centered around discrete user experience factors that can be easily remedied. As we move toward a beta test, the expansion of the “test topography” will mean new coordination around the processes that were not tested in the alpha phase. While this is normal, a beta test that would encompass nearly all (if not all) of a country’s textbook distribution will require a new conversation around responsibilities for these expanded items. Expansion of Actors The following actors have been seen to be necessary when Track and Trace is considered for a beta test that involves a majority, if not the entirety of a country-wide procurement and distribution: PRINTER AS PACKAGE LABEL MAKER As the printer in the alpha test did successfully print the correct S1 English distribution label (see 6

[close]

p. 8

Appendix A), it was done so in an “out of band” manner that is not part of the envisioned process. The labels were agreed upon prior to the alpha field test, and the sequencing of bar code data was correctly applied (unique code for each package). In the cases where a bar code will be unique to an “order,” or a collection of textbook packages for a school delivery, this will have to be tightly coordinated with the printer who will, in fact, be an actor within the system. In this case, the printer will have to log in to the system to view the “orders” and identify how many packages exist under this logical encapsulation. Once this is identified, the printer will then attach the unique bar code for this order to these packages and print the labels to be affixed. In the cases where the bar code is generally unique (each package is different, irrespective of order, school or delivery), the printer will still have to associate orders with unique identifiers. Printers will be an integral actor when determining numbers of packages for orders and whether or not packages are split across containers. For delivery and distribution companies that trans-ship (Rollon/Roll-off, or RORO) containers directly to a distribution matrix, the printer is responsible for ensuring these logical encapsulations of orders are held together and that containers’ space use is maximized as well as adhering to the distributor’s delivery schedule (e.g. School A receives right before School B, so that their packages are ordered in the container first at print). IMPLEMENTING PARTNER AS ORDER ENTRY ACTOR In the Malawi alpha field test, the orders were entered directly by the software development teams, themselves after the books arrived in country. This was done because of the short time frame, the nebulous status of the book delivery, itself, and the need for a deeper coordination with RTI/MERIT. In a beta test, the implementing partner would function ideally as the order entry actor. It was thought in prior iterations that the MOEST would take on this function, but it became clear in the alpha test that the Malawi MOEST ICT members did not have this capability or bandwidth. The implementing partner, however, does have all of the necessary requirements for entering this data:  School name and location hierarchy  EMIS code  Textbook title and quantities The implementing partner must work hand-in-hand with the printer and distributor, therefore, to tie this order data to print run package labels and order association, respectively. Expansion of Endpoint Functionality The additional functionality described below is not strictly necessary for a beta test, but in the light of conversations with the USAID mission as well as Bollore Logistics, the following features may provide enough additional value to enhance adoption as well as serve to be determining criteria for choosing one solution: AUDIT FUNCTIONALITY FOR BOOKS As a means to gauge whether or not the delivered textbooks are being used (or not sold off, subsequently) in their school and community, an audit feature can be a useful item. This could take the form of having a deputized, or responsible community member (such as a PTA, SMC or Mother Group member) receive a message that asks for an audit of the quantity of a certain textbook title. Since the 7

[close]

p. 9

delivered quantity is a known factor within the system, having a “push” message feature to a known subscriber might appear as follows: 1. A flyer is sent out to a school’s area community (catchment zone) calling for responsible members to send a free message with the code *4321 2. A responsible community member who sends *4321 to the free number (shortcode) is registered within the system as the auditing actor for that particular school 3. 30 days after confirmed delivery and receipt of 180 copies of S1 English, a message is sent to the community member to “Please check the book locker and count the S1 English texts. Reply to this message with the number of books.” 4. The community member audits the book locker and responds to the system message with a number 5. The system reconciles the number with the previously-known delivered and received quantity and alerts higher-level actors (MOEST, etc.) if there are discrepancies AD-HOC TEXTBOOK STATUS/QUALITY MESSAGE FUNCTIONALITY Similar to the audit functionality, above, more automatic messages can be sent to registered community actors that are germane to textbook use or quality. An example of this is a message that is sent to the responsible actor of the form, “Please check the book locker for S1 Chichewa text books. Are any books damaged? (Yes / No)” Upon responding to such a message, the system may push more queries to the end user in the form of an expert system that assesses the type of damage, quantity and so forth. This can be done on a pre-determined and scheduled basis, or it may be pushed explicitly to a school or set of schools to satisfy a sampling metric. ELEMENTS OF FUNCTIONALITY AS “ADD-ONS” FOR FIRST-CLASS DISTRIBUTION As was found with the recent in-country textbook distribution by Bollore Transport and Logistics (S1 Chichewa), the success rate for delivery was said to be exceptionally high. Additionally, first-class logistics operations such as Bollore often have their own very advanced distribution tracking system (such as Bollore’s LINK system), and the additional burden of running a separate and concurrent tracking application may be untenable. The Track and Trace application may still find a niche within this ecosystem, wherein certain functional elements may be “broken out” from the main application to augment existing delivery confirmation systems. An example of this would be a system whereby the printer could create labeling through Track and Trace and the PTA members could audit textbook quantity. The delivery and receipt, however, could be selectively culled from the application and not used should the implementing partner or distributor warrant it. Expansion of Cost The expanded cost factors for moving from alpha phase to beta phase, regardless of the expanded functionality (above) are as follows: 8

[close]

p. 10

TELECOM AND AGGREGATOR COSTS: WHO PAYS? For a beta test, it is imperative that all actors utilize a toll-free and short code number. During the beta test, it was necessary to keep purchasing SMS credit for the couriers, teachers and parents, who typically did not have any credit at all to send an outgoing message. The costs for obtaining a toll-free and short code number for SMS messaging in Malawi through Airtel (one of the two main telecom providers) is as follows:  A setup fee of K150,000 (approximately USD208)  A monthly fee of K100,000 (approximately USD139) Additionally, paperwork and registration is expected to take up to a month. For a voice account, such as the use of Interactive Voice Response (IVR), an upfront deposit of K100,000 (approximately USD139) is necessary, as well as the following:  A certificate of registration from a local Malawian organization  A local address  A local bank account Voice messaging is charged at K55 per minute, typically. An aggregator is an attractive option for SMS messaging, however, due to the fact that much of the country is split between two main telecoms (Airtel and TNM), and in some districts, reception is better on one carrier than the other. An aggregator will give users the option of just one toll-free number and short code, instead of having to negotiate this with both Airtel and TNM in Malawi. The disadvantage of this arrangement is a higher SMS per-monthly fee. As of this writing, this exact number is not known, but it can be safe to be said it is higher than K100,000 per month. With either the option of a telecom or aggregator, it will be necessary to determine well in advance who will be paying for the setup and maintenance fees:  The implementing partner?  The solution provider?  Another entity? CIRCULARS OR POSTERS IN COMMUNITIES AND SCHOOLS: WHO PAYS? A beta test will involve a vastly larger quantity of schools than were tested in the alpha test. A typical beta test often will push endpoints up to three orders of magnitude from alpha test quantities. Where 4 schools were tested in this instance, nearly the entire country (5,000+ schools), if not the entire country may be involved. Sensitization of the community via circulars or posters is the only usable and scalable measure to both recruit users and train them on how to use the system. While circulars and posters can be procured with minimal cost, this must be gathered and the responsible party chosen. It must be borne in mind that circulars and posters, while attractive, may well need specific data printed that is distinct from school to school. An example of this is a code that a head teacher may input to confirm delivery of an order to his particular school. This unique data must be allocated when costing, as well as the necessary time it will take to compile these codes and allocate them to specific schools. 9

[close]

p. 11

Along with determining the paying party, it will be necessary to determine the party who will keep these school code-to-system associations correct. JSI/SALESFORCE USER COST: WHO PAYS? Lastly, as has been discussed in previous forums, one of the solution providers, JSI, operates on a platform that is not completely without cost. The SalesForce system does impose a per-user fee, and while this can be kept minimal, it is not without impact and furthermore requires a party to keep the user account information up-to-date. RELAY SERVICE: WHO PAYS? Both CSF and JSI rely on an architecture where the SMS “relay” must reside in either a phone or a server that is connected to the telecommunications infrastructure. During the alpha test, both solution providers deployed their respective SMS services on an Android phone incorporating a local SIM card and telephone number. While this is scalable to many hundreds of users, it remains to be seen whether or not this is usable for thousands of users and messages. Likewise, the fragility of this particular expedient is not recommended for a beta test (e.g. the phone could lose power, local reception can become compromised, a phone could reset for OS changes, etc.). Where a proper relay is deployed on a service, whether it be a local server or a cloud instance, the cost and maintenance will need to be determined, as well as the paying party System Inputs and Outputs IVR IN THE MALAWI EXAMPLE The use of Interactive Voice Response (IVR) was posed to several Malawian end users, and unanimously, in independent incidents, they responded that this was not a useful function. Primary among the objections against IVR was the fact that if a user were not at their phone to hear a message, it would not be recorded. Furthermore, in areas of minimal reception, IVR voice packets would become dropped whereas SMS messages could be delivered without loss. USER INPUTS IN ENGLISH AS OPPOSED TO CHICHEWA IN THE MALAWI EXAMPLE The use of English as a user input (and system output, or message) versus the native language, Chichewa was posed to all actors. Nearly unanimously, they all agreed that English was the clear choice. Primary among the positive aspects of English in the users’ minds was the face that an English phrase such as “confirm 2345” was shorter than its corresponding character string in Chichewa. Similarly, a text message that contains 50 characters in English may go over the 160-character limit in Chichewa. Training and Support Moving toward beta phase distribution will require a larger training and support footprint: HEAD TEACHERS NEED TO BE TRAINED/SENSITIZED: HOW INCENTIVIZED? In the alpha test, two instances (50% of schools tested) arose where the head teacher was not available for training. In one case, an alternate teacher was found and utilized, and in another case the trained teachers actually coached the head teacher through the book reception and confirmation process. 10

[close]

p. 12

Regardless of the end effect, it is plainly clear that the head teachers need “buy-in” on the system and must be sensitized to its use and properly trained. Where this takes place via circular or flyer, a certain amount of quality assurance must be maintained. Where thousands of schools are in question, this can only be done via a distributed and hierarchical system, best done through the District Education Managers, or DEMs. A country-wide sensitization of the DEMs, therefore, is a logical next step, however its implementation and cost is not clear. As before, the responsible parties for both paying and ensuring its continuity are in order; it is perhaps here that the MOEST is closest to becoming a responsible partner, but the incentive to do so is questionable. LOCAL-LEVEL SUPPORT WILL NEED TO BE SOURCED: WHO PAYS? Whereas both solution providers may employ technical support in English for their platforms, local-level support is an eventuality that cannot be overlooked. Local-level support for MOEST actors, distribution partners and implementing partners is something that cannot be passed along to the support that both organizations currently employ, especially when the packages reach gold status and multiple, concurrent deployments are underway. The recruitment of local-level support (even as low as one or two people) during a distribution is therefore necessary and must be costed. 11

[close]

p. 13

BETA TEST PROCESS RECOMMENDATIONS ONE SOLUTION Whereas two solutions were tested in the alpha phase, only one solution should be chosen for the large-scale test that a beta phase requires. It is recommended that both solution providers, CSF and JSI, submit their proposals for the previously-noted expansions of functionality (cost and development time) as well as any additional value that their solutions may bring to the table which will lower the impact on any of the other previously-mentioned actors or responsible parties. DEEPER INTEGRATION WITH IMPLEMENTING PARTNER Given the fact that the USAID (or other) mission in any country may be financially-challenged to help with a beta test, and the MOEST may have little incentive or bandwidth, it is recommended that a deep integration with the implementing partner for book procurement and distribution be obtained first and foremost. The alpha test revolved around the RTI/MERIT procurement of S1 English, and the fact that BlueTree personnel were available to assist proved invaluable. As has been discussed, the expanded actors and costs may fall more and more on the implementing partner. How this may be incentivized and provide value to these partners must be articulated well ahead of time and formalized. 12

[close]

p. 14

LESSONS LEARNED FOR GENERAL PROJECTS GOING FORWARD 1. The alpha field test proved to be both a necessary step for the Track and Trace product offering as well as a template for all such projects of similar scale. Going forward, this progression is recommended as a practice as well as something that should be communicated to solution providers in the earliest stages of a competition. 2. A sincere discussion about multiple alpha tests needs to be had, and in some cases two alpha tests in a context can provide much more meaningful requirements impact than the rollout of a beta test and the sunk costs and misleading expectations that it may engender; early requirements discovery, at the sacrifice of implementation time, can be easier to absorb, both financially and from an engineering perspective, than late discovery at a larger scale. 3. The knowledge of product phases should be made clear to competitors, both from a selfassessment standpoint as well as a formal progression that all parties, both the judging and solving, agree to both take part in. An example of this would be a competition that clearly spells out the product phases and asks for a self-selection by the developers as well as their plans for moving from phase to phase. 4. Cost factors for project that “go to scale” and involve a nation-wide adoption of a solution and processes is something that needs greater focus in the earliest stages of a competition. It is recommended that the associated expansio n factors and their costs are items that future solution providers provide in the initial, inception and judging stage. While many of these estimates may be ill-informed, generalized or naive, the above requirements refinements that have been discovered may speak toward a template for upscaling that can be utilized. 5. Incentivizing the end user is an important aspect to any application, however its importance cannot overshadow the incentivizing of a primary-paying actor. In this particular example, the incentives given to an implementing partner is something that would have borne deeper exploration, earlier. 13

[close]

p. 15

APPENDIX A Sample of book package label, as printed by Al Ghurair 14

[close]

Comments

no comments yet