Jonathan Kohl: Why do we need to test SMS messaging?
Paul Poutanen: Unfortunately, in a lot of cases it does not work. Again, according to Wikipedia, “… around 1 percent to 5 percent of messages are lost entirely, even during normal operation conditions, and others may not be delivered until long after their relevance has passed.”
The way it works is not simple, especially when looking at international SMS routing. There are over 1,000 carriers and network operators in the world. (No one ever seems to know how many.) If every one of those carriers had to make a contract with every other carrier, it would be very difficult (say, 1000!, or 4.0239 x 102567 contracts required). So, they make deals with SMS aggregators. For instance, in one country, a carrier may have a deal with five aggregators that agree to send and receive SMS to every country and carrier in the world. Those five aggregators may have agreements with twenty other aggregators that then have agreements with one hundred other aggregators. By the time the SMS gets to the end carrier, it may have gone through ten servers of aggregators. That would be considered the “route” of the SMS.
You think your SMS has made it through, and you have confirmation of that from the last link in the chain (the carrier). Hurray!
The end carriers are very cognizant of spam SMS. If they think an SMS might be spam, they may not let it through. However, they have sent a signal to the last aggregator that the SMS has been received to the carrier gateway. The aggregator believes this to be a signal that the SMS has been received and their service-level agreement (SLA) has been agreed to.
So, if you roll the dice and your SMS makes it through the last part of the chain, the carrier might block it but not tell you. It isn’t a black hole; in some cases, the carrier tells you the SMS made it to their gateway. That is what most SMS SLAs aim for.
You can see it is a mess, but it gets worse. This process of getting an SMS to the end handset is dynamic. SMS aggregators may change their routes every day, meaning a message that was successful when sent in the morning may not work in the afternoon.
Jonathan Kohl: How do you test SMS messaging?
Paul Poutanen: It is quite simple, if you have the infrastructure in place to manage it. At Mob4Hire, we’ve developed a combination of factors to test this. We have an enormous amount of testers from all over the world (the “crowd”) who have signed up their devices. We have a test system in place just for SMS message testing, and testers register their devices within that system. They install test software on their devices that helps us determine if the message arrived to them or not.
When we test messages, we confirm that SMS messages make it to a particular country and carrier by utilizing testers who use that carrier in that country. The test is simple. It’s binary. It’s a pass or fail. Did the SMS make it or not? Our server software sends indication back to the SMS customer we are testing for with proof the message worked on that particular route. We also show them what their message looked like, especially if they have been utilizing Unicode to cope with different languages.
Jonathan Kohl: One knock on crowdsourced testing is that you can’t guarantee skilled testing. What sort of skill do people need to be able to test SMS messaging?
Paul Poutanen: Our testers really just need to own and have simple operating knowledge of an Android handset that has data and carrier connectivity twenty-four hours a day. There is no skill required, other than basic handset knowledge and being able to download and install an app. There is the extra bonus of having a human at the other end who can actually observe and tell you about any strange problems they are having.
Jonathan Kohl: Why do we need to look at testing services like crowdsourcing for mobile testing, particularly when testing SMS? Why can’t we just test this internally within our organizations?
Paul Poutanen: SMS testing depends on testing a lot of platforms (a combination of a handset, carrier, and country). Testing only within your organization won’t tell you much, unless you have offices in many locations where your services need to be used. Testing SMS requires that you have handsets ready to receive an SMS in the countries and on carriers that you need to confirm. If you have people in 1,000 country-carrier combinations with handsets and paid subscriptions, then you could do it all yourself. This is incredibly difficult and expensive to replicate.
Jonathan Kohl: Remote-device-access services for handsets are becoming more popular. Do they work with SMS testing? What benefits does crowdsourced testing have over these services?