The Challenge
A company offers a service to scrub call lists for numbers from their country's Do Not Call Registry (DNCR). The regional DNCR does have a one-off service for this, but it is time consuming and difficult to manage at scale. The solution was to take advantage of the DNCR API and scrub lists in an automated fashion for quicker turnaround time and accuracy.
The Solution
RDD built a web based application, complete with registration and login, for subscribed users to upload their own call list where it would wash against the DNCR. The application offered users the ability to upload lists, map the data headers to a normalised storage engine (for analytics), and even shake out invalid numbers (so not to waste API calls). The highly automated back end, running on AWS, would ingest the list, normalise it, record which numbers were on DNCR, and return the list back to the user with the results. Once the list was processed, an email would be sent to the requestor (based on their login information) with direct links to the original list as well as the filtered one.
As the solution was dealing with potentially sensitive or personally identifiable information, precautions were taken in order to maintain a high security watermark. Communications were encrypted end-to-end using TLS (from the front end through to the database), and any components storing data at rest are also encrypted. Access management is locked down tight so that users can't access other users content.
Relational Database Service
For the master record of data, Amazon Relational Database Service (MySQL) was the client's preferred database for the sake of developer continuity. Using the RDS SSL certificates, all connections to the database were forcably encrypted from the Lambda functions that were responsible for reads and writes.
The database backs itself up nightly, and operates in a highly available mode such that if the primary database engine were to fail, the backup would automatically take its place. As data is synchronised between the two, this would result in no loss of data.
Lambda and Step Functions
The application itself is able to support a high volume of concurrent users, however, the DNCR API is single-threaded. In consideration for the possibility of multiple users requesting list washing in parallel, the requests for API consumption were queued, which requires that the washing operation is asynchronous.
To account for the asynchrnous need for the application, the work being done by Lambda had to be coordinated and tracked as each job made its way through the pipeline. To orchestrate this, AWS Step Functions became an integral part of the solution so that each user's request would not lose its state even if it were queued up behind others.
API Gateway
Decoupling the front end from the back end is easy with REST APIs. This application communicated entirely with the control plane over REST, completely secured using AWS IAM to ensure segmentation between users. Every single API call made had to be signed by the caller, and every single call is validated for permissions each time.
Cognito
User management for the application is made easy with Amazon Cognito. This service provides a turnkey user management system that was used to secure the underlying API against unathorised data access. Combined with API Gateway, Cognito was be used to issue credentials to the end user (upon successful login) to make authenticated calls to operate the application.
Cognito offers an easy to set up verification system for new users, password complexity and expiration enforcement, and proper storage of user-defined credentials in a one-way hash.