The Story
Around 2009, we worked with a Panamanian company dedicated to advising campaigns for political candidates, from different public positions throughout the American continent. One of the strengths of his services was the analysis of the coverage that his clients received in the press. It is a practice that has always been used in the management of public image and is very useful for the analysis of the opinion that dominate the media during electoral campaigns.
This practice has existed for decades, however, at that time web applications were beginning to dominate the software industry and the benefits of being able to take processes to the internet were very attractive. The problem was that the process of analyzing the daily press is a process that needs to happen in real time, specially during a political campaign. It requires the insertion of a lot of data with great precision and then, it is the combination of these variables that provides the necessary information for who is acttually making decisions on that stream of data.
So, it was necessary that at the beginning of the day all the press would had been analyzed so a few minutes later the team of advisors could analyze the results, to take actions that could be measured in the next day’s press.
“The main goal at the time was combining transacional databases with dimensional databases in a real time datawarehousing solution, as the analysis were made on a daily basis…”
– DANIEL ALVAREZ
Solution
Understanding the analysis and control dashboard that they needed to review daily, we came up with the idea that we should appeal to the best practices of Business Intelligence to be able to use variables and dimensions in different analysis models that would allow them to get the greatest possible meaning out of the data. But that posed a number of significant challenges, all motivated by the time-sensitive need for the data.
The first challenge was that we had to achieve a simple and eloquent data collection form, which would lead to rapid work by the team that began with the first phase of the project. The second challenge was that once that data was collected, it had to be extracted, transformed, and loaded (ETL) into an analysis system that allowed the data to be analyzed, sliced, and grouped in any way the analyst needed.
The third and most complicated challenge was that all this had to happen in a single process. The problem was that transactional data tables (ideal for inserting data) are totally different from dimensional data tables (ideal for analyzing data). By that moment the most popular solution on the market was that daily ETL processes process all the data of the day to summarize and convert it, but in this case it was necessary that after each news item analyzed, its result were available for analysis in real time. Therefore we used several of the new technologies of the time, (Javascript Frameworks, Adobe Flex, Ruby On Rails, Distributed Systems) to create a working buffer that took the transactional data and converted it into real-time processed information. Achieving reduce the time teams could react to what the media was saying.
Main Goals Achieved
- It was possible to activate work terminals from anywhere in the country and at any time, to improve data capture.
- All team members such as candidates, analysts and advisers, could see the information at all times from their computers.
- By being able to transform the transactional data into dimensional data, it was possible to give the data a more explicit context and discard information that was used only by the minute; to make the databases lighter and better use the cloud resources that were implemented in the architecture. .
- The backend was created with the REST API philosophy and then it was possible to create different views adapted to each profile in different development and work environments; making the entire solution more stable, economical and profitable.
