Data Engineering

arrowsOur leading core value at FMTC is to “Be Indispensable“. In order to deliver on that, we place a high priority on the quality of our data. When you use FMTC, your links work, offers are valid and spelling errors are corrected. We want our data to be an indispensable part of the high-quality experience you’re creating for your visitors, readers and members.

We’re pulling data from dozens of sources including affiliate network APIs, private programs, email messages, spreadsheets and so on. Each deal is subjected to a full round of testing, verification and categorization using automated processes and good, old fashioned manual labor.

Automated Data Engineering

Our technology consists of over 80 different custom software programs that:

  • Collect data from dozens of sources
  • Detect dates, amounts, deal types and merchant details and begin sorting the data
  • Identify issues with formats, links and spelling and fix many of them (real people do the rest!)
  • Categorize content
  • Throw out expired deals
  • Format the links to make room for your Affiliate ID
  • Match up offers with Skimlinks URLs

The data is then passed on to our Data Engineering staff…

Manual Data Engineering

Once the machines are done with the data, it’s passed on to our team of stateside Data Engineers. These dedicated team members use our proprietary system to:

  • Read the deals and ensure they make sense
  • Click the links to see that they land in the right spot
  • Test the coupon codes and see if they actually work
  • Fix the spelling errors
  • Remove annoying formatting like LABELS IN ALL CAPS
  • Tag the deals with types and categories so that you can easily find and display them
  • Contact affiliate managers and communicate back and forth to find missing information, fix broken links or make sense of confusing offer details

This work takes a lot of time. Our Data Engineering team logs over 1,000 hours of work each month testing, cleaning and verifying deals.