‹ ryanm.xyz - thoughts on data

Exist Integrations Technical Approach

Mar 06, 2021

I have been out of the development game for over a decade. Even when I was doing it as a profession I was not a member of a team. Source control. Proper architecture. Automated testing. These things meant nothing to me.

I am more of a data guy. I still wrote scripts to work with my personal tracking. They were functional at best. Exist Integrations came from one of those scripts. I hacked together a few PHP scripts, uploaded them to a server and configured a cron job. My Trakt history moved to Exist hourly.

The script could not work for other users. I looked at Exist Integrations as a learning experience. I wanted it to have or use a few tenants

  • Use a modern development framework
  • Have a development environment I could work with on my local machine
  • Use source control
  • Use a pipeline to deploy to production

I had no idea what I was doing. I am most familiar with PHP so after research I settled on Laravel. I was not prepared for the learning curve. A Laracasts subscription was a blessing to get my head around the framework.

Things I liked:

  • With Valet I could develop and test on my machine
  • With GitHub and Forge production builds was a cinch. It is worth the money.
  • Eloquent models are awesome
  • Migrations! They made sense and gave me confidence the database is right.
  • Adding packages to speed up development!

Outside of Models and Controllers I structured Exist Integrations as follows:

Objects

Exist Integrations relies on API calls to services like Exist and Trakt. I sought a way to work with the JSON response. I settled on Data Transfer Objects from spatie. EI casts the JSON to Flexible Data Transfer Objects. This allows me to only care of about the fields needed for EI to function.

Services

These are a huge help. I created an Abstract class to handle all HTTP calls (thank Guzzle!). Then each 3rd party received a dedicated Service. With the Dependency Injection with Laravel I could reference these services elsewhere.

Console Commands

EI relies on scheduled jobs. I created jobs that would call the appropriate Service functions.

Date Dimension

If we look at the Trakt integration EI pulls user’s watch history. After EI pulls the history it will send total minutes to Exist. There is a very real scenario where a user didn’t watch TV in a day. By default Exist would show N/A. I always want a value even if it is zero. For this EI uses a date dimension table. It contains one record per day. Joining this to the Watch History ensures a value is always returned for every day.

EI is working great. I’m hard at work planning the next integration!