I'm working on a website that has a section that shows the company's stock details and investor-related data. This data is retrieved through scheduled SOAP API calls.
I created two scheduled tasks for achieving that:
- feth-data (every 10 minutes): retrieves data and stores it in
cron_last_trade_price_graph
which a graph which represents the 'last trade price' in relation to time throughout the day - end-day (at the end of each day): cleans outdated (last two days) data records of
cron_last_trade_price_graph
and retrieves data to restore it incron_historical_data
that data will be used to populate Historical Data table and graph that show the 'Close Price' for the stock throughout time
Now, this is the first time I had to use schedules tasks, and since I'm working with important data I need to make sure that my implementation is solid and avoid any problems.
So generally my question is: What are the considerations (the dos and don'ts) when working with an application that depends on scheduled tasks?
For example:
- If the server fails to perform a scheduled task (say
end-day
). What is the best approach to re-try to perform that task?
Possible Solution: Making a placeholder row for the next day, whose data needs to be retrieved and stored, that has a columnis_historical_data_row_successfully_added
that has a default value of 'NO' and then create another scheduled task (fix-historical-data
) to check whether that placeholder row has been successfully populated and populate it if not.
Concern: What if thefix-historical-data
also failed? Then would it be best to schedulefix-historical-data
for e.g. 3 times throughout the day? I feel like that's a pretty dirty solution. - Should the application be bootstrapped for every scheduled task?: the application uses Wordpress as its CMS. Should I bootstrap Wordpress for
fetch-data
while it may barely need the functionality of Wordpress? Should I bootstrap Wordpress when the task depends heavily on it and not bootstrap Wordpress when the task doesn't? What's the best practice here? - Code Structure: Should the scheduled tasks have an independent directory? Or should its file and dependencies be mingled with other files of the application?
Those are the concerns that I can think of at the moment. Are there any other considerations that I need to be aware of?