How to write code that syncs one table to another

1. Source must have a timestamp column
2. Target must remember the last timestamp processed.
3. Source queries target for the last processed timestamp
4. Source checks whether target
   1. is still importing previous data
   2. or has data that it hasn’t processed yet (import could have been killed halfway)
5. If target is available to import data, source uploads data and triggers processing
6. As data is imported, target saves the last processed timestamp
7. Errors, exceptions are accumulated and reported

It is also handy to have a script that monitors the status of the batch process.

1. is processing
2. last timestamp processed
3. number of rows left