Remove State Capability

[pnadolny13]

Currently state is a listed capability but we dont bookmark files. Try to implement state tracking for files in an OS agnostic way.

Or just remove the state capability from the README and MeltanoHub and note that it doesnt manage state.

[aaronsteers]

@pnadolny13 - Good call. I'd say as a short term fix, we probably can go ahead and remove the capability.

In regards to how we might implement state tracking, I think we have two approaches which are very different from each other:

1. We could remember file names (and possibly their hashes) and rather than a greater than or equal comparison we'd just have a have we seen it comparison. This option works best when files are date stamped, size limited, and "table growth" actually looks like new files appended over time.
2. We could treat CSVs as tables and expect a user to provide an incremental key in the catalog.json. We probably need to read the full file anyway, but we'd only emit records if the greater than or equal condition passes. This option works best for files that are going to be constantly edited or appended to.

[pnadolny13]

@aaronsteers I agree. I created #12 to remove the capability for now.

I see value in both options but I think its more common for people to use this tap for testing out pipelines vs running large workloads so reading the whole file isnt that big of a deal. I'd vote for options 2 by default but potentially supporting both options as a configurable switch - idk if thats possible or if it would cause problems though.

[aaronsteers]

@pnadolny13 - Yes, agreed. I think the SDK "rails" have an apply_catalog() method which would apply "replication_key" if provided by a user in the catalog. That said, it's untested as of now and might need some adjustment.

There might even by built-in capability to exclude values based on that key's >= comparison.

If you have cycles (and interest) to try this out, I'd be interested to know if it works out of box - or if it needs more work.

[pnadolny13]

Closing because I split the implementation to a separate issue. #13