download all files from google drive shared folder

preserve_download is False, documents downloaded to. download_dir are not removed after processing. preserve_download is True, documents downloaded to. reprocess is True, then documents are always reprocessed. This is made possible by implementing the call to MyIngestDoc.has_output() which is invoked in MainProcess._filter_docs_with_outputs.

output_dir where structured outputs already exists for a given file, the file content is not re-downloaded from the data source nor is it reprocessed. Honors the conventions of BaseConnectorConfig defined in unstructured/ingest/interfaces.py which is passed through the CLI:.The added dependencies should be imported at runtime when the new connector is invoked, rather than as top-level imports.Update the Makefile, adding a target for install-ingest- and adding another pip-compile line to the pip-compile make target.If additional python dependencies are needed for the new connector:.Add a line to test_unstructured_ingest/test-ingest.sh invoking the new test script.Git add the expected outputs under test_unstructured_ingest/expected-structured-output/ so the above test passes in CI.It's json output files should have a total of no more than 100K. Add a script test_unstructured_ingest/test-ingest-.sh.Create a folder under examples/ingest that includes at least one well documented script.Update unstructured/ingest/main.py with support for the new connector.The subclass of BaseIngestDoc overrides process_file() if extra processing logic is needed other than what is provided by auto.partition().

Create a new module under unstructured/ingest/connector/ implementing the 3 abstract base classes, similar to unstructured/ingest/connector/s3_connector.py.Also add test case verifying that 2 files are indeed created, like should be.

Renamed github-access-token, github-branch and github-file-glob to git-access-token, git-branch and git-file-glob, respectively.

Prevent code duplication for functionality between GitHub and GitLab ingest connectors.

Involves more general Git functionality that is shared between the GitHub and GitLab data connectors.

Supports the ability to filter documents through globs.

Supports downloading from a specific git branch.

Files with extensions that are not supported by to are skipped.

The connector can process a single repository and recursively load all documents in the repo.

Download all files from google drive shared folder