README.md

External FTP / SFTP poller

This bootstrap module supports polling remote FTP / SFTP servers for files. When the poller finds files, it copies them to a Stedi bucket. Then, the operation can optionally delete those files from the remote server.

• Poller overview
• Add a configuration entry
• Invoke the poller function
• Configure boostrap to process files automatically

Poller overview

The ftp-external-poller function performs the following steps:

1. Calls Stash to retrieve the configuration for the poller.

2. Looks for the configuration associated with the key that was provided on invocation.

3. Connects to the remote server using the corresponding connection configuration.

4. Looks for files to process on the remote server.

5. For each file to be processed, copies the file to the bucket and path specified in the configuration.

6. Optionally deletes the file from the remote server.

7. After processing all files to be processed, closes the connection.

The poller configuration is stored in Stash with the trading partners configuration in the partners-config keyspace. The poller configuration is stored in the bootstrap|remote-poller-config key, and the value is a map of keys to ftp/sftp configuration entries.

The following code shows the remote poller configuration schema.

type RemotePollerConfig = {
  [key: string]: {
    connectionDetails: {
      protocol: "ftp" | "sftp";
      config: FtpConifig | SftpConfig;
    };
    deleteAfterProcessing: boolean; // default: false
    destination: DestinationBucket;
    remotePath?: string;  // default: "/"
    remoteFiles?: string[];
  };
};

type FtpConfig = {
  host: string;
  port?: number // default: 21
  user: string;
  password: string;
  secure?: boolean | "implicit";
  secureOptions?: {
    rejectUnauthroized: boolean;
  };
};

type SftpConfig = {
  host: string;
  port?: number // default: 22
  username: string;
  password: string;
};

type DestinationBucket = {
  type: "bucket";
  bucketName: string;
  path: string;
};

Add a configuration entry

You must create configuration entry in Stash before you can invoke the poller function.

1. Navigate to the partners-configuration keyspace in the Stash UI.
2. If this is your first remote poller configuration entry, add a new Stash key named bootstrap|remote-poller-config. If you already have a configuration entry, edit the Stash key value to add a new entry to the existing map.

The following example configuration includes two remote poller configuration entries: one for FTP named my-remote-ftp, and one for SFTP named my-remote-sftp. Use this example configuration as a template for your Stash value, updating the configuration parameters to correspond to your remote server.

{
  "my-remote-ftp": {
    "connectionDetails": {
      "config": {
        "host": "ftp.trading-partner-1.com",
        "port": 21,
        "user": "my-ftp-user",
        "password": "my-ftp-password"
      },
      "protocol": "ftp"
    },
    "deleteAfterProcessing": false,
    "destination": {
      "bucketName": "my-sftp-bucket",
      "path": "/trading_partners/my-trading-partner-1/inbound",
      "type": "bucket"
    },
    "remotePath": "/"
  },
  "my-remote-sftp": {
    "connectionDetails": {
      "config": {
        "host": "sftp.trading-partner-2.com",
        "port": 22,
        "username": "my-sftp-user",
        "password": "my-sftp-password"
      },
      "protocol": "sftp"
    },
    "deleteAfterProcessing": false,
    "destination": {
      "bucketName": "my-sftp-bucket",
      "path": "/trading_partners/my-trading-partner-2/inbound",
      "type": "bucket"
    },
    "remotePath": "/outbound"
  }
}

Invoke the poller function

Invoke manually

Once deployed, you can invoke the function with the command line to verify functionality. The following command invokes the ftp-external-poller Stedi function and polls the remote server for files.

npm run execute ftp-external-poller <CONFIGURATION_KEY>

The script output includes a summary of the polling operations.

> [email protected] execute
> ts-node-esm ./src/scripts/execute.ts ftp-external-poller rob-test

Invoking function 'ftp-external-poller' synchronously.
Result:
  {
    "processedFiles": [
      {
        "path": "/outbound",
        "name": "855-1746.edi",
        "lastModifiedTime": 1677874415000
      }
    ],
    "skippedItems": [],
    "processingErrors": []
  }

Invoke automatically

You can invoke the poller function automatically with one of two options.

Orchestration tool

Set up a workflow using your orchestration tool of choice, and use one of the following to invoke the function:

• Functions SDK
• Functions API

GitHub Action

You can use the scheduler GitHub action in this repository to invoke the function automatically. Complete the following steps:

1. Fork this repository.

2. Create a new repository secret in your forked repository named STEDI_API_KEY and save the value of your API key as the secret value. This secret is referenced within the workflow and is passed as an environment variable to the script that invokes the ftp-external-poller Stedi function in your account. Note: Make sure there is no leading or trailing whitespace in the secret value as this will cause authentication to fail.

3. Enable the workflow to run in your forked repository. GitHub requires you to explicitly enable workflows that are copied from a forked repository. In your forked repository, click the Actions tab, and then click the button to enable workflow runs.

To change the schedule for invoking the poller:

1. Modify the cron attribute of the schedule in accordance with the GitHub documentation for workflow schedules.
2. Commit the changes and push them to your forked repository.

Configure bootstrap to process files automatically

Do the following to configure the bootstrap module to process new files automatically:

• Specify an inbound directory as the destination for new files
• Enable bucket notifications in the destination bucket to invoke the edi-inbound function