Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Development to main #199

Merged
merged 50 commits into from
Jan 30, 2025
Merged
Show file tree
Hide file tree
Changes from 44 commits
Commits
Show all changes
50 commits
Select commit Hold shift + click to select a range
8272dfb
First version of the storageService that communicates with the file s…
rjzondervan Jan 15, 2025
aba5f8e
First steps in connecting endpoints with storage
rjzondervan Jan 21, 2025
6a026fd
Add a writeFile function (move to registers) and some docblocks
rjzondervan Jan 21, 2025
9afd48b
Merge remote-tracking branch 'origin/development' into feature/CONNEC…
remko48 Jan 23, 2025
36f3768
fetch logs and contracts after test and run
SudoThijn Jan 23, 2025
40efee0
Added multi file upload
remko48 Jan 23, 2025
a75d3f3
Sync files from source
rjzondervan Jan 23, 2025
f96b7a3
added error reset before upload
SudoThijn Jan 23, 2025
1023eba
Merge pull request #189 from ConductionNL/feature/CONNECTOR-175/multi…
remko48 Jan 23, 2025
c28bcbc
Merge pull request #187 from ConductionNL/feature/IBOC-116/logs-contr…
SudoThijn Jan 23, 2025
41953a3
prettified json
SudoThijn Jan 23, 2025
4365b17
removed endpoint tab
SudoThijn Jan 23, 2025
7a737c7
Merge pull request #190 from ConductionNL/feature/IBOC-112/pretty-jso…
SudoThijn Jan 23, 2025
a82b0c5
Merge pull request #191 from ConductionNL/feature/IBOC-109/remove-end…
SudoThijn Jan 23, 2025
8a4764d
Document fetching and writing files in actions
rjzondervan Jan 23, 2025
d462a72
show list of synchronizations in synchronization tab on source
SudoThijn Jan 24, 2025
ed1be0c
Add fileparts to endpointservice
rjzondervan Jan 28, 2025
1f40bbe
Added rules to synchronizations
remko48 Jan 28, 2025
65fd9cd
Merge pull request #193 from ConductionNL/feature/CONNECTOR-201/add-r…
remko48 Jan 28, 2025
8c789a3
Working fileparts without the need for S3 buckets
rjzondervan Jan 28, 2025
d01e45a
remove var_dump
rjzondervan Jan 28, 2025
4c0c575
Merge remote-tracking branch 'origin/feature/CONNECTOR-190/Fileparts'…
rjzondervan Jan 28, 2025
3e6b3b3
Added timing to rules
remko48 Jan 29, 2025
a99d51f
Merge remote-tracking branch 'origin/feature/CONNECTOR-190/Fileparts'…
remko48 Jan 29, 2025
56cf1fe
Merge branch 'feature/CONNECTOR-203/timing' into feature/CONNECTOR-20…
remko48 Jan 29, 2025
e1b2c85
Added write_file and fetch_file
remko48 Jan 29, 2025
d8e14b0
Merge pull request #194 from ConductionNL/feature/CONNECTOR-203/timing
remko48 Jan 29, 2025
28da706
Merge remote-tracking branch 'origin/feature/CONNECTOR-190/Fileparts'…
remko48 Jan 29, 2025
c488131
Merge pull request #195 from ConductionNL/feature/CONNECTOR-202/sub-f…
remko48 Jan 29, 2025
2bf1832
Cleanup and documentation.
rjzondervan Jan 29, 2025
a0961e7
Docblocks in synchronizationService
rjzondervan Jan 29, 2025
4f5dc8d
Conform to configuration convention used in other rules.
rjzondervan Jan 29, 2025
a85c8ea
Merge remote-tracking branch 'origin/development' into feature/CONNEC…
rjzondervan Jan 29, 2025
5cda745
Updated sourceConfiguration
remko48 Jan 29, 2025
232a74e
Reverded info.xml change and added sourceConfiguration
remko48 Jan 29, 2025
f6dafc3
PR comments
rjzondervan Jan 29, 2025
b4875e5
Corrected checks
rjzondervan Jan 29, 2025
af73fe0
Merge pull request #196 from ConductionNL/feature/CONNECTOR-190/Filep…
rjzondervan Jan 29, 2025
98f34d7
Merge pull request #192 from ConductionNL/feature/IBOC-110/fill-sync-tab
SudoThijn Jan 30, 2025
0ec3476
Added subforms for fileparts_create and filepart_upload
remko48 Jan 30, 2025
3ca9dfb
Add syncing multiple files, writing in the fetch action
rjzondervan Jan 30, 2025
703e712
Code style fixes for pr
WilcoLouwerse Jan 30, 2025
82794b1
Some more style & docblock fixes
WilcoLouwerse Jan 30, 2025
29eb40b
Final try{ style fixes
WilcoLouwerse Jan 30, 2025
592e609
Merge pull request #198 from ConductionNL/feature/CONNECTOR-207/multi…
rjzondervan Jan 30, 2025
d912623
Resolve merge conflict
rjzondervan Jan 30, 2025
42d3975
Small fix
rjzondervan Jan 30, 2025
d996bba
Requested changes
remko48 Jan 30, 2025
e84065c
Merge pull request #197 from ConductionNL/feature/CONNECTOR-206/rule-…
remko48 Jan 30, 2025
d0f73f6
Merge pull request #200 from ConductionNL/fix/main-to-dev
rjzondervan Jan 30, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion appinfo/info.xml
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ The OpenConnector Nextcloud app provides a ESB-framework to work together in an
- 🆓 Map and translate API calls

]]></description>
<version>0.1.29</version>
<version>0.1.31</version>
<licence>agpl</licence>
<category>integration</category>
<author mail="[email protected]" homepage="https://www.conduction.nl/">Conduction</author>
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
{
"@context": {
"schema": "http:\/\/schema.org",
"register": "501"
},
"@type": "mapping",
"@id": "http:\/\/nextcloud.local\/index.php\/apps\/openconnector\/api\/mappings\/1",
"name": "Xxllnc Zaak to Publicatie",
"description": "",
"version": "0.0.3",
"reference": "http:\/\/nextcloud.local\/index.php\/apps\/openconnector\/api\/mappings\/1",
"mapping": {
"title": "{{ values['case.subject'] }}",
"summary": "{{ values['attribute.woo_samenvatting'] }}",
"description": "{{ values['attribute.woo_beschrijving'] }}",
"category": "{{ values['attribute.woo_categorie'] }}",
"featured": false,
"status": "Published",
"attachmentCount": 0,
"published": "{{ values['attribute.woo_publicatiedatum'] }}",
"modified": "{{ \"now\"|date(\"Y-m-d\\\\TH:i:s.v\\\\Z\") }}",
"license": "eupl2",
"catalog": "1bcb99cd-7e47-4242-b7a7-ac7d36fe5d5e",
"attachments": "[{% set fileCount = 0 %}{% for file in values['attribute.woo_publicatie'] %}{% if fileCount > 0 %}, {% endif %}\"\/api\/v1\/case\/{{id}}\/document\/{{file.uuid}}\/download\"{% set fileCount = fileCount + 1 %}{% endfor %}{% for file in values['attribute.woo_inventarisatielijst'] %}{% if fileCount > 0 %}, {% endif %}\"\/api\/v1\/case\/{{id}}\/document\/{{file.uuid}}\/download\"{% set fileCount = fileCount + 1 %}{% endfor %}{% for file in values['attribute.woo_informatieverzoek'] %}{% if fileCount > 0 %}, {% endif %}\"\/api\/v1\/case\/{{id}}\/document\/{{file.uuid}}\/download\"{% set fileCount = fileCount + 1 %}{% endfor %}{% for file in values['attribute.woo_besluit'] %}{% if fileCount > 0 %}, {% endif %}\"\/api\/v1\/case\/{{id}}\/document\/{{file.uuid}}\/download\"{% set fileCount = fileCount + 1 %}{% endfor %}]"
},
"unset": [
""
],
"cast": {
"title": "unsetIfValue==",
"summary": "unsetIfValue==",
"description": "unsetIfValue==",
"category": "unsetIfValue==",
"published": "unsetIfValue==",
"modified": "unsetIfValue==",
"attachments": "jsonToArray"
},
"passThrough": false
}

This file was deleted.

This file was deleted.

Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
{
"@context": {
"schema": "http:\/\/schema.org",
"register": "501"
},
"@type": "rule",
"@id": "http:\/\/nextcloud.local\/index.php\/apps\/openconnector\/api\/rules\/4",
"name": "Fetch Xxllnc files",
"description": null,
"reference": "http:\/\/nextcloud.local\/index.php\/apps\/openconnector\/api\/rules\/4",
"version": "0.0.1",
"action": "post",
"timing": "after",
"conditions": {
"==": [
1,
1
]
},
"type": "fetch_file",
"configuration": {
"fetch_file": {
"source": 3,
"filePath": "attachments",
"method": "GET",
"sourceConfiguration": []
}
},
"order": 0
}
3 changes: 0 additions & 3 deletions css/main.css
Original file line number Diff line number Diff line change
Expand Up @@ -24,9 +24,6 @@
--OC-color-status-emergency: #ffffff;
}




/* Pages */
.pageHeader {
margin-block-start: var(--app-navigation-padding);
Expand Down
40 changes: 40 additions & 0 deletions docs/rules/rules.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,6 +72,46 @@ Upload rules manage file upload functionality and restrictions. Configuration in
- allowedTypes: Comma-separated list of allowed file extensions (e.g., jpg,png,pdf)
- maxSize: Maximum allowed file size in megabytes

#### Partial upload rules

Partial upload rules manage file uploads in multiple parts, allowing for uploads of larger files.
Partial uploads require the created object to contain the file size of the file to be created.

The functionality is split into two rule types:

- `fileparts_create`
- `filepart_upload`

The `fileparts_create` type creates the partial file upload from the endpoint. These partial files will be nested into
the response object, and will be stored with their ids in the saved object. The fileparts will be stored in objects according to a separate schema.

The default schema of these fileparts is:

```json
{
"id": "baed4312-c6b8-48bc-a1b3-c3536d9653be", // The id of the file part object.
"order": 1, // The order number of the file part, starting at 1
"size": 297809, // The size in bytes of the file part that is/will be uploaded
"data": "..." // The data uploaded. This will only exist after uploading, until the data is written into a partial file.
}
```

The `fileparts_create` rule takes the following configuration:

- `sizeLocation` (required): The location in the created object containing the size of the complete file.
- `schemaId` (required): The schema to store the file part in.
- `filenameLocation` (optional): The location in the created object that contains the filename of the file to be created. This defaults to `filename` if it is not set.
- `filePartLocation` (optional): The location in the created object the created fileparts will be written to. Defaults to `fileParts`
- `mappingId` (optional): If the resulting filePart objects have to be mapped to a specific format, the id of the mapping that will map the file parts to the desired format.

The `filepart_upload` type will upload the data in the file part that is uploaded into a temporary file, and
once all fileparts have been uploaded, it will reconcile the partial uploads into one file (deleting the temporary files,
and if no additional data has been put into the folder where the parts are stored, the folder).

The `filepart_upload` rule takes the following configuration:

- `mappingId` (optional): If the file parts are in a specific format, the mapping to map the fileparts to the default format. (Usually this means that this is the inverse mapping of `mappingId` in the corresponding `fileparts_create` rule).

### Locking Rules

Locking rules provide exclusive access control for resources. Configuration includes:
Expand Down
39 changes: 39 additions & 0 deletions docs/synchronization.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
# Synchronization Actions

## Concept

Just as with Endpoints, synchronizations can trigger additional actions (at this moment called rules).
These can change the content of an object, but also trigger additional synchronizations.

These actions can be created by creating a rule (an action) and adding it by id to the property
actions in the Synchronization.

## Synchronizing files
In order to fetch a file from an external source and store it in the Nextcloud Filesystem in a way that OpenRegister can
connect it to an object, there are two predefined actions:

- `fetch_file`: This action downloads the file and substitutes the base encoded content into the variable that contained the file url
- `write_file`: This action takes a file's content in base encoding and the filename (otherwise it will use a default filename), and writes it to the filesystem.

### Fetch file

This action should be run on timing `after` (when the object has been stored).
The action takes the following parameters in the `configuration` property:

- `source` (required): The id of the source where the file can be downloaded
- `filePath` (required): The dot path of the location in the input object that contains the file url or file path.
- `method` (optional): The HTTP method that should be used to fetch the file. Defaults to GET
- `sourceConfiguration` (optional): Additional configuration for the source that only holds for fetching files.

When properly configured this action will download the file from the given source and substitute the base64 encoded content in the returned object.
It is preferred to run this action in combination with `write_file` immediately after, so the file contents are properly stored in the Nextcloud file system instead of written to a database.

### Write file

This action should be run on timing `after`, and if combined with `fetch_file` it should be run in order after `fetch_file`.
The action takes the followin parameters in the `configuration` property:

- `filePath` (required): The dot path of the location in the input object that contains the base64 encoded content of the file.
- `fileNamePath` (required): The dot path of the location in the input object that contains the filename

This will write the file to the nextcloud filesystem in the folder that belongs to the written object, and substitutes the file content in the returned data with the path of the object in the Nextcloud File System.
2 changes: 1 addition & 1 deletion lib/Db/EndpointMapper.php
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,7 @@ public function findAll(?int $limit = null, ?int $offset = null, ?array $filters

private function createEndpointRegex(string $endpoint) {
$regex = '#'.preg_replace(pattern: ['#\/{{([^}}]+)}}\/#', '#\/{{([^}}]+)}}$#'], replacement: ['/([^/]+)/', '(/([^/]+))?'], subject: $endpoint).'#';
if(str_ends_with(haystack: $regex, needle: '?#') === false && str_ends_with(haystack: $regex, needle: '$#') === false) {
if (str_ends_with(haystack: $regex, needle: '?#') === false && str_ends_with(haystack: $regex, needle: '$#') === false) {
$regex = substr($regex, 0, -1). '$#';
}

Expand Down
5 changes: 5 additions & 0 deletions lib/Db/Mapping.php
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,11 @@ public function getJsonFields(): array
);
}

public function getUpdated(): ?DateTime
{
return $this->dateModified;
}

public function hydrate(array $object): self
{
$jsonFields = $this->getJsonFields();
Expand Down
3 changes: 3 additions & 0 deletions lib/Db/Synchronization.php
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,7 @@ class Synchronization extends Entity implements JsonSerializable

protected array $conditions = [];
protected array $followUps = [];
protected array $actions = [];


public function __construct() {
Expand Down Expand Up @@ -69,6 +70,7 @@ public function __construct() {
$this->addType('updated', 'datetime');
$this->addType(fieldName:'conditions', type: 'json');
$this->addType(fieldName:'followUps', type: 'json');
$this->addType(fieldName: 'actions', type: 'json');
}

/**
Expand Down Expand Up @@ -147,6 +149,7 @@ public function jsonSerialize(): array
'updated' => isset($this->updated) === true ? $this->updated->format('c') : null,
'conditions' => $this->conditions,
'followUps' => $this->followUps,
'actions' => $this->actions,
];
}
}
60 changes: 60 additions & 0 deletions lib/Migration/Version1Date20250123100521.php
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
<?php

declare(strict_types=1);

/**
* SPDX-FileCopyrightText: 2025 Nextcloud GmbH and Nextcloud contributors
* SPDX-License-Identifier: AGPL-3.0-or-later
*/

namespace OCA\OpenConnector\Migration;

use Closure;
use OCP\DB\ISchemaWrapper;
use OCP\DB\Types;
use OCP\Migration\IOutput;
use OCP\Migration\SimpleMigrationStep;

/**
* FIXME Auto-generated migration step: Please modify to your needs!
*/
class Version1Date20250123100521 extends SimpleMigrationStep {

/**
* @param IOutput $output
* @param Closure(): ISchemaWrapper $schemaClosure
* @param array $options
*/
public function preSchemaChange(IOutput $output, Closure $schemaClosure, array $options): void {
}

/**
* @param IOutput $output
* @param Closure(): ISchemaWrapper $schemaClosure
* @param array $options
* @return null|ISchemaWrapper
*/
public function changeSchema(IOutput $output, Closure $schemaClosure, array $options): ?ISchemaWrapper {
/**
* @var ISchemaWrapper $schema
*/
$schema = $schemaClosure();

if ($schema->hasTable(tableName: 'openconnector_synchronizations') === true) {
$table = $schema->getTable(tableName: 'openconnector_synchronizations');
if ($table->hasColumn('actions') === false){
$table->addColumn(name: 'actions', typeName: Types::JSON)->setNotnull(false)->setDefault('[]');
}
}

return $schema;
}

/**
* @param IOutput $output
* @param Closure(): ISchemaWrapper $schemaClosure
* @param array $options
*/
public function postSchemaChange(IOutput $output, Closure $schemaClosure, array $options): void {
}
}
Loading