Skip to content

Commit

Permalink
Merge pull request #177 from SMI/release/1.7.0
Browse files Browse the repository at this point in the history
Release 1.7.0
  • Loading branch information
rkm authored Mar 30, 2020
2 parents 90198a7 + 41fbbc6 commit cc7c978
Show file tree
Hide file tree
Showing 40 changed files with 670 additions and 327 deletions.
19 changes: 17 additions & 2 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,21 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),

## [Unreleased]

...
-

## [1.7.0] - 2020-03-30

### Added

- Added undo feature to IsIdentifiableReviewer
- Java microservices now log to SMI_LOGS_ROOT

### Changed

- Upgraded HIC.DicomTypeTranslation from `2.1.2` to `2.2.0`
- This includes an upgrade to fo-dicom from `4.0.1` to `4.0.4`
- Upgraded fo-dicom.Drawing from `4.0.1` to `4.0.4`
- Upgraded HIC.RdmpDicom from `2.0.7` to `2.0.8`

## [1.6.0] - 2020-03-17

Expand Down Expand Up @@ -229,7 +243,8 @@ First stable release after importing the repository from the private [SMIPlugin]
- Anonymous `MappingTableName` must now be fully specified to pass validation (e.g. `mydb.mytbl`). Previously skipping database portion was supported.


[Unreleased]: https://github.com/SMI/SmiServices/compare/v1.6.0...develop
[Unreleased]: https://github.com/SMI/SmiServices/compare/v1.7.0...develop
[1.7.0]: https://github.com/SMI/SmiServices/compare/v1.6.0...v1.7.0
[1.6.0]: https://github.com/SMI/SmiServices/compare/v1.5.2...v1.6.0
[1.5.2]: https://github.com/SMI/SmiServices/compare/v1.5.1...v1.5.2
[1.5.1]: https://github.com/SMI/SmiServices/compare/v1.5.0...v1.5.1
Expand Down
10 changes: 4 additions & 6 deletions PACKAGES.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,13 +11,13 @@
| ------- | ------------| --------| ------- | ------- | -------------------------- |
| CommandLineParser | [GitHub](https://github.com/commandlineparser/commandline) | [2.5.0](https://www.nuget.org/packages/CommandLineParser/2.5.0) | [MIT](https://opensource.org/licenses/MIT)| Command line argument parsing | |
| CsvHelper | [GitHub](https://github.com/JoshClose/CsvHelper) | [12.1.2](https://www.nuget.org/packages/CsvHelper/12.1.2) | [MS-PL and Apache 2.0](https://github.com/JoshClose/CsvHelper/blob/master/LICENSE.txt)| Writting reports out to CSV reports | |
| HIC.DicomTypeTranslation | [GitHub](https://github.com/HicServices/DicomTypeTranslation) | [2.1.2](https://www.nuget.org/packages/HIC.DicomTypeTranslation/2.1.2) | [GPL 3.0](https://www.gnu.org/licenses/gpl-3.0.html) | Translate dicom types into C# / database types | |
| HIC.RDMP.Dicom | [GitHub](https://github.com/HicServices/RdmpDicom) | [2.0.7](https://www.nuget.org/packages/HIC.RDMP.Dicom/2.0.7) | [GPL 3.0](https://www.gnu.org/licenses/gpl-3.0.html) | RDMP Plugin containing data load / pipeline components for imaging, reading dicom files etc | |
| HIC.DicomTypeTranslation | [GitHub](https://github.com/HicServices/DicomTypeTranslation) | [2.2.0](https://www.nuget.org/packages/HIC.DicomTypeTranslation/2.2.0) | [GPL 3.0](https://www.gnu.org/licenses/gpl-3.0.html) | Translate dicom types into C# / database types | |
| HIC.RDMP.Dicom | [GitHub](https://github.com/HicServices/RdmpDicom) | [2.0.8](https://www.nuget.org/packages/HIC.RDMP.Dicom/2.0.8) | [GPL 3.0](https://www.gnu.org/licenses/gpl-3.0.html) | RDMP Plugin containing data load / pipeline components for imaging, reading dicom files etc | |
| HIC.RDMP.Plugin | [GitHub](https://github.com/HicServices/RDMP) | [4.0.2](https://www.nuget.org/packages/HIC.RDMP.Plugin/4.0.2) | [GPL 3.0](https://www.gnu.org/licenses/gpl-3.0.html) | Interact with RDMP objects, base classes for plugin components etc | |
| JetBrains.Annotations | | [2019.1.3](https://www.nuget.org/packages/JetBrains.Annotations/2019.1.3) |[MIT](https://opensource.org/licenses/MIT) | Static analysis tool | |
| Magick.NET-Q16-AnyCPU | [GitHub](https://github.com/dlemstra/Magick.NET) | [7.15.1](https://www.nuget.org/packages/Magick.NET-Q16-AnyCPU/7.15.1) | [Apache License v2](https://github.com/dlemstra/Magick.NET/blob/master/License.txt) | The .NET library for [ImageMagick](https://imagemagick.org/index.php) | |
| Microsoft.CodeAnalysis.CSharp.Scripting | [GitHub](https://github.com/dotnet/roslyn) | [3.5.0-beta2-final](https://www.nuget.org/packages/Microsoft.CodeAnalysis.CSharp.Scripting/3.5.0-beta2-final) | [MIT](https://opensource.org/licenses/MIT) | Supports dynamic rules for cohort extraction logic | |
| Microsoft.Extensions.Caching.Memory | [GitHub](https://github.com/dotnet/extensions) | [3.1.2](https://www.nuget.org/packages/Microsoft.Extensions.Caching.Memory/3.1.2) | [Apache 2.0](https://www.nuget.org/packages/Microsoft.Extensions.Caching.Memory/3.1.2/License) | Caching ID mappings retrieved from Redis/MySQL |
| Microsoft.Extensions.Caching.Memory | [GitHub](https://github.com/dotnet/extensions) | [3.1.3](https://www.nuget.org/packages/Microsoft.Extensions.Caching.Memory/3.1.3) | [Apache 2.0](https://www.nuget.org/packages/Microsoft.Extensions.Caching.Memory/3.1.3/License) | Caching ID mappings retrieved from Redis/MySQL |
| MongoDB.Driver | [GitHub](https://github.com/mongodb/mongo-csharp-driver) |[2.9.3](https://www.nuget.org/packages/MongoDB.Driver/2.9.3)| [Apache 2.0](https://www.nuget.org/packages/MongoDB.Driver/2.8.1/License) | For writting/reading dicom tags into MongoDb databases|
| NLog | [GitHub](https://github.com/NLog/NLog) | [4.6.4](https://www.nuget.org/packages/NLog/4.6.4) | [BSD 3-Clause](https://github.com/NLog/NLog/blob/dev/LICENSE.txt) | Flexible user configurable logging | |
| Newtonsoft.Json | [GitHub](https://github.com/JamesNK/Newtonsoft.Json) | [12.0.3](https://www.nuget.org/packages/Newtonsoft.Json/12.0.3) | [MIT](https://opensource.org/licenses/MIT) | Serialization of objects for sharing/transmission |
Expand All @@ -30,6 +30,4 @@
| System.Security.AccessControl | [GitHub](https://github.com/dotnet/corefx) | [4.7.0](https://www.nuget.org/packages/System.Security.AccessControl/4.7.0) |[MIT](https://opensource.org/licenses/MIT) | File access perimssions| |
| Tesseract | [GitHub](https://github.com/charlesw/tesseract/) | [4.1.0-beta1](https://www.nuget.org/packages/Tesseract/4.1.0-beta1) |[Apache License v2](https://github.com/charlesw/tesseract/blob/master/LICENSE.txt) | Optical Character Recognition in Dicom Pixel data| |
| YamlDotNet | [GitHub](https://github.com/aaubry/YamlDotNet) | [6.0.0](https://www.nuget.org/packages/YamlDotNet/6.0.0) | [MIT](https://opensource.org/licenses/MIT) |Loading configuration files|
| fo-dicom | [GitHub](https://github.com/fo-dicom/fo-dicom) |[4.0.1](https://www.nuget.org/packages/fo-dicom/4.0.1) | [MS-PL](https://opensource.org/licenses/MS-PL) | Handles reading/writing dicom tags from dicom datasets | |
| fo-dicom.Drawing | [GitHub](https://github.com/fo-dicom/fo-dicom) | [4.0.1](https://www.nuget.org/packages/fo-Dicom.Drawing/4.0.1) | [MS-PL](https://opensource.org/licenses/MS-PL)| Support library for reading DICOM pixel data | |
| fo-dicom.Json | [GitHub](https://github.com/fo-dicom/fo-dicom) | [4.0.1](https://www.nuget.org/packages/fo-dicom.Json/4.0.1) | [MS-PL](https://opensource.org/licenses/MS-PL)| Support library for serializing fo-dicom DICOM datasets to json | |
| fo-dicom.Drawing | [GitHub](https://github.com/fo-dicom/fo-dicom) | [4.0.4](https://www.nuget.org/packages/fo-Dicom.Drawing/4.0.4) | [MS-PL](https://opensource.org/licenses/MS-PL)| Support library for reading DICOM pixel data | |
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
![GitHub](https://img.shields.io/github/license/SMI/SmiServices)
[![Total alerts](https://img.shields.io/lgtm/alerts/g/SMI/SmiServices.svg?logo=lgtm&logoWidth=18)](https://lgtm.com/projects/g/SMI/SmiServices/alerts/)

Version: `1.6.0`
Version: `1.7.0`

# SMI Services

Expand Down
6 changes: 3 additions & 3 deletions src/SharedAssemblyInfo.cs
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,6 @@
[assembly: AssemblyCulture("")]

// These should be overwritten by release builds
[assembly: AssemblyVersion("1.6.0")]
[assembly: AssemblyFileVersion("1.6.0")]
[assembly: AssemblyInformationalVersion("1.6.0")] // This one can have the extra build info after it
[assembly: AssemblyVersion("1.7.0")]
[assembly: AssemblyFileVersion("1.7.0")]
[assembly: AssemblyInformationalVersion("1.7.0")] // This one can have the extra build info after it
82 changes: 68 additions & 14 deletions src/applications/IsIdentifiableReviewer/MainWindow.cs
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,8 @@ class MainWindow : View,IRulePatternFactory
private Label _updateRuleLabel;
private CheckBox _cbRulesOnly;

Stack<MainWindowHistory> History = new Stack<MainWindowHistory>();

public MainWindow(List<Target> targets, IsIdentifiableReviewerOptions opts, IgnoreRuleGenerator ignorer, RowUpdater updater)
{
_targets = targets;
Expand Down Expand Up @@ -116,6 +118,13 @@ public MainWindow(List<Target> targets, IsIdentifiableReviewerOptions opts, Igno
Clicked = ()=>GoToRelative(1)
});

frame.Add(new Button("unDo")
{
X = 11,
Y = 2,
Clicked = ()=>Undo()
});

frame.Add(new Label(0,4,"Default Patterns"));

_ignoreRuleLabel = new Label(0,5,"Ignore:");
Expand Down Expand Up @@ -146,6 +155,23 @@ public MainWindow(List<Target> targets, IsIdentifiableReviewerOptions opts, Igno
OpenReport(opts.FailuresCsv,(e)=>throw e, (t)=>throw new Exception("Mode only supported when a single Target is configured"));
}

private void Undo()
{
if (History.Count == 0)
{
ShowMessage("History Empty","Cannot undo, history is empty");
return;
}

var popped = History.Pop();

//undo file history
popped.OutputBase.Undo();

//wind back UI
GoTo(popped.Index);
}

private void GoToRelative(int offset)
{
if(CurrentReport == null)
Expand Down Expand Up @@ -216,7 +242,7 @@ private void Next()
var next = CurrentReport.Current;

//prefer rules that say we should update the database with redacted over rules that say we should ignore the problem
if (!Updater.OnLoad(CurrentTarget?.Discover(),next))
if (!Updater.OnLoad(CurrentTarget?.Discover(),next, out _))
updated++;
else if (!Ignorer.OnLoad(next,out _))
skipped++;
Expand Down Expand Up @@ -253,7 +279,16 @@ private void Ignore()
if(_valuePane.CurrentFailure == null)
return;

Ignorer.Add(_valuePane.CurrentFailure);
try
{
Ignorer.Add(_valuePane.CurrentFailure);
History.Push(new MainWindowHistory(CurrentReport.CurrentIndex,Ignorer));
}
catch (OperationCanceledException)
{
//if user cancels adding the ignore then stay on the same record
return;
}
Next();
}
private void Update()
Expand All @@ -264,7 +299,14 @@ private void Update()
try
{
Updater.Update(_cbRulesOnly.Checked ? null : CurrentTarget?.Discover()
,_valuePane.CurrentFailure,null /*create one yourself*/);
, _valuePane.CurrentFailure, null /*create one yourself*/);

History.Push(new MainWindowHistory(CurrentReport.CurrentIndex,Updater));
}
catch (OperationCanceledException)
{
//if user cancels updating then stay on the same record
return;
}
catch (Exception e)
{
Expand Down Expand Up @@ -411,7 +453,8 @@ bool RunDialog<T>(string title, string message,out T chosen, params T[] options)
chosen = result;
return optionChosen;
}
private bool GetText(string title, string message, string initialValue, out string chosen)
private bool GetText(string title, string message, string initialValue, out string chosen,
Dictionary<string, string> buttons)
{
bool optionChosen = false;

Expand Down Expand Up @@ -462,13 +505,17 @@ private bool GetText(string title, string message, string initialValue, out stri
};
dlg.Add(btn);

var btnClear = new Button(15, line, "Clear")
{
Clicked = () => { txt.Text = ""; }
};
dlg.Add(btnClear);


int x = 10;
if(buttons != null)
foreach (var kvp in buttons)
{
var button = new Button(x, line,kvp.Key)
{
Clicked = () => { txt.Text = kvp.Value; }
};
dlg.Add(button);
x += kvp.Key.Length + 5;
}

dlg.FocusFirst();

Expand All @@ -490,8 +537,15 @@ public string GetPattern(object sender,Failure failure)
var defaultFactory = sender == Updater ? _origUpdaterRulesFactory : _origIgnorerRulesFactory;

var recommendedPattern = defaultFactory.GetPattern(sender,failure);

if (GetText("Pattern", "Enter pattern to match failure", recommendedPattern, out string chosen))

Dictionary<string,string> buttons = new Dictionary<string, string>();
buttons.Add("Clear","");
buttons.Add("Full",_origIgnorerRulesFactory.GetPattern(sender,failure));
buttons.Add("Captures",_origUpdaterRulesFactory.GetPattern(sender,failure));

buttons.Add("Symbols",new SymbolsRulesFactory().GetPattern(sender,failure));

if (GetText("Pattern", "Enter pattern to match failure", recommendedPattern, out string chosen,buttons))
{
Regex regex;

Expand Down Expand Up @@ -521,7 +575,7 @@ public string GetPattern(object sender,Failure failure)
}


throw new Exception("User chose not to enter a pattern");
throw new OperationCanceledException("User chose not to enter a pattern");
}

}
Expand Down
16 changes: 16 additions & 0 deletions src/applications/IsIdentifiableReviewer/MainWindowHistory.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
using IsIdentifiableReviewer.Out;

namespace IsIdentifiableReviewer
{
internal class MainWindowHistory
{
public int Index { get;}
public OutBase OutputBase { get; }

public MainWindowHistory(int index, OutBase outputBase)
{
Index = index;
OutputBase = outputBase;
}
}
}
40 changes: 35 additions & 5 deletions src/applications/IsIdentifiableReviewer/Out/OutBase.cs
Original file line number Diff line number Diff line change
Expand Up @@ -13,11 +13,13 @@ namespace IsIdentifiableReviewer.Out
{
public abstract class OutBase
{
protected List<IsIdentifiableRule> Rules { get;}
public List<IsIdentifiableRule> Rules { get;}
public FileInfo RulesFile { get; }

public IRulePatternFactory RulesFactory { get; set; } = new MatchWholeStringRulePatternFactory();

public Stack<OutBaseHistory> History = new Stack<OutBaseHistory>();

protected OutBase(FileInfo rulesFile)
{
RulesFile = rulesFile;
Expand Down Expand Up @@ -72,13 +74,41 @@ protected IsIdentifiableRule Add(Failure f, RuleAction action)

var serializer = new Serializer();
var yaml = serializer.Serialize(new List<IsIdentifiableRule> {rule});
File.AppendAllText(RulesFile.FullName,
$"#{Environment.UserName} - {DateTime.Now}" + Environment.NewLine +
yaml);

var contents = $"#{Environment.UserName} - {DateTime.Now}" + Environment.NewLine +
yaml;

File.AppendAllText(RulesFile.FullName,contents);
History.Push(new OutBaseHistory(rule,contents));

return rule;
}


public void Undo()
{
if(History.Count == 0)
return;

var popped = History.Pop();

if (popped != null)
{
//clear the rule from the serialized text file
var oldText = File.ReadAllText(RulesFile.FullName);
var newText = oldText.Replace(popped.Yaml, "");

//write to a new temp file
File.WriteAllText(RulesFile.FullName + ".tmp",newText);

//then hot swap them
File.Delete(RulesFile.FullName);
File.Move(RulesFile.FullName + ".tmp",RulesFile.FullName);

//clear the rule from memory
Rules.Remove(popped.Rule);
}
}

/// <summary>
/// Returns true if there are any rules that already exactly cover the given <paramref name="failure"/>
/// </summary>
Expand Down
16 changes: 16 additions & 0 deletions src/applications/IsIdentifiableReviewer/Out/OutBaseHistory.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
using Microservices.IsIdentifiable.Rules;

namespace IsIdentifiableReviewer.Out
{
public class OutBaseHistory
{
public IsIdentifiableRule Rule { get; }
public string Yaml { get; }

public OutBaseHistory(IsIdentifiableRule rule, string yaml)
{
Rule = rule;
Yaml = yaml;
}
}
}
7 changes: 5 additions & 2 deletions src/applications/IsIdentifiableReviewer/Out/RowUpdater.cs
Original file line number Diff line number Diff line change
Expand Up @@ -95,15 +95,18 @@ public void Update(DiscoveredServer server, Failure failure, IsIdentifiableRule
/// </summary>
/// <param name="server"></param>
/// <param name="failure"></param>
/// <param name="rule">The first rule that covered the <paramref name="failure"/></param>
/// <returns>True if <paramref name="failure"/> is novel and not seen before</returns>
public bool OnLoad(DiscoveredServer server,Failure failure)
public bool OnLoad(DiscoveredServer server,Failure failure, out IsIdentifiableRule rule)
{
rule = null;

//we have bigger problems than if this is novel!
if (server == null)
return true;

//if we have seen this before
if (IsCoveredByExistingRule(failure, out IsIdentifiableRule rule))
if (IsCoveredByExistingRule(failure, out rule))
{
//since user has issued an update for this exact problem before we can update this one too
Update(server,failure,rule);
Expand Down
31 changes: 31 additions & 0 deletions src/applications/IsIdentifiableReviewer/Out/SymbolsRulesFactory.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
using System.Text;
using System.Text.RegularExpressions;
using Microservices.IsIdentifiable.Reporting;

namespace IsIdentifiableReviewer.Out
{
public class SymbolsRulesFactory : IRulePatternFactory
{
public string GetPattern(object sender, Failure failure)
{
StringBuilder sb = new StringBuilder();

for (int i = 0; i < failure.ProblemValue.Length; i++)
{
char cur = failure.ProblemValue[i];

if (char.IsDigit(cur))
sb.Append("\\d");
else
if (char.IsLetter(cur))
sb.Append(char.IsUpper(cur) ? "[A-Z]" : "[a-z]");
else
{
sb.Append(Regex.Escape(cur.ToString()));
}
}

return "^" + sb + "$";
}
}
}
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
using System.Collections.Generic;
using System;
using System.Collections.Generic;
using FAnsi.Discovery;
using FAnsi.Discovery.QuerySyntax;
using Microservices.IsIdentifiable.Reporting;
Expand All @@ -23,6 +24,9 @@ public abstract class UpdateStrategy : IUpdateStrategy
protected string GetUpdateWordSql(DiscoveredTable table,
Dictionary<DiscoveredTable, DiscoveredColumn> primaryKeys, IQuerySyntaxHelper syntax, Failure failure,string word)
{
if(string.IsNullOrEmpty(failure.ResourcePrimaryKey))
throw new ArgumentException("Failure record's primary key is blank, cannot update database");

return $@"update {table.GetFullyQualifiedName()}
SET {syntax.EnsureWrapped(failure.ProblemField)} =
REPLACE({syntax.EnsureWrapped(failure.ProblemField)},'{syntax.Escape(word)}', 'SMI_REDACTED')
Expand Down
Loading

0 comments on commit cc7c978

Please sign in to comment.