Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Initial front-end restructure and code re-design #381

Merged
merged 16 commits into from
May 28, 2021

Conversation

MaybeJustJames
Copy link
Collaborator

@MaybeJustJames MaybeJustJames commented Sep 14, 2020

This branch begins a large-scale move of the app to redux and redux sagas for state management. It also restructures the app in a way I think is easier to navigate and think about.

Cleanup

  • Webpack config based on this.
  • Remove unnecessary/unused build machinery and unused files.
  • Remove Google Analytics integration
  • Dynamically generate artifact and use default WebPack output directory
  • Update most dependencies, removed unused or unnecessary dependincies
  • Add a script to visualise the size of the compiled build artifact (bundle-size-analyse)
  • Use ts-loader rather than babel to compile ts/tsx (type-checking)
  • Upgrade to webpack 5
  • Remove reliance on polyfilled Node APIs (path and zip)
  • Add linting for use of abstract equality and block delimiters with braces
  • Add tests for the front-end using Jest and react-testing-library. Including tests for fetch() And GProfiler previously introduced.
  • Use jest-css-modules-transform to handle imports for CSS. Version is intentionally pinned as "4.0.2" breaks tests.
  • Wrap the old API in a scary LegacyAPI name
  • Extract some of the complexity in the App component in new Main and FullPageNotify components.
  • Turn on strict TypeScript type checking and fix all type errors.
  • Remove as much CSS as possible and simplify DOM structure
  • Introduce a TEST_ONLY flag to avoid hard-to-test side-effects from the legacy API while testing.
  • Add local CSS imported only by the component that uses it.
  • Remove as many uses of Lodash as possible
  • Add a utility function zipLists to zip() a list of lists.
  • Refactor the UploadModal to be responsive.
  • Reduce use of default exports

Code structure and redesign

  • Document code style and design decisions in CONTRIBUTING.md
  • Change the structure of getFeature() requests and responses
  • Add a "filter by type/category" argument to getFeature() requests
  • Redisign of FeatureSearchBox components following the guidelines outlines in CONTRIBUTING.md
  • LegacyAPI no longer tracks UUID, sessionMode, sidebar visibility, and cookie consent. These have been moved to Redux.
  • Wire in Redux to existing components where useful.
  • Use immer to update Redux state.
  • Add a Result type similar to the backend Result type.

UI

  • Viewer component is now responsive and does less work to render data
  • Redisign of UI, the whole UI is a little more responsive and more traditionally "single-page".
  • Centre the Welcome page and make it scrollable.

Breaking changes and known issues

Copy link
Contributor

@dweemx dweemx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor things that I picked up.
Otherwise nice stuff in there: cleanup & testing 👍

Experienced a problem when trying to run SCope:

> [email protected] dev ~/SCope/master
> cross-env NODE_ENV=development NODE_TYPE=local webpack-dev-server --hot

~/SCope/master/node_modules/terser-webpack-plugin/node_modules/p-limit/index.js:30
		} catch {}
		        ^

SyntaxError: Unexpected token {
    at new Script (vm.js:51:7)
    at createScript (vm.js:136:10)
    at Object.runInThisContext (vm.js:197:10)
    at Module._compile (internal/modules/cjs/loader.js:618:28)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:665:10)
    at Module.load (internal/modules/cjs/loader.js:566:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:506:12)
    at Function.Module._load (internal/modules/cjs/loader.js:498:3)
    at Module.require (internal/modules/cjs/loader.js:598:17)
    at require (internal/modules/cjs/helpers.js:11:18)
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] dev: `cross-env NODE_ENV=development NODE_TYPE=local webpack-dev-server --hot`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the [email protected] dev script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!    ~/.npm/_logs/2020-09-18T12_47_32_043Z-debug.lo

even after removing node_modules and package-lock.json

opt/scopeserver/dataserver/modules/gserver/GServer.py Outdated Show resolved Hide resolved
opt/scopeserver/dataserver/utils/search.py Outdated Show resolved Hide resolved
opt/scopeserver/dataserver/utils/search.py Outdated Show resolved Hide resolved
src/api/features.ts Outdated Show resolved Hide resolved
src/api/features.ts Outdated Show resolved Hide resolved
src/components/Search/FeatureSearchBox.tsx Outdated Show resolved Hide resolved
src/components/Search/FeatureSearchBox.tsx Outdated Show resolved Hide resolved
src/components/Search/model.ts Outdated Show resolved Hide resolved
src/components/common/Histogram.jsx Outdated Show resolved Hide resolved
webpack.config.js Outdated Show resolved Hide resolved
Copy link
Contributor

@KrisDavie KrisDavie left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Really nice changes, but still some work to get it functional and user friendly.

I couldn't exhaustively test the regulon/compare pages or the clusters etc. with the broken bits, will test them once they're functional.

opt/scopeserver/dataserver/utils/search.py Outdated Show resolved Hide resolved
src/components/Search/FeatureSearchBox.tsx Outdated Show resolved Hide resolved
src/components/Search/FeatureSearchBox.tsx Outdated Show resolved Hide resolved
src/components/Search/FeatureSearchBox.tsx Outdated Show resolved Hide resolved
src/api/features.ts Show resolved Hide resolved
@@ -222,28 +227,42 @@ def get_final_feature_and_type(
return features, feature_types


def get_search_results(search_term: str, loom: Loom, data_hash_secret: str) -> Dict[str, List[str]]:
def get_search_results(search_term: str, category: str, loom: Loom, data_hash_secret: str) -> List[CategorisedMatches]:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The lru_cache was removed from GServer. Did you consider putting it back here?

I know it was added a while ago, before the search refactor, so might be worth testing the search times in the current codebase to see if it is still worth it.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No I didn't. Have you got a test load? lru_cache makes me very nervous unless it's super obvious that it's a pure function. This is absolutely not a pure function!

src/components/AppHeader.jsx Outdated Show resolved Hide resolved
src/components/Search/effects.ts Outdated Show resolved Hide resolved
src/components/common/Viewer.jsx Outdated Show resolved Hide resolved
@MaybeJustJames
Copy link
Collaborator Author

Add PR summary points to documentation (conribution manual??)

@dweemx
Copy link
Contributor

dweemx commented Oct 26, 2020

/!\ I noticed that <- Pevious and Next -> buttons of the cluster controls (right side bar) don't work anymore. They work in the master branch

@MaybeJustJames
Copy link
Collaborator Author

/!\ I noticed that <- Pevious and Next -> buttons of the cluster controls (right side bar) don't work anymore. They work in the master branch

Reproductions steps:

  1. Query "Unannotated Cluster 0"
  2. Click on "Next ->" button in the right side bar should take you to the next cluster i.e.: "Unannotated Cluster 1"
  3. Click on "<- Previous" button in the right side bar should take you to the previous cluster i.e.: "Unannotated Cluster 0"

@MaybeJustJames MaybeJustJames changed the base branch from master to develop December 9, 2020 15:31
Copy link
Contributor

@dweemx dweemx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In general, nice cleanup and refactoring! I noticed a few bugs across the app tough:

  • Running the app using Node v10.22.1 fails with a huge stack trace. I was able to run it using v11.15.1. So I suggest we use .nvmrc file to properly state which Node version this app requires.
  • The markers table should not be displayed (actually not exist) when querying for All Clusters in the Gene page.
    image
  • When switching from loom, the viewer is not reloaded with the embedding
  • After having switched from loom, colors are not properly assigned to dots anymore:
    • To reproduce:
      1. Load one loom, search for All Clusters
      2. Load other loom, clear search box and search for All Clusters
  • When switching from embeddings/coordinates, the scatterplot is not updated.
  • All annotations page is empty/blank while loom contains community annotations
  • Some viewers are missing in the compare tab. When switching to the Compare tab only 2 are shown.
  • Compare tab break when performing a DnD of one of the annotation onto a viewer.

index.html Outdated Show resolved Hide resolved
opt/scopeserver/dataserver/modules/gserver/GServer.py Outdated Show resolved Hide resolved
package.json Outdated Show resolved Hide resolved
package.json Outdated Show resolved Hide resolved
src/components/App.jsx Outdated Show resolved Hide resolved
src/components/common/ClusterOverlapsTable.tsx Outdated Show resolved Hide resolved
Comment on lines +27 to +72
<div className='scope-row'>
<FileReaderInput
as='binary'
id='my-file-input'
onChange={this.selectLoomFile.bind(this)}>
<Input
label='File to be uploaded:'
labelPosition='left'
action={{
color: this.state.uploadLoomFile
? 'grey'
: 'orange',
content: 'Select a file...',
}}
fluid
placeholder={this.state.uploadLoomFileName}
error={this.state.uploadError}
/>
</FileReaderInput>
<Button
className='scope-row-element'
color={
this.state.uploadLoomFile
? 'orange'
: 'grey'
}
onClick={this.uploadLoomFile.bind(this)}
disabled={
!this.state.uploadLoomFile ||
this.state.uploadLoomProgress > 0
}>
{' '}
Upload!
</Button>
</div>
<div>
<span>Upload progress:</span>
<span>
<Progress
percent={this.state.uploadLoomProgress}
error={this.state.uploadError}
indicating
progress
disabled></Progress>
</span>
</div>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not crucial but to improve ux/ui, we could remove upload progress and only show the progress when this.state.uploadLoomProgress > 0 and < 100.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not quite sure what you mean. Isn't that the case now?

src/components/pages/Regulon.jsx Outdated Show resolved Hide resolved
src/result.ts Show resolved Hide resolved
webpack.config.js Show resolved Hide resolved
Copy link
Contributor

@KrisDavie KrisDavie left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall nice changes and restructuring. As discussed, the remaining issues can be dealt with in develop in order to get this merged after the main bits above are resolved.

webpack.config.js Show resolved Hide resolved
src/js/http.js Outdated Show resolved Hide resolved
src/components/Main.tsx Outdated Show resolved Hide resolved
src/components/Main.tsx Show resolved Hide resolved
@@ -332,14 +331,10 @@ class AppSidebar extends Component {
return (
<Sidebar
as={Menu}
animation='push'
animation='overlay'
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After playing around a bit, I'm really inclined to side with @dweemx here. It feels very "broken" as an overlay. Mainly because it isn't intuative the the viewers are hidden especially when the sidebar is interacted with so much. Switching between looms or changing coordinates is done very regularly and to have to reopen/close the sidebar for this time is irritating.

src/components/AppHeader.jsx Show resolved Hide resolved
src/components/AppHeader.jsx Outdated Show resolved Hide resolved
src/components/AppHeader.jsx Show resolved Hide resolved
src/components/App.jsx Outdated Show resolved Hide resolved
src/components/App.jsx Show resolved Hide resolved
@MaybeJustJames
Copy link
Collaborator Author

@dweemx wrt an .nvmrc Node 10 is EOL end of this month. I'm fine leaving this out. What about you?

@MaybeJustJames
Copy link
Collaborator Author

@dweemx Can you elaborate on what "switch from loom" means?

@dweemx
Copy link
Contributor

dweemx commented Apr 2, 2021

@dweemx wrt an .nvmrc Node 10 is EOL end of this month. I'm fine leaving this out. What about you?

You mean not having a .nvmrc file or just upgrade to use higher Node version ?

@MaybeJustJames
Copy link
Collaborator Author

MaybeJustJames commented Apr 2, 2021

@dweemx wrt an .nvmrc Node 10 is EOL end of this month. I'm fine leaving this out. What about you?

You mean not having a .nvmrc file or just upgrade to use higher Node version ?

No I mean it's a users fault if they use an EOL node version. So why have an .nvmrc?

@dweemx
Copy link
Contributor

dweemx commented Apr 7, 2021

@dweemx wrt an .nvmrc Node 10 is EOL end of this month. I'm fine leaving this out. What about you?

You mean not having a .nvmrc file or just upgrade to use higher Node version ?

No I mean it's a users fault if they use an EOL node version. So why have an .nvmrc?

Because the user will be sure that the software will work with that version of Node.

Comment on lines +33 to +51
presets: [
[
'@babel/preset-env',
{
targets: {
esmodules: true,
},
},
],
'@babel/preset-react',
],
plugins: [
[
'@babel/plugin-proposal-class-properties',
{ loose: true },
],
['@babel/plugin-proposal-object-rest-spread'],
['@babel/transform-runtime'],
],
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems redundant. Shouldn't these configs be picked up by babel.config.js ?

Comment on lines +15 to +19
if (!has(action.payload.field, draft)) {
(draft as State)[action.payload.field] = init(
action.payload.field
);
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a particular reason for its redundancy ? Why not put this into a function ?

* Move to typescript
* Remove unnecessary and unused code
* Upgrade semantic-ui-react to a more recent version
* Re-organise some dependant code
* Update packages and install immer
This can be used to only search for genes or regulons, etc. that match
the search query rather than everything that matches the search query.

* Add "MENU" text to the header menu button
* Default session mode should be "read"
* Remove dropdown on the FeatureSearchBox. Filter is imposed by the
  page and is not user selectable.
* Adjust getFeatures gRPC/protocol buffers interface
* Modify the gRPC definition of getFeatures to be more convenient
* Fix a react warning
     The warning triggered when trying to update the old state management
     system using onResultSelect() while rendering. This is a no-no
     so now it just calls the old state management update API.
     This also moves remaining logic out of the component
* Do some error handling
* Some documentation
* Add basic Saga side-effect generation testing
* Add Reducer type annotation (thanks @dweemx)
* Use a clearer variable name for search results
* Prefer `type` over `interface`
* Use constants in App
* The name "handleUpdateScatterPlot" is a more accurate than "handleUpdateTSNE"
* Do not open the bundle analyzer by default
* Bump semantic-ui-react version
* use Sagas debounce() rather than throttle()
* Remove unused import of lodash
* Fix bug: Cannot model category as strict type, revert to string
* Fix bug: Correct ordering in displayed search results
* Add some tests
* Wire up redux hooks and fix tests
* Fix "cancel" button in search box
* Track cookie consent in redux
So that users immediately see where to load their data.
Also centres content in the Welcome screen so still
visible with the sidebar open.
* Remove all uses of abstract (in)equality
     Abstract (in)equality is not type-safe and leads to
     weird type coercion.
* eslint eqeqeq check
* Enforce all blocks should have brace delimiters
* Move definition of FEATURE_COLOURS constant
* Remove GeneSet page
* Use ts-loader instead of babel for typescript (ts-loader does type checking)
* Use pako for zlib compression
* Replace some uses of Lodash with Ramda
* Upgrade from Webpack 4 to Webpack 5
* Add another zipLists test
* Looping requests when selecting search result
  "Unnannotated clustrer 0"
* UI disappears when ViewerSidebar is too tall
* Fix non-display of data in gene tab
* Add a contributing file
* Add a new contributing style rule
Concern for discoverability with the sidebar defaulted to hidden means
that it should default to visible. The animation can make styling the
rest of the app difficult. So this commit reduces/removes these
complexities by making the sidebar permanently visible.

* Finally remove the bin/ directory
GServer.py::getNextCluster had not been updated to match the
change in results returned by get_search_results(). This issue is
fixed along with having ViewerSidebar dispatch the Search 'SELECT'
action. This action updates the search box ui.
@MaybeJustJames MaybeJustJames merged commit 8413fa7 into develop May 28, 2021
@dweemx dweemx mentioned this pull request Jun 23, 2021
@dweemx dweemx deleted the cleanup/373-frontend-stage3 branch June 24, 2021 14:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants