MIMBCD-UI Multi Modality Prototype

⚠️ This repository information is not yet curated. For similar use, please follow the prototype-multi-modality-assistant repository on the MIDA organization.

In this repository, we present one of the used prototypes for our User Tests and Analysis 4 (UTA4) study. Work and results are published on a top Human-Computer Interaction (HCI) conference named AVI 2020 (page). Results were analyzed and interpreted from Statistical Analysis charts. The user tests were made in clinical institutions, where clinicians diagnose several patients for a Single-Modality vs Multi-Modality comparison. For example, in these tests, we used both prototype-single-modality and prototype-multi-modality repositories for the comparison. The MIMBCD-UI is a research project that deals with the use of a recently proposed technique in literature: Deep Convolutional Neural Networks (CNNs). These deep networks will incorporate information from several different modes by a User Interface (UI) implemented based on our Prototype Breast Screening repository. The hereby repository is a mirror of our Prototype Breast Screening repository which is an Open Source Project with the goal to deliver an example of web based medical imaging platform for the breast cancer diagnosis. We also have several demos to see in our YouTube Channel, please follow us.

Citing

We kindly ask scientific works and studies that make use of the repository to cite it in their associated publications. Similarly, we ask open-source and closed-source works that make use of the repository to warn us about this use.

You can cite our work using the following BibTeX entry:

@inproceedings{10.1145/3399715.3399744,
author = {Calisto, Francisco Maria and Nunes, Nuno and Nascimento, Jacinto C.},
title = {BreastScreening: On the Use of Multi-Modality in Medical Imaging Diagnosis},
year = {2020},
isbn = {9781450375351},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
url = {https://doi.org/10.1145/3399715.3399744},
doi = {10.1145/3399715.3399744},
abstract = {This paper describes the field research, design and comparative deployment of a multimodal medical imaging user interface for breast screening. The main contributions described here are threefold: 1) The design of an advanced visual interface for multimodal diagnosis of breast cancer (BreastScreening); 2) Insights from the field comparison of Single-Modality vs Multi-Modality screening of breast cancer diagnosis with 31 clinicians and 566 images; and 3) The visualization of the two main types of breast lesions in the following image modalities: (i) MammoGraphy (MG) in both Craniocaudal (CC) and Mediolateral oblique (MLO) views; (ii) UltraSound (US); and (iii) Magnetic Resonance Imaging (MRI). We summarize our work with recommendations from the radiologists for guiding the future design of medical imaging interfaces.},
booktitle = {Proceedings of the International Conference on Advanced Visual Interfaces},
articleno = {49},
numpages = {5},
keywords = {user-centered design, multimodality, medical imaging, human-computer interaction, healthcare systems, breast cancer, annotations},
location = {Salerno, Italy},
series = {AVI '20}
}

Instructions

First of all, you will need NodeJS installed locally on your machine. This project needs both npm and http-server dependencies to install and run the core project. If you do not have those installed please follow the INSTALL instructions.

DICOM Server

The following assumes you will be using a git version control for this repository, storing thanks to GitHub. First, Download and Install git. Our system needs to be integrated with WADO-URI servers, DICOMWeb servers or any HTTP based server that returns a DICOM P10 instances. We suggest you to use an Orthanc server, since it is a simple and powerful standalone DICOM server by providing a RESTful API.

You can download a latest version or you can use our own sample of an Orthanc version with our examples of patient images. The instructions to use our solution are as follows.

Clone

1.1.1. Clone the DICOM Server repository:

git clone git@github.com:MIMBCD-UI/dicom-server.git

1.1.2. Go inside the project folder:

cd dicom-server/

Install

1.2.1. Install the local dependencies:

npm install

1.2.2. You can now Run the project, just follow the next section.

Run

1.3.1. Inside the project folder:

cd dicom-server

1.3.2. Start the DICOM Server (Orthanc) for MacOS:

npm run start:multi

cd ..
cd dicom-server/orthancAndPluginsOSX.stable/
./startOrthanc.command

NOTE: If you are not using MacOS, for instance, if you are using Windows or Debian you have a documentation for that. Just follow the Windows or Debian documentations. You also have several other options.

1.3.3. Open the link:

localhost:8248

NOTE: If you need some help see the Demo.

Main Server

Our main server uses NodeJS and has several dependencies. For the following steps you must have already installed both NodeJS and npm in your machine.

Clone

2.1.1. Clone the project repository:

git clone git@github.com:MIMBCD-UI/prototype-multi-modality.git

2.1.2. Go inside the project folder:

cd prototype-multi-modality/

Install

2.2.1. Install the local dependencies:

npm install

2.2.2. You can now Run the project, just follow the next section.

Run

2.3.1. Inside the project folder:

cd prototype-multi-modality/

2.3.2. If you have already run the DICOM Server on a previous section, please jump to the 2.3.3. point, otherwise do:

npm run dicom-server

2.3.3. Run the code:

npm run build:multi

2.3.4. Start the project:

npm run start:multi

2.3.5. Open the link:

localhost:8286/src/public/index.html

Allow-Control-Allow-Origin

Access-Control-Allow-Origin is a CORS (Cross-Origin Resource Sharing) header. If you want to know How does Access-Control-Allow-Origin header work? follow the link.

Google Chrome

To deal with the CORS issue it is necessary to open Google Chrome with --disable-web-security flag on:

open /Applications/Google\ Chrome.app --args --disable-web-security --user-data-dir

Or install the CORS plugin for Google Chrome.

About

For more information about the MIMBCD-UI Project just follow the link. Pieces of information about details of this repository are also in a wiki. This prototype was developed using several libraries and dependencies. Despite that all libraries had their importance and supported the development, one of it was of chief importance. The CornerstoneJS library and secondary libraries, respectively, are supporting this prototype. We Acknowledge all people involved in the path.

License

The prototype-multi-modality repository is distributed under the terms of both Academic License and Commercial License, for academic and commercial purpose, respectively. For more information regarding the License of the hereby repository, just follow both ACADEMIC and COMMERCIAL files.

Copyright

When source code, documentation and other content is contributed to one of our repositories, the copyrights in those contributions remain owned by the original copyright holders. The copyrights are not assigned to the organization itself. Instead, they are licensed for distribution as part of the project. Whether a project uses new incremental solutions, the original copyright holders retain their copyrights.

Intellectual Property

The content of the present repository has obtained the patent right of World Intellectual Property Organization (WIPO) invention. Moreover, the hereby invention of the prototype-multi-modality repository is under protection of the patent number WO2022071818A1 with the application number PCT/PT2021/050029. The title of the invention is “Computational Method and System for Improved Identification of Breast Lesions”, registered under the WO patent office.

Acknowledgments

A special thanks to Chris Hafey, the propelling person of CornerstoneJS, who also developed the cornerstoneDemo. Not forgetting the three supporters of the CornerstoneJS library, Aloïs Dreyfus, Danny Brown and Erik Ziegler. We also would like to give a special thanks to Erik Ziegler who support several issues during this path.