Information


The platform was designed as a shared space for humans and machines to work together. If you are a human, the best way to start with the system is to browse examples of systematic reviews that have already been manually verified. If you are a machine, then you probably already know why you are here.

If you haven’t already registered as a user, then contact us, answer a few questions, and we should be able to provide you with access to the full functionality of the website and access to the underlying data.


The research that sits under the platform was supported by a grant from the Agency for Healthcare Research and Quality (R03HS024798) and the National Library of Medicine of the National Institutes of Health (R01LM012976), with investigators Florence Bourgeois, Adam Dunn, and Kenneth Mandl. We also received support from the AWS Cloud Credits for Research program. This website and the underlying systems were developed and updated by Paige Martin and Jason Dalmazzo.


The best reference to use when citing the platform:

P Newman, R Bashir, D Surian, FT Bourgeois, AG Dunn (2018)
trial2rev: combining machine learning and crowd-sourcing to create a shared space for updating systematic reviews
JAMIA Open. doi: 10.1093/jamiaopen/ooy062.

Other related research includes the following:

D Surian, AG Dunn, L Orenstein, R Bashir, E Coiera, FT Bourgeois (2018)
A shared latent space matrix factorisation method for recommending new trial evidence for systematic review updates
Journal of Biomedical Informatics, 79:32-40. doi: 10.1016/j.jbi.2018.01.008; preprint: arXiv: 1709.06758.
AG Dunn, E Coiera, FT Bourgeois (2018)
Unreported links between trial registrations and published articles were identified using document similarity measures in a cross-sectional analysis of ClinicalTrials.gov
Journal of Clinical Epidemiology, 95:94-101. doi: 10.1016/j.jclinepi.2017.12.007; preprint: arXiv: 1709.02116.
R Bashir, FT Bourgeois, AG Dunn (2017)
A systematic review of the processes used to link clinical trial registrations to their published results
Systematic Reviews, 6:123. doi: 10.1186/s13643-017-0518-3.


If you want to see what we know so far about a published systematic review, simply copy and paste the PubMed identifier for your systematic review into the search box at the top of this page. If it is not already in our database, then the system will add it to the database and then look for the set of trials that were included in the review and have registrations in ClinicalTrials.gov.

If we have found what we think are included trials, those will be listed on the left. If you are a registered user and logged in, you can check through them to see if some are incorrect or missing. As you add new trials, other software agents will start to vote up or down on other trials with registrations in ClinicalTrials.gov to recommend trials that might be included in an update on the right, as well as suggest other related systematic reviews on the far left.

Once you are satisfied that all of the trials included in the published systematic review are present on the left, you can look through the details of the similar trials (on the right) and vote up or down on those that are likely or unlikely to be relevant. If there are trials that you can’t immediately see in the list but you know are likely to be relevant to an update, you can vote them in manually.


Yes, once you have registered and logged in, the changes you make are persistent. As you add or confirm trials that are included in systematic reviews or vote on trials that are relevant to their updates, then you are helping us to train our software agents to be smarter and better at predicting which new trials may be relevant to already-published systematic reviews.


When viewing a systematic review, click on the star at the top right to save it to your list of saved reviews. The systematic review will appear in your saved reviews list and will be available each time you log in. You can remove reviews from your saved list too.


You can use the create function to fill in preliminary details of a systematic review or its protocol to check whether there are other recently published systematic reviews asking the same question. There are two ways to do this.

If you have a list of one or more trials that are likely to be included in your systematic review, add them using their NCT Number to the list of flagged trials on the left. If we can find relevant systematic reviews that have included any of those trials, they will be displayed on the far left. We will also display a list of potentially relevant trials that you can flag to get a more precise idea of the set of systematic reviews in the area.

If you have a short description, a PICO specification, or a protocol for your systematic review you can simply copy and paste any or all the text into the abstract section at the top of the create page. If we can find trials that might be relevant those will be listed in order of relevance on the right, and related systematic reviews will be listed on the far left. You can then change the text or flag any trials that might be relevant to further refine the systematic review idea and check for other similar published systematic reviews.


We provide up-to-date access to all available links between published systematic reviews and included trial registrations, and scores on trials that may be relevant to an update—all as a single large sparse matrix. While the matrix is incomplete and imperfect right now, we hope it will function as a shared resource for researchers in the area to help us all improve machine learning tools used in trial screening as well as tools used to support the prioritisation of systematic review updates where there is a risk of a change in conclusion.

If you are interested in accessing the messy and incomplete dataset, contact us directly.


The 3S3 system is also designed to help you quickly estimate if a published review might be outdated. We have included an early version of a tool that estimates the risk that a systematic review would change its conclusions if it was updated based on evidence from trials that have recently been completed.

Please note that the 3S3 system is provided as-is and we make no guarantees about the accuracy of the data or any of the predictions. It is not intended to replace expert guidance on the current value or credibility of any systematic review.


If you are interested in how the interface works, want to examine one or more of the software agents we use to fetch new data, or want to improve on the machine learning methods we use to train our other intelligent agents, read the published work listed above or visit our GitHub project page for more documentation and code.


Send us a message via our contact page.


Your privacy is important to us. It is 3S3's policy to respect your privacy regarding any information we may collect from you across our website, https://surveillance-chi.mq.edu.au/, and other sites we own and operate.

We only ask for personal information when we truly need it to provide a service to you. We collect it by fair and lawful means, with your knowledge and consent. We also let you know why we’re collecting it and how it will be used.

We only retain collected information for as long as necessary to provide you with your requested service. What data we store, we’ll protect within commercially acceptable means to prevent loss and theft, as well as unauthorised access, disclosure, copying, use or modification.

We don’t share any personally identifying information publicly or with third-parties, except when required to by law.

Our website may link to external sites that are not operated by us. Please be aware that we have no control over the content and practices of these sites, and cannot accept responsibility or liability for their respective privacy policies.

You are free to refuse our request for your personal information, with the understanding that we may be unable to provide you with some of your desired services.

Your continued use of our website will be regarded as acceptance of our practices around privacy and personal information. If you have any questions about how we handle user data and personal information, feel free to contact us.

This policy is effective as of 27 August 2018.


Terms

By accessing the website at https://surveillance-chi.mq.edu.au/, you are agreeing to be bound by these terms of service, all applicable laws and regulations, and agree that you are responsible for compliance with any applicable local laws. If you do not agree with any of these terms, you are prohibited from using or accessing this site. The materials contained in this website are protected by applicable copyright and trademark law.

Use License

a. Permission is granted to temporarily download one copy of the materials (information or software) on 3S3's website for personal, non-commercial transitory viewing only. This is the grant of a license, not a transfer of title, and under this license you may not:

  • i. modify or copy the materials;
  • ii. use the materials for any commercial purpose, or for any public display (commercial or linon-commercial);
  • iii. attempt to decompile or reverse engineer any software contained on 3S3's website;
  • iv. remove any copyright or other proprietary notations from the materials; or
  • v. transfer the materials to another person or "mirror" the materials on any other server.

b. This license shall automatically terminate if you violate any of these restrictions and may be terminated by 3S3 at any time. Upon terminating your viewing of these materials or upon the termination of this license, you must destroy any downloaded materials in your possession whether in electronic or printed format.

Disclaimer

a. The materials on 3S3's website are provided on an 'as is' basis. 3S3 makes no warranties, expressed or implied, and hereby disclaims and negates all other warranties including, without limitation, implied warranties or conditions of merchantability, fitness for a particular purpose, or non-infringement of intellectual property or other violation of rights.

b. Further, 3S3 does not warrant or make any representations concerning the accuracy, likely results, or reliability of the use of the materials on its website or otherwise relating to such materials or on any sites linked to this site.

Limitations

In no event shall 3S3 or its suppliers be liable for any damages (including, without limitation, damages for loss of data or profit, or due to business interruption) arising out of the use or inability to use the materials on 3S3's website, even if 3S3 or a 3S3 authorized representative has been notified orally or in writing of the possibility of such damage. Because some jurisdictions do not allow limitations on implied warranties, or limitations of liability for consequential or incidental damages, these limitations may not apply to you.

Accuracy of materials

The materials appearing on 3S3's website could include technical, typographical, or photographic errors. 3S3 does not warrant that any of the materials on its website are accurate, complete or current. 3S3 may make changes to the materials contained on its website at any time without notice. However 3S3 does not make any commitment to update the materials.

Links

3S3 has not reviewed all the sites linked to its website and is not responsible for the contents of any such linked site. The inclusion of any link does not imply endorsement by 3S3 of the site. Use of any such linked website is at the user's own risk.

Modifications

3S3 may revise these terms of service for its website at any time without notice. By using this website you are agreeing to be bound by the then current version of these terms of service.

Governing Law

These terms and conditions are governed by and construed in accordance with the laws of NSW and you irrevocably submit to the exclusive jurisdiction of the courts in that State or location.