Our process

Literature review methods were approved by the Retail Marijuana Public Health Advisory Committee (RMPHAC). Medline was the priority research database used to obtain articles for the review, though the Embase biomedical database and gray literature were secondarily reviewed when references in included articles were not included in the initial Medline search. Relevant articles cited in reviews or other primary studies also were included. Studies of marijuana use in humans were the primary focus of the review. Review of animal studies was reserved for specific topics with limited human research. In general, highly specialized research, such as brain imaging studies not directly associated with measurable clinical outcomes, was not evaluated in-depth unless an appropriately experienced reviewer was available. Research databases other than Medline were searched primarily when time allowed, though little additional data were found via these additional searches. All identified peer-reviewed literature on a given topic was reviewed, regardless of positive or negative findings or quality of the methods. For Medline searches, the appropriate Medical Subject Heading (MeSH) terms were chosen for each topic and used for the search. To find newer articles relevant to the topic (those without MeSH yet applied), a list of specific terms was established for each topic area. For example, the general search string used for marijuana was: "Cannabis"[Mesh] OR Marijuana Smoking"[Mesh] OR "Marijuana Abuse"[Mesh] OR cannabis OR marijuana OR marihuana OR hash oil OR hashish.

Findings were rated as high, medium, or low quality based on the strengths and limitations of the methods. Evaluation of the strengths and limitations was based on criteria in the “GRADE approach to evaluating the quality of evidence.”2 The GRADE system is a well-established method for systematic literature review and has been used by the Cochrane Collaboration, British Medical Journal, American College of Physicians, World Health Organization, and many others.2   

High quality 

The official definition is: “We are very confident that the true effect lies close to that of the estimate of the effect outlined in the study.” High quality findings originate from well-designed and well-controlled studies with few limitations. In the context of observational epidemiology studies, which was the most common study type in this systematic review, high quality does not necessarily imply causation. High quality implies that an observed association persists between an exposure and effect in an appropriately-sized study population after adjusting for appropriate confounders. 

Medium quality 

The official definition is: “We are moderately confident in the effect estimate outlined in the study. The true effect is likely to be close to the estimate of the effect, but there is a possibility that it is substantially different.” Moderate quality findings originate from studies that may be well designed, but have limitations that affect the interpretation of the results. In the context of observational epidemiology studies, moderate quality implies the finding of an observed association with an interpretation that may be limited by a small study population or insufficient adjustment for important confounders. 

Low quality 

The official definition is: “Our confidence in the effect estimate outlined in the study is limited. The true effect may be substantially different from the estimate of the effect.” Low quality findings originate from studies with significant methodological limitations that affect the interpretation of the results. In the context of observational epidemiology studies, low quality implies the finding of an observed association with an interpretation that is significantly restricted by study limitations. 

When critically reviewing the literature, all findings were initially considered medium quality and subsequently adjusted up or down in quality based on the strengths and limitations of the methodology. Quality ratings were applied to individual outcomes; therefore, it was possible for a single study to have multiple findings of differing quality. Criteria for evaluating strengths and limitations for this literature review included: 

• Methods of selecting exposed and comparison groups. 
• Relevance of study population to the population of interest. 
• Method for measuring exposure (self-report or other methods). 
• Method for describing the extent of exposure or marijuana use (e.g., ever vs. never, frequency measured by days used, measured by number of times used, etc.). 
• Adequacy of exposure and outcome group sizes. 
• Methods for measurement of the outcome(s) (validated tools, blinded if subjective, etc.). 
• Adequacy of adjustment for confounders (e.g., tobacco smoking, other drug use, education level, etc.) for both positive effects and lack of positive effect. 
• Full reporting vs. selective outcome reporting. 
• Effect size and width of confidence intervals. 
• Temporal relationship between exposure and effect. 
• Completeness of follow-up. 
• Adequacy of sample size for assessing a lack of positive effect.  

Findings were usually grouped based on outcome (health effect). However, in some situations, findings could be further subdivided based on factors such as: age range of the exposed population, special subject circumstances such as pregnancy or breastfeeding, level or method of marijuana use, and time period since last use of marijuana. Standardized definitions of level of use and age groups were established to help facilitate the grouping of findings. Findings from individual studies were grouped together to facilitate weighing the overall scientific evidence for specific topics.

Once findings were appropriately grouped, the overall evidence was weighed based on the quantity and quality rating of each individual finding. Depending on the weight, the overall evidence was then categorized into evidence levels. The following criteria was used to define each level of evidence:

Substantial evidence refers to:  

1. Robust scientific findings that support the outcome with no credible opposing scientific evidence. This was defined as any of the following: 
   • At least one high-quality positive finding, plus supporting findings at least one of which is medium quality, with no opposing findings (must include studies of at least two cohorts). 
   • At least three medium-quality positive findings from studies of at least two cohorts, with no opposing findings. 
   • Many high- and medium- quality positive findings from studies of at least two cohorts that heavily outweigh opposing findings.  
   • At least two high-quality positive findings from systematic reviews or meta-analyses published within the past 10 years. 
2. A robust body of scientific literature that has examined the outcome and failed to demonstrate a positive finding. This was defined as any of the following: 
   • At least one high-quality study lacking a positive finding, plus at least one medium quality supporting study, and no opposing findings (must include studies of at least two cohorts). 
   • At least three medium-quality studies lacking a positive finding from studies of at least two cohorts, and no opposing findings. 
   • Many high- and medium-quality studies lacking a positive finding that heavily outweigh opposing findings. 
   • At least two high-quality systematic reviews or meta-analyses published within the past 10 years lacking positive findings. 

Moderate evidence refers to: 

1. Strong scientific findings that support the outcome, but these findings have some limitations. This was defined as any of the following: 
   • A single high-quality positive finding, with no opposing findings. 
   • At least one medium-quality positive finding, plus supporting findings with no opposing findings; supporting findings can include animal studies. 
   • Many medium- and low-quality positive findings from studies of at least two cohorts that heavily outweigh opposing findings. 
   • A single high-quality positive finding from a systematic review or meta-analysis published within the past 10 years. 
2. A strong body of scientific literature that has examined the outcome and failed to demonstrate a positive finding. This was defined as any of the following: 
   • A single high-quality study lacking a positive finding, and no opposing findings. 
   • At least one medium-quality study lacking a positive finding, plus supporting findings, and no opposing findings. 
   • Many medium- and low-quality studies lacking positive findings from studies of at least two cohorts that heavily outweigh opposing findings. 
   • A single high-quality systematic review or meta-analysis published within the past 10 years lacking positive findings. 

Limited evidence refers to: 

1. Modest scientific findings that support the outcome, but these findings have significant limitations. This was defined as any of the following: 
   • A single medium-quality positive finding. 
   • Two or more low-quality positive findings from studies of at least two cohorts. 
   • Many low-quality positive findings from studies of at least two cohorts that outweigh opposing findings. 
2. Modest scientific findings that have examined the outcome and failed to demonstrate a positive finding. This was defined as any of the following: 
   • A single medium-quality study lacking a positive finding. 
   • Two or more low-quality studies lacking positive findings from studies of at least two cohorts. 
   • Many low-quality studies lacking positive findings from studies of at least two cohorts that outweigh opposing findings. 

Mixed evidence refers to: 

Mixed evidence indicates both supporting and opposing scientific findings for the outcome with neither direction dominating. Mixed was defined as “mixed findings, with neither direction dominating.” 

Insufficient refers to: 

Insufficient evidence indicates that the outcome has not been sufficiently studied. Insufficient was 
defined as: 

• A single low quality positive finding with no supporting findings. 
• There are no studies examining the outcome or relevant parameters. 

Once the findings were weighted and evidence level determined, evidence statements were drafted using standardized language per evidence level. Every Evidence Statement is approved by the Retail Marijuana Public Health Advisory Committee (RMPHAC).

• Substantial positive evidence becomes: “We found substantial evidence…” 
• Substantial lack of positive evidence becomes: “We found a substantial body of research that failed to show an association…” 
• Moderate positive evidence becomes: “We found moderate evidence…” 
• Moderate lack of positive evidence becomes: “We found a moderate body of research that failed to show an association…” 
• Limited evidence becomes: “We found limited evidence…” 
• Limited lack of positive evidence becomes: “We found a limited body of research that failed to show an association…” 
• Mixed evidence becomes: “We found mixed evidence for whether or not…” 
• Insufficient evidence becomes: “There is insufficient evidence to determine…” 

Evidence statements were drafted by CDPHE technical staff, revised based on committee review and feedback from technical advisors and public stakeholders, and approved by a vote of the committee. 

Evidence statements were translated into public health statements using a standardized convention to ensure traceability back to the scientific literature. Public health statements were designed to accurately reflect the evidence statements using language the public can understand. The Retail Marijuana Public Health Advisory Committee (RMPHAC) sought to ensure the public health statements:

1. Conveyed the volume and quality of research related to the outcome;
2. Provided a generalized framework to allow consistent language for all findings regardless of topic;
3. Allowed the statement to stand on its own without context.

These statements were drafted by CDPHE technical staff, revised based on comments from the committee, technical advisors and public stakeholders, and approved by a vote of the committee. The standardized convention used for the translation is shown below: 
Standardized convention: <level of> marijuana use <by specific group> <strength of relationship> associated with <outcome>, <specific circumstances>. 
A specific example: “Weekly or more frequent marijuana use by adolescents and young adults is associated with impaired learning, memory, math and reading achievement, even 28 days after last use.” 

Standard language was chosen for the “strength of relationship,” corresponding to the level of evidence from the evidence statements: 

• Substantial positive evidence becomes “is strongly associated.” 
• Substantial research lacking positive evidence becomes “an association is unlikely.” 
• Moderate positive evidence becomes “is associated.” 
• Moderate research lacking positive evidence becomes “an association appears unlikely.” 
• Limited evidence becomes “may be associated.” 
• Limited research lacking positive evidence becomes “might not be associated." 
• Mixed evidence becomes “There is conflicting evidence for whether or not ___ is associated.” 

The wording “associated with” was specifically chosen to represent epidemiologic (i.e., statistical) associations, and NOT to imply causality. 

Based on the literature review, public health recommendations were drafted. The Retail Marijuana Public Health Advisory Committee (RMPHAC) separated recommendations by data quality issues, surveillance, and education recommendations. Data quality issues were defined as recommendations to improve current data collection deficiencies at the clinical or governmental level that prevent full analysis of public health outcomes related to marijuana use. Public health surveillance recommendations were based on improving capacity to detect an acute public health danger (e.g., real-time emergency department surveillance for detection of poisonings from contaminated products); the ability to characterize chronic public health dangers to support policy and other intervention decisions (e.g., surveillance of marijuana-related traffic fatalities or skiing injuries); or the ability to generate epidemiologic data (e.g., BRFSS survey questions), to contribute to planning and evaluating population level interventions. Education recommendations were included to ensure evidence-based information on potential health effects of marijuana use is provided to the appropriate target audiences.

CDPHE technical staff formally presented findings, evidence statements, public health statements, public health recommendations, and research gaps to the Retail Marijuana Public Health Advisory Committee (RMPHAC) for review and revision during open public meetings. During these public meetings, interested stakeholders and members of the public were invited to provide comments relevant to the topics presented. The RMPHAC chairperson facilitated a consensus process to ensure all RMPHAC  members could agree on the scientific evaluation and wording. Once consensus was achieved, the RMPHAC voted to officially accept these statements and recommendations.

The Retail Marijuana Public Health Advisory Committee (RMPHAC) will continue to meet quarterly. All approved evidence statements, public health statements, public health recommendations, and research gaps will be reviewed on a quarterly cycle and updated as needed based on new evidence. The RMPHAC will also expand the reviewed literature to include new topics as new research becomes available or new public health concerns arise.

1. Moher D, Liberati A, Tetzlaff J, Altman DG, Group P. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. J Clin Epidemiol. 2009;62(10):1006-1012. 
2. GRADE guidelines - best practices using the GRADE framework. GRADE working group http://training.cochrane.org/path/grade-approach-evaluating-quality-evi…, 2014.