Icbo.buffalo.edu

Reporting adverse events: basis for a common representation M´elanie Courtot1, Ryan R. Brinkman1,2 and Alan Ruttenberg3
1BC Cancer Agency, Vancouver, BC, Canada, 2Department of Medical Genetics, University of British Columbia, Vancouver, BC, Canada, 3School of Dental Medicine, University at Buffalo, NY, USA Abstract. Reports of adverse events aim to monitor the status of patients in clinical trials or to provide
ongoing monitoring of the safety of interventions once they are in the market. They help identify issues
with treatment safety and efficacy, and allow for better education of health practitioners and the general
public, ultimately allowing us to learn from our mistakes. However if such reports are to be maximally
useful, the information they contain must be unambiguously shared, standardized, and accurately docu-
mented. Towards this end, we briefly review existing reporting standards and then define adverse event in
a manner consistent with the use of the term in existing reporting guidelines. Novel aspects of this work
include attention to the distinction between the classification of adverse events based on reporting ver-
sus the pathological process types they attempt to monitor, and integration with relevant OBO ontologies
to minimize redundant definitional work as well as enable integration of adverse event reporting into
the broader landscape of representation for translational medicine. Implementation of a prototype that
incorporates this approach is discussed - the Adverse Events Reporting Ontology (AERO).
Introduction
Adverse events reporting is a major part of clinical research, and an important tool to improve patientsafety. By collecting and analyzing adverse events we can better understand and prevent them, as wellas communicate issues and evidence among researchers, policy-makers and public, letting us learn from,and take action based on, our mistakes. However, the manner by which adverse events are classified andreported differs from agency to agency and from treatment type to treatment type. Therefore, in order toachieve large scale integration of reports of adverse events, a careful approach to the representation ofadverse events must be agreed upon.
This paper presents such an approach, using examples from the Brighton Collaboration guidelines for reporting adverse events following immunization, and addresses current limitations in reporting systemsthat limit their effective wider scale use. An early implementation of this approach is our Adverse EventsReporting Ontology (AERO).
Background
The Guidance for Clinical Safety Data Management: Definitions and Standards for Expedited Reporting [1],defines an adverse even as “Any untoward medical occurrence in a patient or clinical investigation subjectadministered a pharmaceutical product and which does not necessarily have to have a causal relationshipwith this treatment.” The guide then adds “An adverse event (AE) can therefore be any unfavourable andunintended sign (including an abnormal laboratory finding, for example), symptom, or disease temporallyassociated with the use of a medicinal product, whether or not considered related to the medicinal product.”The Report of Adverse Event Following Immunization (AEFI) user guide [2] from the Public Health Agencyof Canada (PHAC) adheres to this definition and adapts it for AEFI reporting: “An AEFI is any untowardmedical occurrence in a vaccine which follows immunization and which does not necessarily have a causalrelationship with the administration of the vaccine”.
Not detailed in this statement is the additional fact that reporting guidelines often provide protocols for determining and reporting the likelyhood that specific pathological processes have occured, and that suchprotocols and reporting conventions differ from jurisdiction to jurisdiction, from investigation to investiga-tion, and by symptom and severity.
Therefore adverse events as recorded in reports, contrary to what might otherwise be presupposed, are not necessarily processes, are not necessarily of the type reports say they are, are not necessarily causallyrelated to the intervention which led to them being reported, and the terms used to describe them are notnecessarily univocal.
It is a goal of our work to nonetheless provide a coherent account and workable system for managing these report in such a way as to maximize their utility.
The definitions above states that causality doesn’t need to be established for the event to be reported, andmatches the usage made for example within the Vaccine Adverse Event Reporting System (VAERS) [3],who mentions “VAERS collects data on any adverse event following vaccination, be it coincidental or trulycaused by a vaccine”.
Therefore, at the time of data entry by the physician and submission into adverse events reporting sys- tems, no hypothesis of causality is necessary. Rather, current guidelines [2] specify that events should bereported on the basis of their temporal association with the medical intervention, and depending on (i)the type of immunizing agent (30 days after live vaccine or 7 days after killed or subunit vaccine) or (ii)biological mechanism (up to 8 weeks for immune-mediated events). Even though in some cases, and basedon their personal experience, clinicians may think that some adverse events are most probably caused bythe intervention, and even take action based to guard the patient’s health based on this assessment, theynonetheless must report any event occurring in the respective corresponding time frame. In that waysrecords accumulated from many clinicians may be reviewed safety committees, where evidence towardscausality establishment will be reviewed and policy recommendations, based on the best available evi-dence, can be made.
Issues with current adverse event reporting systems While all practitioners agree on the importance of reporting adverse event in increasing public health safety,current methods used for spontaneous adverse events reporting are not sufficient, mitigating their use-fulness. For example, there is no standardization of the terminology used in the current Electronic DataCapture (EDC) used by PHAC. At best, a Medical Dictionary of Regulatory Activities (MedDRA) [4] codeis assigned after parsing the clinician’s input, but this code is not linked to any definition. This in turn maylead to heterogeneity in the diagnostics recorded - physicians may have slightly different interpretationsof what constitutes a seizure for example. Several studies highlight the potential issues in using MedDRAfor adverse event reporting, ranging from inaccurate reporting as several terms are non-exact synonyms, tolack of semantic grouping features impairing processing in pharmacovigilance [5–8]. Additionally, in manysystems, only the adverse event code as determined by the system (e.g. resulting from parsing the textualinput) is saved, and information about signs and symptoms used in the determination of that code are lost.
This limits the ability of analysts to review the set of symptoms observed in order to establish a consistentdiagnosis. The resultant lack of consistency limits the ability to query and assess important safety issues theresulting datasets might otherwise support.
A general review of systems used in other countries provides similar results. The Adverse Event Report- ing System (AERS) [9] and VAERS systems used in the US rely on MedDRA to encode adverse events. Theyfollow the international safety reporting guidance [10] which specifies:“Only the MedDRA Lowest LevelTerm (LLT) most closely corresponding to the reaction/event as reported by the primary source shouldbe provided” In Europe, the Vaccine European New Integrated Collaboration Effort (VENICE) [11] groupreports [12] that only 71% (17/24) of the countries states have adopted a classification of AEFIs, and thatthose chosen classifications are heterogeneous: 38% WHO 1 and 62% other or not specified.
1 The reports can be difficult to even interpret - the WHO Adverse Reaction Terminology (WHO-ART) is a non-open terminology and only a 1997 version appears to be publicly visible, hosted at http://bioportal.bioontology.
org/ontologies/40404. It lacks many terms that are essential for AE reporting, such as those related to seizure.
The Brighton Collaboration [13] is a global network of experts that aims to provide high quality vaccinesafety information. It has done extensive work towards standardizing the assessment and reporting ofadverse events following vaccination. The Brighton Collaboration publish four artifacts of interest to ourcurrent work. The case definitions they provide relate symptoms and signs to assessments of whether aparticular type of pathological process has occurred, assigning qualitative levels of certainty. They provideguidelines for three activities - data collection, analysis, and presentation of results, aiming to make col-lected data comparable, informed by the case definitions. By determing and publishing these guidelines,the collaboration creates methodological standards that enable accurate risk assessment. The case defini-tions neither require, nor assses a causal relation between a given adverse event and the immunizationprocess. Rather, the case definitions are designed to define levels of diagnostic certainty based on knowninformation about AEFIs.
The Brighton Collaboration has published a number of papers presenting these guidelines and case definitions, each aimed at reporting potential pathological processes, such as Seizure [14] and Guillain-Barr´e Syndrome [15]. In our prototype we have worked with the seizure case definition.
Implementation
Our prototype is aimed to address a number of issues. While complete, the textual article-like format of theBrighton case definitions makes it difficult for a clinician to confirm that she sees the relevant symptomswhen making the adverse event diagnostic: case definitions are buried within the scientific paper. In thetextual form, the case definition is not amenable to automated diagnosis - clinicians cannot choose whichsymptoms are observed and then infer the proper event diagnostic. A formal and logical description ofvaccine adverse events would allow software tools to process the information and present only relevantitems in a checklist to the physician, making it easier to validate upon data entry. An ontology-based sys-tem at the time of data entry will increase data accuracy and completeness. For example, when the clinicianselects seizure as adverse event, he will be offered a list of symptoms that may have manifested. By se-lecting the ones he did observe, the system will be able to confirm his diagnostic, potentially specifying it,such as assigning a level of certainty based on the Brighton case definition. The system will also be able toinfirm the diagnostic, by warning that the set of events selected doesnt allow for unambiguous diagnostic.
In the latter case, the system will also provide a list of such events that would allow determination. Takentogether, those will enable, at the time of data entry, to unambiguously refer to a specific set of symptoms,each carefully defined, and establish a diagnostic, which remains linked to its associated symptoms. Theadverse event will also be formally expressed, making it amenable to further querying for example for sta-tistical analysis “what percentage of patients presented with motor manifestations?”) at different levels ofgranularity (e.g., facilitating queries such as “what percentage of patients presented with tonic-clonic motormanifestations?”) Finally, by agreeing on a common defined vocabulary we can increase data interoperabil-ity, and enable cross databases queries across different centres or against public datasets, such as literaturereferences or other AEFI datasets.
In the following sections we present a proof of concept prototype, the AERO, based on the seizure case In implementing a distinct resource for adverse event reporting, care was taken to reuse, when possible,work done in the context of other efforts. Reusing terms from other resources allowed us to rely on knowl-edge of domain experts who curated them, to dedicate more work time for terms that need to be createdde novo and increases interoperability within resources. When only few terms of interest were identified More importantly, WHO-ART follows a 4-level structure similar to MedDRA, and therefore suffers some of the samedefects.
in external ontologies, those have been imported relying on the Minimum Information to Reference anExternal Ontology Term (MIREOT) guideline [16]. For example the Vaccine Ontology (VO) [17] definesthe vaccination process as an “administering substance in vivo that involves in adding vaccine into a host(e.g., human, mouse) in vivo with the intend to invoke a protective immune response”, and we use it as asynonym of the ”immunization process” needed to define vaccine adverse events. Similarly, we use classesfrom the Ontology for General Medical Science (OGMS) [18]. OGMS aims at modeling pathological entities,diseases and diagnosis, and some of its classes such as disorder and sign are at the root of very importantAERO hierarchies; more details on their usage is shown below. In other cases, we decided to import exter-nal ontologies as a whole: (i) the Relations Ontology (RO) [19] contains a set of common relations, (ii) theInformation Artifact Ontology (IAO) [20] deals with information entities and metadata, and (iii) the BasicFormal Ontology (BFO) is used as our upper-level ontology. Those resources are commonly used by theOpen Biomedical Ontologies (OBO) Foundry [21] ontologies, of which AERO aims to be a part; relying onthem for our prototype will improve integrability of our resource within the Foundry framework.
Consider the following cases in which the clinician wishes to report adverse events: – sensorineural deafness reported after measles, mumps, and rubella immunization. This disturbance of
the cochlea or auditory nerve results in hearing impairment, often loss of ability to hear high frequencies[22], – predisposition to infection by another agent such as in the case of leflunomide in treatment of arthritis
– any of the dermatological adverse events observed in patients treated with etanercept [24],
– headaches reported following use of proton pump inhibitors such as lansoprazole [25],
– or even simple rashes, extremely common for example at the injection site.
These cases indicate that adverse events can not only be bfo:occurrent but also bfo:continuants, and that both should be recorded. OGMS currently defines sign as “A quality of a patient, a material entity that is partof a patient, or a processual entity that a patient participates in, any one of which is observed in a physicalexamination and is deemed by the clinician to be of clinical significance.” and symptom as “A quality of apatient that is observed by the patient or a processual entity experienced by the patient, either of which ishypothesized by the patient to be a realization of a disease.”. Those classes are sibling of the bfo:continuantand bfo:occurrent classes, directly asserted under bfo:entity2. Adverse events clearly match those definition:they can be quality of the patient (for example, headaches), a material entity part of the patient (e.g., rash),or a processual entity that a patient participates in (e.g., seizure). Adverse events are observed during aphysical examination (in which case they are a ogms:sign) or self-reported (for ogms:symptom). Based onthis, we created a top level equivalent class aero:sign or symptom which is an helper class defined as theunion of both types, and allows us to built our adverse event hierarchy.
Following this, we logically define aero:adverse event as a sign which is the union of aero:adverse event processand aero:disorder resulting from an adverse event process (i.e., the adverse event continuant described above).
An aero:adverse event process is “a processual entity occurring in a pre determined time frame following ad-ministration of a coumpound or usage of a device”; this can be logically translated as (using the ManchesterOWL syntax [26]): 2 Throughout this paper we will adopt the notation prefix:label for entities, where prefix is the commonly used resource Fig. 1. The disorder hierarchy as built in AERO, under the ogms:disorder class.The class adverse event rash is logically
defined as the intersection of disorder resulting from an adverse event process and rash.
(’adding a material entity into a target’ or ’administering substance in vivo’)) where the classes adding a material entity into a target and administering substance in vivo are imported from the Ontology of Biomedical Investigations (OBI) [27]. The AERO definition of adverse event processis meant to be inclusive, and cover cases such as those described by the Manufacturer and User FacilityDevice Experience (MAUDE); for example the case of a patient fitted with bioprosthetic heart valves whodies within the following 4 months3. It is also worth noting that this definition of adverse event doesn’timply causation between the sign observed and the coumpond administration/device utilization, but israther based on temporal association.
The adverse event continuant hierarchy was built under the ogms:disorder class (Figure 1), which is de- fined as “A material entity which is clinically abnormal and part of an extended organism. Disorders arethe physical basis of disease.” To avoid any language ambiguity by associating the terms event and con-tinuant in the label of the class adverse event continuant, it was renamed disorder resulting from an adverseevent process. As a general way of overcoming the potential issue between terms in use by clinicians and on-tological usage in the context of the OBO Foundry, in which it may be confusing to associate the word“event” to a hierarchical position under continuant, we chose to rely on the OBO Foundry unique labelIAO annotation property (http://purl.obolibrary.org/obo/IAO_0000589). Classes such as ad-verse event rash (EquivalentTo: disorder resulting from an adverse event process andrash) will therefore have an OBO Foundry unique label annotation with value “rash resulting from anadverse event process”, which can be processed by the OBO package manager currently being written bythe OBO Foundry.
Finally, as presented in the background section, adverse events definitions are based on the Brighton case definitions. We built under the IAO class directive information entity and defined a clinical guideline asa “A directive information entity that establishes a diagnostic based on a set of signs or symptoms” andits subclass Brighton case definition with definition “A clinical guideline in which a set of signs is describedto establish a diagnostic of adverse events with a related degree of certainty, as defined by the BrightonCollaboration, http://www.brightoncollaboration.org/”.
The resulting inferred hierarchy, presented in figure 2, shows how defined classes are positioned under 3 http://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfMAUDE/detail.cfm?mdrfoi__id= Fig. 2. The resulting inferred hierarchy. Leaves tems such as AEFI rash are inferred under several parents based on their
logical restrictions.
Discussion
As pointed out by Ceusters and al.[28], the term adverse event has been described in multiple ways. Theyconsider adverse events as being those for which causality has been demonstrated, arguing that “the pastcannot be changed: something which is not an adverse event at the time it happens cannot become oneat some point thereafter.” This view is adopted by the Adverse Events Ontology (AEO)4, which definesadverse event as “a pathological bodily process that is induced by a medical intervention”. This definitionhowever presents several issues that make it unusable for reporting vaccine adverse events. First, at thetime of data entry, there is no certainty that the adverse event has been induced by the intervention. In-stead, in clinical settings, all reactions are reported and forwarded to a committee that will later on try andestablish causality. As demonstrated above, such causality will never be formerly established, rather a net-work of proofs is collected via epidemiologic studies and/or case reports, and supported by demonstrationof biological plausibility. Second, reported adverse events may not always be processes - continuants suchas rash are reported by clinicians and need to be accommodated as well. Describing the process leadingto the rash is a first step as is done in the AEO, but i) this process is not always known ii) clinicians don’treport not are particularly interested in the process - they rather care about the dermatological disorderobserved. Finally, we argue that an adverse event rash and a rash non temporally associated with a medicalintervention are the same entity, which should be described in a distinct, external symptom ontology - viewshared by Dr. Ceusters 5. It then becomes obvious that this rash is the universal being observed, and its na-ture - whether we can prove or not that it is caused by the intervention - doesn’t change. We can declare theobserved rash as an adverse event on the basis of its temporal association, via a defined class as is done inAERO, describe how it follows some reporting guidelines such as Brighton, and later on assign a category 4 http://sourceforge.net/projects/aeo/5 http://sourceforge.net/mailarchive/message.php?msg_id=27040555 of evidence for causality, which would be an information entity attached to the original adverse event rashafter review by the safety committee.
While implementing the prototype, several ontological issues arose. Adverse event is defined as a process
or continuant preceded by some medical intervention or drug administration. The preceded by relation is im-
ported from the RO, and defined as “P preceded by P’ if and only if: given any process p that instantiates P
at a time t, there is some process p’ such that p’ instantiates P’ at time t’, and t’ is earlier than t”. This relation
doesn’t specify a timeframe for the events to be considered related, and an immunization process happen-
ing right after one’s birth would de facto precede most of the subsequent events in their life. A comment on
this relation in the RO file6 indicates that this is an area RO developers are considering improving, and they
suggest stronger relations such as immediately preceded by or an indication that the instances P and P’ share
participants. Another issue related to use of relations appears when logically defining disorder resulting from
an adverse event process. We use the relation is specified output to represent that each disorder results from the
adverse event process. However, the range of this relation is planned process - processes executed following
a plan and with the intent to achieve a specific objective - which obviously adverse event processes are not.
We expect BFO27 to provide a suitable relation for this case. Finally, there is currently no relation linking
a disease and its set of signs and symptoms. This issue has been raised in the OGMS8 but poses the ques-
tion of the universality of the association between a disease and its symptoms. We currently are unable to
associate all instances of the disease with the whole set of symptoms (e.g., not every instance of influenza
disease is associated with an instance of fever, and certainly not vice versa). We propose here to use the case
definitions (or any other source of canonical knowledge) to relate a set of signs and symptoms to a specific
class, defined based on the guideline considered. For example, while not all seizures cases present with
witnessed loss of consciousness (and vice versa, not all loss of consciousness are associated with a seizure
episode), we can say that, according to the Brighton case definition, if there is a sudden witnessed loss of
consciousness (in conjunction with another set of symptoms) then we can diagnose a seizure with level 1
of certainty.
Future work
Some elements of the current prototype deserve a bit more attention. For example, the levels of diagnosticcertainty as defined by Brighton should probably be some type of information entity that would then beattached to the AEFI seizure class. Similarly, the degree of severity of adverse event, as well as their ex-pectedness, is important information that should be modeled in the ontology. Serious adverse events 9 arelife-threatening or causing death, as well as requiring hospitalization or permanent disability. Unexpectedadverse events 10are those not mentioned in drug manufacturers notices for examples. Of course, unex-pected, serious, adverse event are of special concern to the safety committee. We also expect to be able tooutsource our definitions of “normal” symptoms and signs. For example, the rash class in figure 1 should beimported from a common symptom ontology. This would however require consensus definition for thoseelements, which may prove difficult. We have had extensive discussion with the developers of the AEO thataided the preparation of this paper, and both groups hope we will be able to reconcile the two resources inthe future.
6 http://www.obofoundry.org/ro/ro.owl7 http://code.google.com/p/bfo/8 http://code.google.com/p/ogms/issues/detail?id=459 http://www.hc-sc.gc.ca/dhp-mps/prodpharma/applic-demande/guide-ld/ich/efficac/ 10 http://www.hc-sc.gc.ca/dhp-mps/prodpharma/applic-demande/guide-ld/ich/efficac/ Finally, we would like to proceed with testing of the prototype in a real use context, and are in contact with the Dacima developers to implement an extension of the Daciforms user interface clinicians withinthe PHAC/CIHR Influenza Research Network (PCIRN) network are already familiar with. At data entrytime, they will be presented with a succession of choices augmented by their precise description and checksensuring signs described match the adverse event reported and vice-versa. This will lead in an increasedaccuracy and quality of reported adverse events, and will ultimately improve patient safety.
Availability
The AERO project, including ontology and documentation which is available at http://purl.obolibrary.
org/obo/aero. AERO is also listed on the OBO library at http://obofoundry.org/cgi-bin/detail.
cgi?id=AERO and under BioPortal at http://bioportal.bioontology.org/visualize/45521.
Participation in the AERO is welcome and contributions in any form are encouraged.
Acknowledgments
The authors’ work was partially supported by funding from the Public Health Agency of Canada / Cana-dian Institutes of Health Research Influenza Research Network (PCIRN), and the Michael Smith Foundationfor Health Research.
Bibliography
[1] Canada Minister of Health. Clinical Safety Data Management Definitions and Standards for Expedited Report- ing - http://www.hc-sc.gc.ca/dhp-mps/prodpharma/applic-demande/guide-ld/ich/efficac/e2a-eng.php#a2A1.
[2] Public Health Agency of Canada. User Guide: Report of Adverse Events Following Immunization (AEFI) - http: //www.phac-aspc.gc.ca/im/pdf/AEFI-ug-gu-eng.pdf.
[3] US Food and Drug Administration. Vaccine Adverse Event Reporting System Data - http://vaers.hhs.gov/ [4] MedDRA Maintenance and Support Services Organization. Medical Dictionary of Regulatory Activities - http: [5] Merrill G. The MedDRA paradox. AMIA Annual Fall Symposium 2008, pages 470–474, 2008.
[6] Krischer J Richesson R, Fung K. Heterogeneous but standard coding systems for adverse events: Issues in achiev- ing interoperability between apples and oranges. Contemp Clin Trials, 29, 2008.
[7] Mozzicato P. Standardised meddra queries: their role in signal detection. Drug Safety, 30, 2007.
[8] Almenoff J, Tonning J, Gould A, and Szarfman A et al. Perspectives on the use of data mining in pharmaco- vigilance. Drug Safety, 28:981–1007, 2005.
[9] US Food and Drug Administration. Adverse Event Reporting System Data - http://www.fda.gov/Drugs/ GuidanceComplianceRegulatoryInformation/Surveillance/AdverseDrugEffects/default.
[10] ICH Expert Working Group E2B(R). E2B(R) Clinical Safety Data Management: Data elements for transmission of individual case safety report - http://www.fda.gov/downloads/RegulatoryInformation/Guidances/ucm129399.pdf.
[11] Vaccine European New Integrated Collaboration Effort - http://venice.cineca.org/.
[12] VENICE project. Final Report on the Survey on AEFI Monitoring Systems in Member States - http://venice.
[13] The Brighton Collaboration - http://www.brightoncollaboration.org.
[14] Bonhoeffer J et al.; Brighton Collaboration Seizure Working Group. Generalized convulsive seizure as an adverse event following immunization: case definition and guidelines for data collection, analysis, and presentation. Vac-cine, 22(5-6):557–62, 2004.
[15] Sejvar JJ et al. Brighton Collaboration GBS Working Group. Guillain-Barr´e syndrome and Fisher syndrome: case definitions and guidelines for collection, analysis, and presentation of immunization safety data. Vaccine,29(3):599–612, 2004.
[16] M. Courtot, F. Gibson, A.L. Lister, J. Malone, Schober D., Brinkman R., and Ruttenberg A. MIREOT: the minimum information to reference an external ontology term. Applied ontology, in press, 2011.
[17] The Vaccine Ontology - http://www.violinet.org/vaccineontology/.
[18] Ontology for General Medical Science (OGMS) - http://code.google.com/p/ogms/.
[19] B. Smith, W. Ceusters, B. Klagges, J. Kohler, A. Kumar, J. Lomax, C. Mungall, F. Neuhaus, A. L. Rector, and C. Rosse.
Relations in biomedical ontologies. Genome biology, 6(5):R46, 2005.
[20] The Information Artifact Ontology (IAO), http://code.google.com/p/information-artifact-ontology/.
[21] Ashburner M. Smith B., Rosse C., Bard J., Bug W., Ceusters W., Goldberg L. J., Eilbeck K., Ireland A., Mungall C. J., Leontis N. OBI Consortium, Rocca-Serra P., Ruttenberg A., Sansone S. A., Scheuermann R. H., Shah N., Whetzel P.
L., and Lewis S. The OBO foundry: coordinated evolution of ontologies to support biomedical data integration.
Nature biotechnologypplied ontology, 25(11):1251–1255, 2007.
[22] B J Stewart and P U Prabhu. Reports of sensorineural deafness after measles, mumps, and rubella immunisation.
Archives of Disease in Childhood, 69(1):153–154, 1993.
[23] Madhok R Alcorn N, Saunders S. Benefit-risk assessment of leflunomide: an appraisal of leflunomide in rheuma- toid arthritis 10 years after licensing. Drug Safety, 32(12):1123–34, 2009.
[24] Lidian L. A. Lecluse, Emmilia A. Dowlatshahi, C. E. Jacqueline M. Limpens, Menno A. de Rie, Jan D. Bos, and Phyllis I. Spuls. Etanercept: An overview of dermatologic adverse events. Arch Dermatol, 147(1):79–94, 2011.
[25] Angela A. M. C. Claessens, Eibert R. Heerdink, Jacques T. H. M. van Eijk, Cornelis B. H. W. Lamers, and Hubert G. M. Leufkens. Determinants of headache in lansoprazole users in the netherlands: Results from a nested case-control study. Drug Safety, 25(4), 2002.
[26] M. Horridge, N. Drummond, J. Goodwin, A. Rector, R. Stevens, and H.H. Wang. The manchester owl syntax. In Bernardo Cuenca Grau, Pascal Hitzler, Conor Shankey, and Evan Wallace, editors, Proceedings of OWL Experiencesand Directions Workshop (OWLED06), 2006.
[27] OBI Ontology, http://purl.obolibrary.org/obo/obi.
[28] Ceusters W, Capolupo M, de Moor G, Devlies J, and Smith B. An evolutionary approach to realism-based adverse event representations. Methods Inf Med, 50(1):62–73, 2011.

Source: http://icbo.buffalo.edu/2011/workshop/adverse-events/docs/papers/CourtotAEICBO2011_submission.pdf