Skip to main content

Software Testing of AI-based systems

School of Electronics, Electrical Engineering and Computer Science | PHD
Funding
Unfunded
Reference Number
CSC/2020/10
Application Deadline
15 March 2020
Start Date
1 October 2020

Overview

Recent advances in Artificial Intelligence (AI) have considerably accelerated its use in many software-intensive systems in recent years. Nowadays, we witness AI’s applications in almost any industry, ranging from autonomous cars to consumer electronics to banking and to city planning. Given the growing importance of AI-based systems in our daily lives, and often in case of using AI in safety-critical contexts (e.g., autonomous cars), it is important to ensure their reliability. However, despite such efforts, these systems, just like any conventional software, often demonstrate incorrect or even unsafe (dangerous) behavior. One example is several fatality cases caused by real-world accidents involving autonomous cars. As a consequence, the need for systematic and rigorous software testing and quality assurance of AI software has increased tremendously, and given its criticality, the issue is under discussion even in policy venues such as the World Economic Forum. There are fundamental differences between developing AI -based software compared to conventional software systems. Conventional software is developed deductively, by writing down the rules that govern the behavior of the system as program code. However, with AI, these rules are inferred from training data. Thus, the behavior of an AI system is heavily dependent on the training data from the external world, before and after deployment in the field. Such a mechanism could lead to non-deterministic behavior in AI software which, in turn, results in systems that are intrinsically challenging to test and verify. Although software-testing approaches are well established for conventional (non-AI-based) software, testing methods for AI systems are still in their infancy. This PhD thesis will explore novel approaches for testing AI software, and will focus on different types of AI systems, e.g., those using Machine learning (ML), Deep Learning (DL) and neural networks (NN). When applicable, existing software testing techniques, such as coverage or mutation testing, will be adapted to the AI systems context. Also, novel (new) software testing techniques for testing of AI systems will be developed. The project has the potential to engage with the work of several colleagues in EEECS (e.g., Youcheng Sun) and to strengthen existing collaborations in the UK and Europe. The PhD student will conduct all the above research steps using the action-research and empirical-research approaches. Several real-world AI systems will be considered as the Systems Under Test (SUTs) and will be used in the case studies. The supervisor has several ongoing industrial and also international collaborations on this topic, and the project is expected to be integrated into some of those collaborations. The PhD student, to be hired to work on this project, will gain both research (academic) expertise and will also gain practical experience via academia-industrial collaborations in this area.

This PhD project will include the following draft objectives (phases):
1.A systematic literature review of software testing of AI systems
2.Systematic characterization of challenges faced in software testing of AI systems
3.Exploring relationships among training and validation of AI software versus its quality assurance (QA)
4.Adaption of existing software testing techniques (such as coverage or mutation testing) to the testing of AI systems (whenever applicable)
5.Development of novel software testing techniques for testing of AI systems
6.Conducting all the above objectives using the action-research and empirical-research approach, by considering several real-world AI systems as the Systems Under Test (SUTs)
7.Contributing to both the state-of the art (academia) and also to the state-of the-practice (software industry)

Project Summary
Supervisor

Dr Vahid Garousi


Mode of Study

Full-time: 3 years


Apply now Register your interest

Computer Science overview

The School of Electronics, Electrical Engineering and Computer Science (EEECS) aims to enhance the way we use technology in communication, data science, computing systems, cyber security, power electronics, intelligent control, and many related areas.

You’ll be part of a dynamic doctoral research environment and will study alongside students from over 40 countries world wide; we supervise students undertaking research in key areas of computer science, including: computing systems, artificial intelligence and cybersecurity. As part of a lively community of over 100 full-time and part-time research students you’ll have the opportunity to develop your research potential in a vibrant research community that prioritises the cross-fertilisation of ideas and innovation in the advancement of knowledge.

Many PhD studentships attract scholarships and top-up supplements. Both the MPhil and PhD programmes provide our students with the opportunity to acquire an extensive training in research techniques.

Within the School we have a number of specialist research centres including a Global Research Institute, the Institute of Electronics, Communications and Information Technology (ECIT) specialising in Cyber Security, Wireless and Data Science, and scalable computing.

Computer Science Highlights
Industry Links
  • Queen’s researchers have strong links with the local industry, which boasts a rich mix of local startups and multi-nationals. Belfast is the second fastest growing region in the UK in terms of Knowledge Economy activity (Northern Ireland Economy Report, 2018).
World Class Facilities
  • The state-of-the-art £14m Computer Science Building and the Institute of Electronics, Communications and Information Technology offer bespoke research environments.
Internationally Renowned Experts
  • You will be working under the supervision of leading international academic experts.
Key Facts

Research students are encouraged to play a full and active role in relation to the wide range of research activities undertaken within the School and there are many resources available including:

  • A wide range of personal development and specialist training courses offered through the Personal Development Programme
  • Access to the Queen's University Postgraduate Researcher Development Programme
  • Office accommodation with access to computing facilities and support to attend conferences for full-time PhD students
Brexit Advice

Information on the implications of Brexit for prospective students.

Course content

Research Information

Associated Research
Research within the School is organised into research themes combining strengths by working together on major projects, in many cases in collaboration with key technology companies.
ECIT brings together internationally recognised research groups specialising in key areas of advanced digital and communications technology.

PhD Opportunities
PhD Opportunities are available in a wide range of computer science subjects, aligned to the specific expertise of our PhD supervisors.

Research Impact
Queen’s is a leader in commercial impact and one of the five highest performing universities in the UK for intellectual property commercialisation. We have created over 80 spin-out companies. Three of these -
Kainos, Andor Technology and Fusion Antibodies - have been publicly listed on the London Stock Exchange.

Research Projects
Queen’s has strong collaborative links with industry in Northern Ireland, and internationally. It has a strong funding track record with EPSRC and the EC H2020 programme.

Research Success
The research profile produced by the 2014 UK Research Excellence Framework (REF) graded 80 per cent of our research activity as 'world-leading' or 'internationally excellent', confirming the School's reputation as an internationally-leading department.

Career Prospects

Introduction
For further information on career opportunities at PhD level please contact the Faculty of Engineering and Physical Sciences Student Recruitment Team on askEPS@qub.ac.uk. Our advisors - in consultation with the School - will be happy to provide further information on your research area, possible career prospects and your research application.

People teaching you




Learning Outcomes

Course structure

There is no specific course content as such. You are expected to take research training modules that are supported by the School which focus on quantitative and qualitative research methods. You are also expected to carry out your research under the guidance of your supervisor.

Over the course of study you can attend postgraduate skills training organised by the Graduate School.

You will normally register, in the first instance, as an ‘undifferentiated PhD student’ which means that you have satisfied staff that you are capable of undertaking a research degree. The decision as to whether you should undertake an MPhil or a PhD is delayed until you have completed ‘differentiation’.

Differentiation takes place about 8-9 months after registration for full time students and about 16-18 months for part time students: You are normally asked to submit work to a panel of up two academics and this is followed up with a formal meeting with the ‘Differentiation Panel’. The Panel then make a judgement about your capacity to continue with your study. Sometimes students are advised to revise their research objectives or to consider submitting their work for an MPhil qualification rather than a doctoral qualification.

To complete with a doctoral qualification you will be required to submit a thesis of approx 80,000 words and you will be required to attend a viva voce [oral examination] with an external and internal examiner to defend your thesis.

A PhD programme runs for 3-4 years full-time or 6-8 years part-time. Students can apply for a writing up year should it be required.

The PhD is open to both full and part time candidates and is often a useful preparation for a career within academia or consultancy.

Full time students are often attracted to research degree programmes because they offer an opportunity to pursue in some depth an area of academic interest.

The part time research degree is an exciting option for professionals already working in the education field who are seeking to extend their knowledge on an issue of professional interest. Often part time candidates choose to research an area that is related to their professional responsibilities.

If you meet the Entry Requirements, the next step is to check whether we can supervise research in your chosen area. We only take students to whom we can offer expert research supervision from one of our academic staff. Therefore, your research question needs to engage with the research interests of one of our staff.

Assessment

- Assessment processes for the Research Degree differ from taught degrees. Students will be expected to present write up their work at regular intervals to their supervisor who will provide written and oral feedback; a formal assessment process takes place annually.

This Annual Progress Review requires students to present their work in writing and orally to a panel of academics from within the School. Successful completion of this process will allow students to register for the next academic year.

The final assessment of the doctoral degree is both oral and written. Students will submit their thesis to an internal and external examining team who will review the written thesis before inviting the student to orally defend their work at a Viva Voce.

Feedback

- Supervisors will offer feedback on the research work at regular intervals throughout the period of registration on the degree.

Facilities
Full time PhD students will have access to a shared office space and access to a desk with personal computer and internet access.

Entrance requirements

Graduate
The minimum academic requirement for admission to a research degree programme is normally an Upper Second Class Honours degree from a UK or ROI HE provider, or an equivalent qualification acceptable to the University. Further information can be obtained by contacting the School.

International Students

For information on international qualification equivalents, please check the specific information for your country.

English Language Requirements

Evidence of an IELTS* score of 6.0, with not less than 5.5 in any component or equivalent qualification acceptable to the University is required (*taken within the last 2 years).

International students wishing to apply to Queen's University Belfast (and for whom English is not their first language), must be able to demonstrate their proficiency in English in order to benefit fully from their course of study or research. Non-EEA nationals must also satisfy UK Visas and Immigration (UKVI) immigration requirements for English language for visa purposes.

For more information on English Language requirements for EEA and non-EEA nationals see: www.qub.ac.uk/EnglishLanguageReqs.

If you need to improve your English language skills before you enter this degree programme, INTO Queen's University Belfast offers a range of English language courses. These intensive and flexible courses are designed to improve your English ability for admission to this degree.

  • Academic English: an intensive English language and study skills course for successful university study at degree level
  • Pre-sessional English: a short intensive academic English course for students starting a degree programme at Queen's University Belfast and who need to improve their English.

Tuition Fees

Northern Ireland (NI) £4,407
England, Scotland or Wales (GB) £4,407
Other (non-UK) EU £4,407
International £21,300

More information on postgraduate tuition fees.

Computer Science costs

There are no specific additional course costs associated with this programme.

Additional course costs

All Students

Depending on the programme of study, there may also be other extra costs which are not covered by tuition fees, which students will need to consider when planning their studies . Students can borrow books and access online learning resources from any Queen's library. If students wish to purchase recommended texts, rather than borrow them from the University Library, prices per text can range from £30 to £100. Students should also budget between £30 to £100 per year for photocopying, memory sticks and printing charges. Students may wish to consider purchasing an electronic device; costs will vary depending on the specification of the model chosen. There are also additional charges for graduation ceremonies, and library fines. In undertaking a research project students may incur costs associated with transport and/or materials, and there will also be additional costs for printing and binding the thesis. There may also be individually tailored research project expenses and students should consult directly with the School for further information.

How do I fund my study?
1.PhD Opportunities

Find PhD opportunities and funded studentships by subject area.

2.Doctoral Training Centres at Queen's

Queen's has eight outstanding competitive Doctoral Training Centres, with each one providing funding for a number of PhD positions and most importantly a hub for carrying out world class research in key disciplines.

3.PhD loans

The Government offers doctoral loans of up to £26,445 for PhDs and equivalent postgraduate research programmes for English- or Welsh-resident UK and EU students, £10,000 for students in Scotland and up to £5,500 for Northern Ireland students.

4.International Scholarships

Information on Postgraduate Research scholarships for international students.

Funding and Scholarships

The Funding & Scholarship Finder helps prospective and current students find funding to help cover costs towards a whole range of study related expenses.

How to Apply

Apply using our online Postgraduate Applications Portal go.qub.ac.uk/pgapply and follow the step-by-step instructions on how to apply.

Find a supervisor

If you're interested in a particular project, we suggest you contact the relevant academic before you apply, to introduce yourself and ask questions.

To find a potential supervisor aligned with your area of interest, or if you are unsure of who to contact, look through the staff profiles linked here.

You might be asked to provide a short outline of your proposal to help us identify potential supervisors.