Send your request Join Sii

Lately there is a lot of talk on using the artificial intelligence in many fields. Recently social media were flooded with a wave of graphics generated by AI (Artificial Intelligence) – among others by the Lens application. Therefore, it is an interesting topic to explore in the context of using this technology for testing.

In the article I will be covering the usage of the AI in testing, concentrating on the ChatGPT tool. In the text you will find a couple questions and issues related to testing and I will present how ChatGPT reacted to them. I will check the AI responses concerning i.e.:

  • Creating test cases,
  • Automated testing,
  • Accessibility.

What is ChatGPT?

Let’s ask ChatGPT itself what it is and what sources it uses to give responses:

What is ChatGPT and what sources it uses – AI response
Fig. 1 What is ChatGPT and what sources it uses – AI response

Applying AI in testing

Test cases

Let’s begin with the issue, which most of the testers encounter at the beginning of their career path. Let’s see how ChatGPT deals with two questions regarding the creation of test cases. I asked ChatGPT about logging to a webpage and a mobile app test cases. You can see the responses below.

ChatGPT response to the question regarding creating a test case for logging to a webpage
Fig. 2 ChatGPT response to the question regarding creating a test case for logging to a webpage
ChatGPT response to the question regarding creating a test case for mobile app
Fig. 3 ChatGPT response to the question regarding creating a test case for mobile ap

Currently I am working on a mobile app project in a medical market. That is why I have added more details to the query above.

Here is the result:

Odpowiedź ChatGPT na pytanie dotyczące tworzenia przypadku aplikacji mobilnej usług medycznych
Fig. 4 ChatGPT response to the question regarding creating a test case for medical services mobile app

ChatGPT’s response was surprising to me, because – apart from a couple differences – I could use this test case in the project.

Questions on automated tests

We will now look at how AI will deal with a harder subject regarding automated tests.

Let’s begin with something easier, so a query on how to open a webpage using Selenium.

Odpowiedź ChatGPT na pytanie dotyczące otwarcia strony www w Selenium
Fig. 5 ChatGPT response to the question regarding opening a webpage in Selenium

The code above can be considered correct for the Python language.

Example of Selenium documentation
Fig. 6 Example of Selenium documentation

We did not specify the name of any programming language in our query. Let’s see what happens when we specify it and indicate a specific language.

Odpowiedź ChatGPT na pytanie dotyczące otwarcia strony www w Selenium w Java
Fig. 7 ChatGPT response to the question regarding opening a webpage in Selenium in Java

I have checked if the above code works. Below is the result:

Przykład uruchomienia kodu wygenerowanego przez ChatGPT
Fig. 8 Example of running the code generated by ChatGPT

I did not have any problem with running the above code, and the outcome was as expected, which is Sii website has been properly opened in the browser. 

Of course, we can continue to improve our query. Below I have modified it, so that ChatGPT tries to verify if the webpage title is correct.

Odpowiedź ChatGPT na pytanie dotyczące otwarcia strony www w Selenium i sprawdzenia tytułu strony
Fig. 9 ChatGPT response to the question regarding opening a webpage in Selenium and verifying the webpage title

Naturally, in real life we should not use such code – instead use an assertion. Therefore, now will ask ChatGPT for an assertion.

ChatGPT response to the question regarding opening a webpage in Selenium and verifying the webpage title with the use of an assertion
Fig. 10 ChatGPT response to the question regarding opening a webpage in Selenium and verifying the webpage title with the use of an assertion

In the examples above ChatGPT has no problem with indicating libraries, which should be imported for the code to work properly. Still, I was surprised with the fact, that the tool has also managed to prepare the code with the inclusion of assertion in TestNG.

Question on Robot Framework

Unfortunately, ChatGPT not always generates response in such a way. In case of a query regarding Robot Framework, the tool did not indicate the necessary library to import.

Odpowiedź ChatGPT na pytanie dotyczące otwarcia strony www w Robot Framework
Fig. 11 ChatGPT response to the question regarding opening a webpage in Robot Framework

In this case the SeleniumLibrary should be imported.

Questions on Selenium, Cypress and Playwright

In case of Selenium the most frequently asked question concerns the differences between implicit wait and explicit wait. Therefore, it is worth checking if artificial intelligence will handle that issue.

ChatGPT response to the question regarding the difference between implicit wait and explicit wait
Fig. 12 ChatGPT response to the question regarding the difference between implicit wait and explicit wait

I leave the evaluation of this response to you 😉 An excerpt from the documentation on this issue may help you with it.

ChatGPT is able to respond to even more complicated queries:

ChatGPT response to the question regarding opening a webpage in Selenium with the use f PageObject Pattern
Fig. 13 ChatGPT response to the question regarding opening a webpage in Selenium with the use f PageObject Pattern

A tester does not live by Selenium alone 😉 Nowadays, Cypress is also frequently used and Playwright framework is gaining popularity. Can ChatGPT be helpful in this case?

In case of Cypress the code generated by the tool was working and performed the test correctly:

describe('www.sii.pl', () => {
  it('should open the homepage', () => {
    cy.visit('https://www.sii.pl');

    // Assert that the page has loaded successfully
    cy.title().should('eq', 'Strona główna - SII');
  });
});

Example:

Fig. 14 Example of running the code, which ChatGPT generated for Cypress
Fig. 14 Example of running the code, which ChatGPT generated for Cypress

Unfortunately, the situation was not that colorful in case of Playwright. ChatGPT prepared the following code for me:

const { chromium } = require('playwright');

describe('www.sii.pl', () => {
  it('should open the homepage', async () => {
    const browser = await chromium.launch();
    const page = await browser.newPage();
    await page.goto('https://www.sii.pl');

    // Assert that the page has loaded successfully
    expect(await page.title()).toEqual('Strona główna - SII');

    await browser. Close();
  });
});

Taking into consideration the format, which ChatGPT has adopted, in my opinion the code should look like this:

import { test, expect } from '@playwright/test'

test.describe.only('www.sii.pl', () => {
  test('should open the homepage', async ({ page }) => {
    await page.goto('https://www.sii.pl');

    // Assert that the page has loaded successfully
    expect(await page.title()).toEqual('Strona główna - SII');
  })
})

Of course, in my research I have also checked other issues concerning automated tests and programming, including queries regarding:

  • Appium,
  • HTML,
  • JavaScript.

ChatGPT and the issue of accessibility

For some time now I have been very interested in the topic of the accessibility (you can learn more on that issue from the articles I have written in PL: part I and part II). The WCAG standards are connected with that topic.

ChatGPT response to the question regarding accessibility in accordance with WCAG standards
Fig 15. ChatGPT response to the question regarding accessibility in accordance with WCAG standards

ChatGPT responds in different languages. For me, the response in Polish was surprising. I also think, that creating web content accessible also for infants is very hard 😉

ChatGPT response which recommends creating the content in accordance with WCAG standard, accessible i.e. for infants
Fig. 16 ChatGPT response which recommends creating the content in accordance with WCAG standard, accessible i.e. for infants (PL)

Of course, also in the case of A11Y (accessibility) we can ask more specific questions.

ChatGPT response to the question regarding adding an aria label in HTML
Fig. 17 ChatGPT response to the question regarding adding an aria label in HTML

It should be mentioned here that ChatGPT does not always work as expected. In the example above, I asked the question in English and the answer was given in Polish.

Controversies concerning AI

The main goal of the article was to present AI possibilities in the context of testing, however I think that the controversies concerning artificial intelligence are worth mentioning.

Copyrights

Media and authors report, that AI very often uses other people’s work. Midjourney creator said to the Forbes magazine:

There isn’t really a way to get a hundred million images and know where they’re coming from. It would be cool if images had metadata embedded in them about the copyright owner or something. But that’s not a thing; there’s not a registry. There’s no way to find a picture on the Internet, and then automatically trace it to an owner and then have any way of doing anything to authenticate it.

From these words, in my opinion, confirmation of the use of other people’s works by artificial intelligence can be deducted. It has a particular significance in the context of the yet another case of right infringement of millions of creators, who decided to file a class action lawsuit.

Education and critical thinking

Schools in New York have restricted access to ChatGPT. Below are the words of Jenny Lyle, spokesperson for the U.S. Department of Education:

Due to concerns about negative impacts on student learning, and concerns regarding the safety and accuracy of content, access to ChatGPT is restricted on New York City Public Schools’ networks and devices.”

While the tool may be able to provide quick and easy answers to questions, it does not build critical-thinking and problem-solving skills, which are essential for academic and lifelong success.

I do agree with these words. I live in a city without access to any sea or lake, and ChatGPT tries to convince me, that there are beautiful beaches in Reda.

Odpowiedź ChatGPT na pytanie o miejscowość Reda
Fig. 18 ChatGPT response to the question about Reda town

Can AI be relied upon?

Questions arise:

  • Can AI replace a human being?
  • Can responses generated by these tools be trusted?

Programs using artificial intelligence still often give wrong answers or provide outdated information.

Above, I have presented and example of the code, that needed to be corrected in order to work, as well as a response, in which ChatGPT proposed ensuring website accessibility to infants.

In my history of using the tool I have also probably encountered a situation of providing outdated information. I asked about differences between Cypress and Playwright. The response was as follows:

Playwright supports a wider range of browsers than Cypress, including Chromium (used by Google Chrome), Firefox, and WebKit (used by Safari). Cypress currently only supports Chromium.

I decided to look into that response and verify it. On the official website, Cypress informs about the support of many browsers and engines, i.e.: Chrome, Firefox, Edge, Webkit. In this case, we may be dealing with a ChatGPT limitation. ChatGPT contains information until 2021.

ChatGPT limitations
Fig. 19 ChatGPT limitations

Therefore, it is a confirmation, that a human and their control are essential.

Other IT areas are also considering the possibilities of using AI in their work. On YouTube you can find materials on the use of artificial intelligence in the process of creating a website or application. It is worth looking into what they say.

Summary

In my opinion artificial intelligence can only be used as a support in daily work of, for example a tester, and we should not rely on it immensely. Data obtained through AI should be verified and approached with great caution. I am a big opponent of creating content based on ChatGPT responses on a copy/paste basis.

An open issue is still the question of ethics and copyrights. To me it seems that it is something we have to face in the next few years. In relation to the above doubts, I believe that AI cannot replace a human being.

***

Artificial intelligence subject has also been covered in other articles by our experts – we encourage you to read them.

5/5 ( votes: 26)
Rating:
5/5 ( votes: 26)
Author
Avatar
Łukasz Krause

Working in Sii as a Test and Analysis Engineer. Interested in computers and technological innovations. Enjoys testing new devices and apps, so he decided to pursue a postgraduate degree in testing. Trying to master the secrets off automation, and also lose weight, although the first point will be probably achieved faster 😉 Privately, a father of a cat, who likes low-budget travels 😉

Leave a comment

Your email address will not be published. Required fields are marked *

You might also like

More articles

Don't miss out

Subscribe to our blog and receive information about the latest posts.

Get an offer

If you have any questions or would like to learn more about our offer, feel free to contact us.

Send your request Send your request

Natalia Competency Center Director

Get an offer

Join Sii

Find the job that's right for you. Check out open positions and apply.

Apply Apply

Paweł Process Owner

Join Sii

SUBMIT

Ta treść jest dostępna tylko w jednej wersji językowej.
Nastąpi przekierowanie do strony głównej.

Czy chcesz opuścić tę stronę?