Feat/test list #48

Ma11hewThomas · 2024-11-05T18:09:08Z

No description provided.

github-actions · 2024-11-05T18:09:32Z

Suite folded

Test Suite Summary

5 passed, 3 failed, and 3 skipped/pending/other

Test Suite	Passed	Failed	Skipped	Duration
✅ login.test.ts > login ✅ should be able to login ⏭️ should be able to logout	✅ 1		⏭️ 1	1.2s
❌ login.test.ts > profile ❌ should display title ✅ should be able to update profile	✅ 1	❌ 1		2.0s
✅ settings.test.ts > settings ✅ should validate user settings	✅ 1			1.1s
❌ network.test.ts > network ❌ should fail to update profile on network failure ❌ should fail to update profile on network failure ✅ should handle session timeouts ❓ should clean up user session on logout ✅ should allow user to change password	✅ 2	❌ 2	❓ 1	5.1s
✅ Unknown Suite ⏳ should load user data			⏳ 1	1ms

Github Test Reporter CTRF

github-actions · 2024-11-05T18:09:38Z

With PR Comment - Run #70

Tests 📝	Passed ✅	Failed ❌	Skipped ⏭️	Pending ⏳	Other ❓	Flaky 🍂	Duration ⏱️
10	5	3	1	1	1	3	11.0s

❌ Some tests failed!

Name	Status	Failure Message
should display title	failed ❌	Timed out 5000ms waiting for expect(locator).toHaveTitle(expected) Locator: locator(':root') Expected pattern: /Playwrc cight/ Received string: "Fast and reliable end-to-end testing for modern web apps \| Playwright" Call log: expect.toHaveTitle with timeout 5000ms waiting for locator(':root') locator resolved to <html lang="en" dir="ltr" data-theme="light" data-has-…>… unexpected value "Fast and reliable end-to-end testing for modern web apps \| Playwright"
should fail to update profile on network failure	failed ❌	Network Timeout
should fail to update profile on network failure	failed ❌	No failure message

Github Test Reporter CTRF

github-actions · 2024-11-05T18:09:38Z

With PR Comment No Fails - Run #70

Tests 📝	Passed ✅	Failed ❌	Skipped ⏭️	Pending ⏳	Other ❓	Flaky 🍂	Duration ⏱️
10	10	0	0	0	0	0	11.0s

🎉 All tests passed!

Github Test Reporter CTRF

github-actions · 2024-11-05T18:09:39Z

With PR Comment On Fail Only With Fails - Run #70

Tests 📝	Passed ✅	Failed ❌	Skipped ⏭️	Pending ⏳	Other ❓	Flaky 🍂	Duration ⏱️
10	5	3	1	1	1	3	11.0s

❌ Some tests failed!

Name	Status	Failure Message
should display title	failed ❌	Timed out 5000ms waiting for expect(locator).toHaveTitle(expected) Locator: locator(':root') Expected pattern: /Playwrc cight/ Received string: "Fast and reliable end-to-end testing for modern web apps \| Playwright" Call log: expect.toHaveTitle with timeout 5000ms waiting for locator(':root') locator resolved to <html lang="en" dir="ltr" data-theme="light" data-has-…>… unexpected value "Fast and reliable end-to-end testing for modern web apps \| Playwright"
should fail to update profile on network failure	failed ❌	Network Timeout
should fail to update profile on network failure	failed ❌	No failure message

Github Test Reporter CTRF

github-actions · 2024-11-05T18:09:40Z

Create a Custom Summary!

You can effortlessly generate custom test summaries and PR comments using your CTRF report, Handlebars and Markdown.

Example: Generating a Summary

Let's start with the basics, a summary table that gives an overview of your test ctrf (you might have seen this one somewhere!). This table is straightforward yet powerful, helping you quickly communicate the state of your tests:

Tests 📝	Passed ✅	Failed ❌	Skipped ⏭️	Pending ⏳	Other ❓	Flaky 🍂	Duration ⏱️
10	5	3	1	1	1	3	00:00:11

This table is generated by referencing various properties from your CTRF report, making it easy to create detailed and customized summaries.

In addition, you can leverage custom methods like countFlaky and formatDuration to enhance your summary:

Count of flaky tests: 3
Duration of execution: 00:00:11

Remember, you're not just writing markdown—you're writing markdown enhanced with Handlebars, which means you can take full advantage of markdown's features while adding dynamic content.

Example: Iterating Over the Tests Array

Next, let's focus on generating a report specifically for failed tests. To do this, we'll iterate through the CTRF tests array and filter out the failed tests:

Name	Status	Failure Message
should display title	failed ❌	�[31mTimed out 5000ms waiting for �[39m�[2mexpect(�[22m�[31mlocator�[39m�[2m).�[22mtoHaveTitle�[2m(�[22m�[32mexpected�[39m�[2m)�[22m

Locator: locator(':root')
Expected pattern: �[32m/Playwrc cight/�[39m
Received string: �[31m"Fast and reliable end-to-end testing for modern web apps | Playwright"�[39m
Call log:
�[2m- expect.toHaveTitle with timeout 5000ms�[22m
�[2m - waiting for locator(':root')�[22m
�[2m - locator resolved to <html lang="en" dir="ltr" data-theme="light" data-has-…>…</html>�[22m
�[2m - unexpected value "Fast and reliable end-to-end testing for modern web apps | Playwright"�[22m
|
| should fail to update profile on network failure | failed ❌ | Network Timeout |
| should fail to update profile on network failure | failed ❌ | No failure message |

You might have noticed, this one is using a helper called eq. You can use this for property comparisons directly within your template logic.

HTML

If raw markdown isn't quite working for you, GitHub also renders some HTML.

Here is the same failed table in HTML:

Name	Status	Failure Message
should display title	failed ❌	Timed out 5000ms waiting for expect(locator).toHaveTitle(expected) Locator: locator(':root') Expected pattern: /Playwrc cight/ Received string: "Fast and reliable end-to-end testing for modern web apps \| Playwright" Call log: expect.toHaveTitle with timeout 5000ms waiting for locator(':root') locator resolved to <html lang="en" dir="ltr" data-theme="light" data-has-…>…</html> unexpected value "Fast and reliable end-to-end testing for modern web apps \| Playwright"
should fail to update profile on network failure	failed ❌	Network Timeout
should fail to update profile on network failure	failed ❌	No failure message

Notice the use of the stripAnsi method, as some error messages contain ANSI.

Adding Extra Markdown Elements

You can further enrich your summaries by tagging collaborators, adding tasks, and linking to important resources. Here's how:

Tag a collaborator:

You might want to notify someone specific, like @Ma11hewThomas.

You can add tasks!

Passed all unit tests
Review code quality
Update documentation

Include useful links:

GitHub Actions Documentation

Adding Collapsible Sections

If you have extensive details that might clutter your summary, you can use collapsible sections to hide and reveal information as needed:

See detailed test results

Name	Status	Failure Message
should be able to login	passed	No failure message
should display title	failed	Timed out 5000ms waiting for expect(locator).toHaveTitle(expected)

Locator: locator(':root')
Expected pattern: /Playwrc cight/
Received string: "Fast and reliable end-to-end testing for modern web apps | Playwright"
Call log:

expect.toHaveTitle with timeout 5000ms
waiting for locator(':root')
locator resolved to <html lang="en" dir="ltr" data-theme="light" data-has-…>…</html>
unexpected value "Fast and reliable end-to-end testing for modern web apps | Playwright"
|
| should be able to update profile | passed | No failure message |
| should be able to logout | skipped | No failure message |
| should validate user settings | passed | No failure message |
| should fail to update profile on network failure | failed | Network Timeout |
| should fail to update profile on network failure | failed | No failure message |
| should load user data | pending | No failure message |
| should handle session timeouts | passed | No failure message |
| should clean up user session on logout | other | No failure message |
| should allow user to change password | passed | No failure message |

GitHub properties

There are various GitHub properties you can tap into!

Repository name: ctrf-io/github-test-reporter
Branch name: 48/merge
Run number: 70
Job name: testing
Workflow ID:
Workflow name: pull-request
Actor name: Ma11hewThomas
Event name: pull_request
Run ID: 11690040969
Pull request number: 48
API URL: https://api.github.com
Base URL: https://github.com
Build URL: https://github.com/ctrf-io/github-test-reporter/actions/runs/11690040969#summary

github-actions · 2024-11-05T18:09:41Z

Pull Request Arg

Tests 📝	Passed ✅	Failed ❌	Skipped ⏭️	Pending ⏳	Other ❓	Flaky 🍂	Duration ⏱️
10	5	3	1	1	1	3	11.0s

Github Test Reporter CTRF

Failed Tests

Name	Status	Failure Message
should display title	failed ❌	Timed out 5000ms waiting for expect(locator).toHaveTitle(expected) Locator: locator(':root') Expected pattern: /Playwrc cight/ Received string: "Fast and reliable end-to-end testing for modern web apps \| Playwright" Call log: expect.toHaveTitle with timeout 5000ms waiting for locator(':root') locator resolved to <html lang="en" dir="ltr" data-theme="light" data-has-…>… unexpected value "Fast and reliable end-to-end testing for modern web apps \| Playwright"
should fail to update profile on network failure	failed ❌	Network Timeout
should fail to update profile on network failure	failed ❌	No failure message

Github Test Reporter CTRF

Flaky Tests

Name	Status	Retries	Flaky
should be able to update profile	passed ✅	2	🍂
should handle session timeouts	passed ✅	1	🍂
should allow user to change password	passed ✅	3	🍂

Github Test Reporter CTRF

Detailed Test Results

Name	Status	Duration	Flaky
should be able to login	passed ✅	1.2s
should display title	failed ❌	800ms
should be able to update profile	passed ✅	1.2s	🍂
should be able to logout	skipped ⏭️	1ms
should validate user settings	passed ✅	1.1s
should fail to update profile on network failure	failed ❌	900ms
should fail to update profile on network failure	failed ❌	900ms
should load user data	pending ⏳	1ms
should handle session timeouts	passed ✅	950ms	🍂
should clean up user session on logout	other ❓	1.1s
should allow user to change password	passed ✅	1.3s	🍂

Github Test Reporter CTRF

github-actions · 2024-11-05T18:15:05Z

Suite folded

Test Suite Summary

5 passed, 3 failed, and 3 skipped/pending/other

Test Suite	Passed	Failed	Skipped	Duration
✅ login.test.ts > login ✅ should be able to login ⏭️ should be able to logout	✅ 1		⏭️ 1	1.2s
❌ login.test.ts > profile ❌ should display title ✅ should be able to update profile	✅ 1	❌ 1		2.0s
✅ settings.test.ts > settings ✅ should validate user settings	✅ 1			1.1s
❌ network.test.ts > network ❌ should fail to update profile on network failure ❌ should fail to update profile on network failure ✅ should handle session timeouts ❓ should clean up user session on logout ✅ should allow user to change password	✅ 2	❌ 2	❓ 1	5.1s
✅ Unknown Suite ⏳ should load user data			⏳ 1	1ms

Github Test Reporter CTRF

github-actions · 2024-11-05T18:15:12Z

With PR Comment - Run #72

Tests 📝	Passed ✅	Failed ❌	Skipped ⏭️	Pending ⏳	Other ❓	Flaky 🍂	Duration ⏱️
10	5	3	1	1	1	3	11.0s

❌ Some tests failed!

Name	Status	Failure Message
should display title	failed ❌	Timed out 5000ms waiting for expect(locator).toHaveTitle(expected) Locator: locator(':root') Expected pattern: /Playwrc cight/ Received string: "Fast and reliable end-to-end testing for modern web apps \| Playwright" Call log: expect.toHaveTitle with timeout 5000ms waiting for locator(':root') locator resolved to <html lang="en" dir="ltr" data-theme="light" data-has-…>… unexpected value "Fast and reliable end-to-end testing for modern web apps \| Playwright"
should fail to update profile on network failure	failed ❌	Network Timeout
should fail to update profile on network failure	failed ❌	No failure message

Github Test Reporter CTRF

github-actions · 2024-11-05T18:15:13Z

With PR Comment No Fails - Run #72

Tests 📝	Passed ✅	Failed ❌	Skipped ⏭️	Pending ⏳	Other ❓	Flaky 🍂	Duration ⏱️
10	10	0	0	0	0	0	11.0s

🎉 All tests passed!

Github Test Reporter CTRF

github-actions · 2024-11-05T18:15:14Z

With PR Comment On Fail Only With Fails - Run #72

Tests 📝	Passed ✅	Failed ❌	Skipped ⏭️	Pending ⏳	Other ❓	Flaky 🍂	Duration ⏱️
10	5	3	1	1	1	3	11.0s

❌ Some tests failed!

Name	Status	Failure Message
should display title	failed ❌	Timed out 5000ms waiting for expect(locator).toHaveTitle(expected) Locator: locator(':root') Expected pattern: /Playwrc cight/ Received string: "Fast and reliable end-to-end testing for modern web apps \| Playwright" Call log: expect.toHaveTitle with timeout 5000ms waiting for locator(':root') locator resolved to <html lang="en" dir="ltr" data-theme="light" data-has-…>… unexpected value "Fast and reliable end-to-end testing for modern web apps \| Playwright"
should fail to update profile on network failure	failed ❌	Network Timeout
should fail to update profile on network failure	failed ❌	No failure message

Github Test Reporter CTRF

github-actions · 2024-11-05T18:15:15Z

Create a Custom Summary!

You can effortlessly generate custom test summaries and PR comments using your CTRF report, Handlebars and Markdown.

Example: Generating a Summary

Let's start with the basics, a summary table that gives an overview of your test ctrf (you might have seen this one somewhere!). This table is straightforward yet powerful, helping you quickly communicate the state of your tests:

Tests 📝	Passed ✅	Failed ❌	Skipped ⏭️	Pending ⏳	Other ❓	Flaky 🍂	Duration ⏱️
10	5	3	1	1	1	3	00:00:11

This table is generated by referencing various properties from your CTRF report, making it easy to create detailed and customized summaries.

In addition, you can leverage custom methods like countFlaky and formatDuration to enhance your summary:

Count of flaky tests: 3
Duration of execution: 00:00:11

Remember, you're not just writing markdown—you're writing markdown enhanced with Handlebars, which means you can take full advantage of markdown's features while adding dynamic content.

Example: Iterating Over the Tests Array

Next, let's focus on generating a report specifically for failed tests. To do this, we'll iterate through the CTRF tests array and filter out the failed tests:

Name	Status	Failure Message
should display title	failed ❌	�[31mTimed out 5000ms waiting for �[39m�[2mexpect(�[22m�[31mlocator�[39m�[2m).�[22mtoHaveTitle�[2m(�[22m�[32mexpected�[39m�[2m)�[22m

Locator: locator(':root')
Expected pattern: �[32m/Playwrc cight/�[39m
Received string: �[31m"Fast and reliable end-to-end testing for modern web apps | Playwright"�[39m
Call log:
�[2m- expect.toHaveTitle with timeout 5000ms�[22m
�[2m - waiting for locator(':root')�[22m
�[2m - locator resolved to <html lang="en" dir="ltr" data-theme="light" data-has-…>…</html>�[22m
�[2m - unexpected value "Fast and reliable end-to-end testing for modern web apps | Playwright"�[22m
|
| should fail to update profile on network failure | failed ❌ | Network Timeout |
| should fail to update profile on network failure | failed ❌ | No failure message |

You might have noticed, this one is using a helper called eq. You can use this for property comparisons directly within your template logic.

HTML

If raw markdown isn't quite working for you, GitHub also renders some HTML.

Here is the same failed table in HTML:

Name	Status	Failure Message
should display title	failed ❌	Timed out 5000ms waiting for expect(locator).toHaveTitle(expected) Locator: locator(':root') Expected pattern: /Playwrc cight/ Received string: "Fast and reliable end-to-end testing for modern web apps \| Playwright" Call log: expect.toHaveTitle with timeout 5000ms waiting for locator(':root') locator resolved to <html lang="en" dir="ltr" data-theme="light" data-has-…>…</html> unexpected value "Fast and reliable end-to-end testing for modern web apps \| Playwright"
should fail to update profile on network failure	failed ❌	Network Timeout
should fail to update profile on network failure	failed ❌	No failure message

Notice the use of the stripAnsi method, as some error messages contain ANSI.

Adding Extra Markdown Elements

You can further enrich your summaries by tagging collaborators, adding tasks, and linking to important resources. Here's how:

Tag a collaborator:

You might want to notify someone specific, like @Ma11hewThomas.

You can add tasks!

Passed all unit tests
Review code quality
Update documentation

Include useful links:

GitHub Actions Documentation

Adding Collapsible Sections

If you have extensive details that might clutter your summary, you can use collapsible sections to hide and reveal information as needed:

See detailed test results

Name	Status	Failure Message
should be able to login	passed	No failure message
should display title	failed	Timed out 5000ms waiting for expect(locator).toHaveTitle(expected)

Locator: locator(':root')
Expected pattern: /Playwrc cight/
Received string: "Fast and reliable end-to-end testing for modern web apps | Playwright"
Call log:

expect.toHaveTitle with timeout 5000ms
waiting for locator(':root')
locator resolved to <html lang="en" dir="ltr" data-theme="light" data-has-…>…</html>
unexpected value "Fast and reliable end-to-end testing for modern web apps | Playwright"
|
| should be able to update profile | passed | No failure message |
| should be able to logout | skipped | No failure message |
| should validate user settings | passed | No failure message |
| should fail to update profile on network failure | failed | Network Timeout |
| should fail to update profile on network failure | failed | No failure message |
| should load user data | pending | No failure message |
| should handle session timeouts | passed | No failure message |
| should clean up user session on logout | other | No failure message |
| should allow user to change password | passed | No failure message |

GitHub properties

There are various GitHub properties you can tap into!

Repository name: ctrf-io/github-test-reporter
Branch name: 48/merge
Run number: 72
Job name: testing
Workflow ID:
Workflow name: pull-request
Actor name: Ma11hewThomas
Event name: pull_request
Run ID: 11690123356
Pull request number: 48
API URL: https://api.github.com
Base URL: https://github.com
Build URL: https://github.com/ctrf-io/github-test-reporter/actions/runs/11690123356#summary

github-actions · 2024-11-05T18:15:15Z

Pull Request Arg

Tests 📝	Passed ✅	Failed ❌	Skipped ⏭️	Pending ⏳	Other ❓	Flaky 🍂	Duration ⏱️
10	5	3	1	1	1	3	11.0s

Github Test Reporter CTRF

Failed Tests

Name	Status	Failure Message
should display title	failed ❌	Timed out 5000ms waiting for expect(locator).toHaveTitle(expected) Locator: locator(':root') Expected pattern: /Playwrc cight/ Received string: "Fast and reliable end-to-end testing for modern web apps \| Playwright" Call log: expect.toHaveTitle with timeout 5000ms waiting for locator(':root') locator resolved to <html lang="en" dir="ltr" data-theme="light" data-has-…>… unexpected value "Fast and reliable end-to-end testing for modern web apps \| Playwright"
should fail to update profile on network failure	failed ❌	Network Timeout
should fail to update profile on network failure	failed ❌	No failure message

Github Test Reporter CTRF

Flaky Tests

Name	Status	Retries	Flaky
should be able to update profile	passed ✅	2	🍂
should handle session timeouts	passed ✅	1	🍂
should allow user to change password	passed ✅	3	🍂

Github Test Reporter CTRF

Detailed Test Results

Name	Status	Duration	Flaky
should be able to login	passed ✅	1.2s
should display title	failed ❌	800ms
should be able to update profile	passed ✅	1.2s	🍂
should be able to logout	skipped ⏭️	1ms
should validate user settings	passed ✅	1.1s
should fail to update profile on network failure	failed ❌	900ms
should fail to update profile on network failure	failed ❌	900ms
should load user data	pending ⏳	1ms
should handle session timeouts	passed ✅	950ms	🍂
should clean up user session on logout	other ❓	1.1s
should allow user to change password	passed ✅	1.3s	🍂

Github Test Reporter CTRF

Ma11hewThomas added 2 commits November 5, 2024 18:04

feat: add test list view

d3b975f

chore: flaky rate

8f0a2e5

docs: test list docs - version

e91bfc9

Ma11hewThomas merged commit 619df16 into main Nov 5, 2024
20 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feat/test list #48

Feat/test list #48

Ma11hewThomas commented Nov 5, 2024

github-actions bot commented Nov 5, 2024

github-actions bot commented Nov 5, 2024

github-actions bot commented Nov 5, 2024

github-actions bot commented Nov 5, 2024

github-actions bot commented Nov 5, 2024

github-actions bot commented Nov 5, 2024

github-actions bot commented Nov 5, 2024

github-actions bot commented Nov 5, 2024

github-actions bot commented Nov 5, 2024

github-actions bot commented Nov 5, 2024

github-actions bot commented Nov 5, 2024

github-actions bot commented Nov 5, 2024

Feat/test list #48

Feat/test list #48

Conversation

Ma11hewThomas commented Nov 5, 2024

github-actions bot commented Nov 5, 2024

Suite folded

Test Suite Summary

github-actions bot commented Nov 5, 2024

With PR Comment - Run #70

❌ Some tests failed!

github-actions bot commented Nov 5, 2024

With PR Comment No Fails - Run #70

🎉 All tests passed!

github-actions bot commented Nov 5, 2024

With PR Comment On Fail Only With Fails - Run #70

❌ Some tests failed!

github-actions bot commented Nov 5, 2024

Create a Custom Summary!

Example: Generating a Summary

Example: Iterating Over the Tests Array

HTML

Adding Extra Markdown Elements

Tag a collaborator:

You can add tasks!

Include useful links:

Adding Collapsible Sections

GitHub properties

github-actions bot commented Nov 5, 2024

Pull Request Arg

Failed Tests

Flaky Tests

Detailed Test Results

github-actions bot commented Nov 5, 2024

Suite folded

Test Suite Summary

github-actions bot commented Nov 5, 2024

With PR Comment - Run #72

❌ Some tests failed!

github-actions bot commented Nov 5, 2024

With PR Comment No Fails - Run #72

🎉 All tests passed!

github-actions bot commented Nov 5, 2024

With PR Comment On Fail Only With Fails - Run #72

❌ Some tests failed!

github-actions bot commented Nov 5, 2024

Create a Custom Summary!

Example: Generating a Summary

Example: Iterating Over the Tests Array

HTML

Adding Extra Markdown Elements

Tag a collaborator:

You can add tasks!

Include useful links:

Adding Collapsible Sections

GitHub properties

github-actions bot commented Nov 5, 2024

Pull Request Arg

Failed Tests

Flaky Tests

Detailed Test Results