Unit Test (Stability (Moving Flaky Test cases to DEBUG mode (:red_flag:…
Moving Flaky Test cases to DEBUG mode
:red_flag:: How TLs ensure engineers verify in Debug mode
:red_flag: How to handle the real issues in the production if missed in DEBUG mode.
:red_flag: How to ensure engineers executed full suite in Debug mode?
:question: Automate to run the DEBUG mode for each component?
:question: Expect Engineers to run all the test in DEBUG for each check-in
:question: Expect Build team to run full product build in DEBUG mode once every week. Re-run the failure to identify the real issue
:red_flag: How to identify if the model fix breaks moved downstream test case before reaching to trunk?
treat it like ACID / SAT tests - go as bugs onto a team backlog
Current Unit Test bed provides an illusion of quality - do we actually see value in it?
Time vs bug finds?
Time to run in 'branch' builds
time for every engineer to run before check in
bugs it catches in that process?
Do we think less about the testing of code because we 'think' a test will already cover us?
If I run the tests and nothing breaks its good code!?
Can we archive the whole test bed for Trunk?
Begin a new test bed for each release
archived test bed still available for engineers to test and bring forward into new test bed
Continue running the old test bed for ICs
Build & Run UT on the same machine
:check: Product registration issue identified before executing UT
:check: Save Time :
Engineering Test on CD Cut machine
:check: Test like a customer
:check: Less changes of residues issue
:check:Easy to run in any environment
Segregate Test Case
:check: Save time
Automated and non automated test cases
Only Automated test can be used to determine red squares
non-automated cause bugs to be raised on team backlogs
Flaky Test Case
Re-run the Flaky test required 2 to 4 (min) for each branch
Too much effort for re-running test case
Slow down CI process
Some test takes longer time
Dependency between the test cases causes inconsistent result
between Build machines
between engineers' machines
NO Standard dev environment
Single Test Framework Framework
:check: single test can be executed in all Runtimes (like CLR, DBGR, MCP)