Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add simple API for generating testharnesses inline #4629

Open
wants to merge 28 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 19 commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
db99d18
add HasTests
tmckay-sifive Jan 16, 2025
d96675c
reformat
tmckay-sifive Jan 16, 2025
4ffc80c
make it easier to override testharness generation
tmckay-sifive Jan 17, 2025
c9b887b
add scaladoc
tmckay-sifive Jan 17, 2025
80f5a31
reformat
tmckay-sifive Jan 17, 2025
90f68a6
flatten traits
tmckay-sifive Jan 21, 2025
c18e660
refactor based on PR feedback
tmckay-sifive Jan 27, 2025
a46103d
reformat
tmckay-sifive Jan 27, 2025
5e5bdd4
fix inlinetest package organization
tmckay-sifive Jan 28, 2025
60f2e5f
rename to InlineTest.scala
tmckay-sifive Jan 31, 2025
a1a1a7f
make TestParameters not a case class
tmckay-sifive Jan 31, 2025
502e0e9
make TestParameters private[inlinetest]
tmckay-sifive Jan 31, 2025
1436ea2
rename TestHarness to TestHarnessGenerator
tmckay-sifive Jan 31, 2025
d8aaaa7
add example that utilizes DUT type in testharness
tmckay-sifive Jan 31, 2025
3b24f28
REVERTME: comment out failing test
tmckay-sifive Jan 31, 2025
941cc4b
scala format
tmckay-sifive Jan 31, 2025
0488111
Revert "REVERTME: comment out failing test"
tmckay-sifive Jan 31, 2025
2e715dd
update test
tmckay-sifive Jan 31, 2025
77291c9
Merge remote-tracking branch 'origin/main' into tmckay/inline-testing
tmckay-sifive Feb 4, 2025
81202f6
add license string
tmckay-sifive Feb 5, 2025
92e208b
remove extra space
tmckay-sifive Feb 5, 2025
abe0c6b
update tests per feedback
tmckay-sifive Feb 5, 2025
f7b5887
move test: src/test/scala/chisel3 -> src/test/scala/chiselTests
tmckay-sifive Feb 5, 2025
6cf7428
misc feedback
tmckay-sifive Feb 5, 2025
70b5b40
add TestHarnessRawModule and TestHarnessModule traits
tmckay-sifive Feb 5, 2025
659adc9
reformat
tmckay-sifive Feb 5, 2025
6760320
PR feedback
tmckay-sifive Feb 5, 2025
fb52653
Merge branch 'main' into tmckay/inline-testing
tmckay-sifive Feb 6, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
80 changes: 80 additions & 0 deletions src/main/scala/chisel3/experimental/inlinetest/InlineTest.scala
Original file line number Diff line number Diff line change
@@ -0,0 +1,80 @@
package chisel3.experimental.inlinetest
tmckay-sifive marked this conversation as resolved.
Show resolved Hide resolved

import chisel3._
import chisel3.experimental.hierarchy.{Definition, Instance}

/** Per-test parametrization needed to build a testharness that instantiates
* the DUT and elaborates a test body.
tmckay-sifive marked this conversation as resolved.
Show resolved Hide resolved
*
* @tparam M the type of the DUT module
* @tparam R the type of the result returned by the test body
*/
class TestParameters[M <: RawModule, R] private[inlinetest] (
/** The [[desiredName]] of the DUT module. */
val dutName: String,
/** The user-provided name of the test. */
val testName: String,
/** A Definition of the DUT module. */
val dutDefinition: Definition[M],
/** The body for this test, returns a result. */
val body: Instance[M] => R
tmckay-sifive marked this conversation as resolved.
Show resolved Hide resolved
)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is almost a case class. Any reason to not make it one?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I originally had it that way, but according to @jackkoenig that is a problem for binary compatibility. I don't fully understand the reason for that, I'll admit.


/** An implementation of a testharness generator.
*
* @tparam M the type of the DUT module
* @tparam R the type of the result returned by the test body
*/
trait TestHarnessGenerator[M <: RawModule, R] {
tmckay-sifive marked this conversation as resolved.
Show resolved Hide resolved

/** Generate a testharness module given the test parameters. */
def generate(test: TestParameters[M, R]): RawModule with Public
}

object TestHarnessGenerator {

/** The minimal implementation of a unit testharness. Has a clock input and a synchronous reset
* input. Connects these to the DUT and does nothing else.
*/
Comment on lines +71 to +73
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is clock and reset sufficient? @fabianschuiki has brought up some points about: (1) knowing whether or not the test is done and (2) knowing what the exit status is.

Does this motivate a moderately more heavyweight interface of:

class UnitTestBundle extends Bundle {
  val clock = Flipped(Clock())
  val reset = Flipped(Bool())
  val done = Bool()
  val exitCode = UInt(32.W)
}

Alternatively, maybe there are two kinds of tests:

  1. Tests that finish with either an assertion failure or a stop.
  2. Tests that finish using the done/exitCode

Note: if the stop is dropped from (1), then (1) becomes purely a subset of (2).

Copy link
Contributor Author

@tmckay-sifive tmckay-sifive Feb 5, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes great point. This was my main motivation for making it easy for users to override the testharness generator and result from each test. I imagine each project might have their own contract with a test driver to communicate this information. I agree that done/exitCode is the obvious way to do that and more or less required for a sensible unit testing setup.

Since I do think most projects will settle on some testharness-testdriver contract to communicate done/pass, think it comes down to whether we prefer to ship an example implementation of this by default, or expect users to implement their own if they want it.

I landed on not adding done/pass because it felt less "presumptuous" of the users simulation environment. But I'm open to arguments that that comes at the cost of usability.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The reason why I'm leaning towards it is that we want to have the ability to do both test-discovery and test-execution for any test defined this way. The part which is missing is that we can do test-discovery with some small changes to FIRRTL to add a simulate statement/region similar to how we have a formal statement/region (see: https://github.com/chipsalliance/firrtl-spec/blob/main/spec.md#formal-unit-tests). Essentially, these tests just need a marker to indicate that they are tests and not just modules.

However, while formal has enough information to tell it also how to run the test (because a formal test is both simpler and the formal construct is powerful enough to capture stimulus via assume), simulate does not have enough information to run the test. Even in the simplest formulation of just having clock/reset, there are assumptions about how to drive the clock and the reset.

It's kind of like defining an entry point for a program (C/C++'s main function void main(int argc, char ** argv) (or the equivalent in other languages) comes to mind as does function calling convention).

FWIW, stop does include an exit code, so perhaps that is actually sufficient to use with only the clock/reset formulation. That actually sounds good.

Copy link
Contributor Author

@tmckay-sifive tmckay-sifive Feb 5, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense to me. If this is not blocking feedback, I will leave it for now.

I think it is a reasonable expectation that these testharness modules are not the top when we actually compile the simulation--there will be a test-driver layer above that generating clocks and resets, writing waves, etc. So, I think what we are looking for is an ABI between the test-driver and the testharness.

Perhaps testharnesses that can be simulated must implement this ABI. At which point, circt-test could have its own intrinsic test-driver that works well with verilator and has sane/configurable logic for clock/reset generation.

class UnitTestHarness[M <: RawModule](test: TestParameters[M, Unit]) extends Module with Public {
override def resetType = Module.ResetType.Synchronous
override val desiredName = s"test_${test.dutName}_${test.testName}"
tmckay-sifive marked this conversation as resolved.
Show resolved Hide resolved
val dut = Instance(test.dutDefinition)
test.body(dut)
}

implicit def unitTestHarness[M <: RawModule]: TestHarnessGenerator[M, Unit] = new TestHarnessGenerator[M, Unit] {
def generate(test: TestParameters[M, Unit]): RawModule with Public = new UnitTestHarness(test)
tmckay-sifive marked this conversation as resolved.
Show resolved Hide resolved
}
}

/** Provides methods to build unit testharnesses inline after this module is elaborated.
*
* @tparam TestResult the type returned from each test body generator, typically
* hardware indicating completion and/or exit code to the testharness.
*/
trait HasTests[M <: RawModule] { module: M =>

/** A Definition of the DUT to be used for each of the tests. */
private lazy val moduleDefinition =
module.toDefinition.asInstanceOf[Definition[module.type]]

/** Generate an additional parent around this module.
*
* @param parent generator function, should instantiate the [[Definition]]
*/
protected final def elaborateParentModule(parent: Definition[module.type] => RawModule with Public): Unit =
afterModuleBuilt { Definition(parent(moduleDefinition)) }

/** Generate a public module that instantiates this module. The default
* testharness has clock and synchronous reset IOs and contains the test
* body.
*
* @param body the circuit to elaborate inside the testharness
*/
protected final def test[R](testName: String)(body: Instance[M] => R)(implicit th: TestHarnessGenerator[M, R]): Unit =
elaborateParentModule { moduleDefinition =>
val test = new TestParameters[M, R](desiredName, testName, moduleDefinition, body)
th.generate(test)
}
}
161 changes: 161 additions & 0 deletions src/test/scala/chisel3/experimental/InlineTestSpec.scala
tmckay-sifive marked this conversation as resolved.
Show resolved Hide resolved
Original file line number Diff line number Diff line change
@@ -0,0 +1,161 @@
package chiselTests

import chisel3._
import chisel3.util.Enum
import chisel3.testers._
import chisel3.experimental.inlinetest._
import chisel3.experimental.hierarchy._

class TestResultBundle extends Bundle {
val finish = Output(Bool())
val code = Output(UInt(8.W))
}

// Here is a testharness that consumes some kind of hardware from the test body, e.g.
// a finish and pass/fail interface.
object TestHarnessWithResultIO {
class TestHarnessModule[M <: RawModule](test: TestParameters[M, TestResultBundle]) extends Module {
override def resetType = Module.ResetType.Synchronous
override val desiredName = s"test_${test.dutName}_${test.testName}"
val dut = Instance(test.dutDefinition)
tmckay-sifive marked this conversation as resolved.
Show resolved Hide resolved
val result = IO(new TestResultBundle)
result := test.body(dut)
}
implicit def testharnessGenerator[M <: RawModule]: TestHarnessGenerator[M, TestResultBundle] =
new TestHarnessGenerator[M, TestResultBundle] {
def generate(test: TestParameters[M, TestResultBundle]): RawModule with Public =
new TestHarnessModule(test) with Public
}
}

object TestHarnessWithMonitorSocket {
// Here is a testharness that expects some sort of interface on its DUT, e.g. a probe
// socket to which to attach a monitor.
class TestHarnessModule[M <: RawModule with HasMonitorSocket](test: TestParameters[M, Unit]) extends Module {
override def resetType = Module.ResetType.Synchronous
override val desiredName = s"test_${test.dutName}_${test.testName}"
val dut = Instance(test.dutDefinition)
test.body(dut)
val monitor = Module(new ProtocolMonitor(dut.monProbe.cloneType))
monitor.io :#= probe.read(dut.monProbe)
}
implicit def testharnessGenerator[M <: RawModule with HasMonitorSocket]: TestHarnessGenerator[M, Unit] =
new TestHarnessGenerator[M, Unit] {
def generate(test: TestParameters[M, Unit]): RawModule with Public =
new TestHarnessModule(test) with Public
}
}

@instantiable
trait HasMonitorSocket { this: RawModule =>
protected def makeProbe(bundle: ProtocolBundle): ProtocolBundle = {
val monProbe = IO(probe.Probe(chiselTypeOf(bundle)))
probe.define(monProbe, probe.ProbeValue(bundle))
monProbe
}
@public val monProbe: ProtocolBundle
}

class ProtocolBundle(width: Int) extends Bundle {
val in = Flipped(UInt(width.W))
val out = UInt(width.W)
}

class ProtocolMonitor(bundleType: ProtocolBundle) extends Module {
val io = IO(Input(bundleType))
assert(io.in === io.out, "in === out")
}

@instantiable
class ModuleWithTests(ioWidth: Int = 32) extends Module with HasMonitorSocket with HasTests[ModuleWithTests] {
@public val io = IO(new ProtocolBundle(ioWidth))

override val monProbe = makeProbe(io)

io.out := io.in

test("foo") { instance =>
instance.io.in := 3.U(ioWidth.W)
assert(instance.io.out === 3.U): Unit
}

test("bar") { instance =>
instance.io.in := 5.U(ioWidth.W)
assert(instance.io.out =/= 0.U): Unit
}

{
import TestHarnessWithResultIO._
test("with_result") { instance =>
val result = Wire(new TestResultBundle)
val timer = RegInit(0.U)
timer := timer + 1.U
instance.io.in := 5.U(ioWidth.W)
val outValid = instance.io.out =/= 0.U
when(outValid) {
result.code := 0.U
result.finish := timer > 1000.U
}.otherwise {
result.code := 1.U
result.finish := true.B
}
result
}
}

{
import TestHarnessWithMonitorSocket._
test("with_monitor") { instance =>
instance.io.in := 5.U(ioWidth.W)
assert(instance.io.out =/= 0.U): Unit
}
}
}

class HasTestsSpec extends ChiselFlatSpec with FileCheck {
it should "generate a public module for each test" in {
generateFirrtlAndFileCheck(new ModuleWithTests)(
"""
| CHECK: module ModuleWithTests
|
| CHECK: public module test_ModuleWithTests_foo
| CHECK: input clock : Clock
| CHECK: input reset : UInt<1>
| CHECK: inst dut of ModuleWithTests
tmckay-sifive marked this conversation as resolved.
Show resolved Hide resolved
|
| CHECK: public module test_ModuleWithTests_bar
| CHECK: input clock : Clock
| CHECK: input reset : UInt<1>
| CHECK: inst dut of ModuleWithTests
|
| CHECK: public module test_ModuleWithTests_with_result
| CHECK: input clock : Clock
| CHECK: input reset : UInt<1>
| CHECK: output result : { finish : UInt<1>, code : UInt<8>}
| CHECK: inst dut of ModuleWithTests
|
| CHECK: public module test_ModuleWithTests_with_monitor
| CHECK: input clock : Clock
| CHECK: input reset : UInt<1>
| CHECK: inst dut of ModuleWithTests
| CHECK: inst monitor of ProtocolMonitor
| CHECK: connect monitor.clock, clock
| CHECK: connect monitor.reset, reset
| CHECK: connect monitor.io.out, read(dut.monProbe).out
| CHECK: connect monitor.io.in, read(dut.monProbe).in
"""
)
}

it should "compile to verilog" in {
generateSystemVerilogAndFileCheck(new ModuleWithTests)(
"""
| CHECK: module ModuleWithTests
| CHECK: module test_ModuleWithTests_foo
| CHECK: module test_ModuleWithTests_bar
| CHECK: module test_ModuleWithTests_with_result
| CHECK: module test_ModuleWithTests_with_monitor
"""
)
}
}