Normative and non-normative requirements of "witness" don't specify that they shouldn't do things outside of their scope or convey the intentions of Sam Smith that witnesses should be constrained to a small set of duties

[daidoji]

Here's a quick copy/paste job of most of the normative requirements regarding witnesses in the spec. There are not many non-normative descriptions outside of these sections but I cover the definition and the main sections describing witnesses here.

Witness
a witness is an entity or component designated (trusted) by the controller of an identifier. The primary role of a witness is to verify, sign, and keep events associated with an identifier. A witness is the controller of its own self-referential identifier which may or may not be the same as the identifier to which it is a witness. See Annex A under KAWA (KERI’s Algorithm for Witness Agreement).

Note: There is no "Annex A" this could be another issue.
Although a primary role is specified secondary or subsequent roles are not ruled out

The requirements from: https://trustoverip.github.io/tswg-keri-specification/#indirect-exchange-via-witnesses-and-watchers

1. Each controller of an AID MAY create or choose to use a set or pool of witnesses for that AID.
2. It MAY use a witness service provided by some other party or MAY directly host its own witnesses in its own infrastructure or some combination of the two.
3. the composition of the Witness pool is under the ultimate control of the AID’s controller, which means the controller MAY change the witness infrastructure at will
   Witnesses could do a lot of things but don't seem to have to do anything specific from this section

From section https://trustoverip.github.io/tswg-keri-specification/#indexed-signatures

1. Witnesses MUST use only nontransferable identifiers, which include the controlling public key.
   This is a normative spec for witnesses but included in the indexed signatures section

Validation Rules: https://trustoverip.github.io/tswg-keri-specification/#validation-rules

1. the event’s witness MUST first verify the event’s controller signatures before it can sign (witness) and accept that event into its copy of that event’s KEL.
2. The witness then SHOULD create a receipt of the event with the attached controller and witness signatures and can then propagate that event to other validators.
3. Given a remote event, the event’s witness SHOULD NOT sign or otherwise accept that event into its copy of that event’s KEL.
4. The witness SHOULD NOT propagate that event to other validators including witnesses.
   These normative requirements are hard to parse, but witness verifies signatures before it signs and accepts that copy of the KEL as "first seen", they should disseminate that event to "other validators"?, they shouldn't accept remote events or propagate them remotely. Sure

KAWA

Only normative requirement is that witnesses abide by KAWA which is a set of declarative requirements for a loose protocol but does not specify the mechanism of KAWA in specifics.

Roles in OOBI Sections

Finally there's the section on OOBIs in which roles are discussed, one of which is witness. However, this mostly deals with the signaling mechanism of the OOBI file itself and not of any restricted API or normative constraints upon that role.

So while we have a declarative description of what witnesses should do in terms of KAWA, Validation, and what controllers "could do" with witnesses in the descriptions of witness/watcher/judge/juror ecosystem. There are no normative requirements or even non-normative descriptions or guidance that witnesses shouldn't perform other operations or should be actively discouraged from doing things that aren't explicitly just writing receipts on their AIDs and engaging in KAWA consensus, publicly witnessing AIDs that have achieved KAWA consensus (which I believe was apparently the intent).

In the KERIpy dev meeting for today 8/13/2024, (see recording for more details) it became clear that @SmithSamuelM's intention was that the witness (and presumably the watcher/judge/juror ) have a constrained set of behavior limited to a small set of functionality and that any extraneous responsibilities or duties of a witness (in this specific case) be considered outside the scope of the spec. This issue is to track the fact that this intention seems quite ambiguous from the spec as written.

A fix to the spec would probably be to:

1. Create a non-normative guidance of this intention in a section singled out for the various agent roles in the keri ecosystem (witness/watcher/judge/juror/validator) that describes the scope and constraints of each of these agents as intended.
2. Create a set of normative requirements that say that witnesses should not be used for anything other than the strict set of duties to which they are confined.

[SmithSamuelM]

@daidoji

Create a non-normative guidance of this intention in a section singled out for the various agent roles in the keri ecosystem (witness/watcher/judge/juror/validator) that describes the scope and constraints of each of these agents as intended.

I agree that there should be some non-normative guidance. Not sure if that should be in the spec or just referenced in the spec. It would have to be generated first and then we can decide where to put it.

Create a set of normative requirements that say that witnesses should not be used for anything other than the strict set of duties to which they are confined.

I disagree. One can always decide to overload a witness with additional functionality. The risk is that it will not be interoperable. With rare exception, all the extra functionality we are talking about is what a watcher does already. So one makes a design choice to attempt to combine witness and watcher functionality but does so at ones peril. We don't need the spec to say this. This goes without saying in general. Normative requirements are normative because they guarantee a set of features. One can always choose to add other features. The question is, are the normative requirements for watchers sufficiently well defined. For example the concept of "remote" versus "local" event has meaning when applied to witnesses and controllers. By definition all events are remote for watchers. So if logically, the processing of attachments to an event is different if its (local vs remote, locallyWitnessed, locallyOwened) then its not possible to act as both a witness and a watcher for that same event. The event has to go to a different endpoint. It can't be combined. So informative language might better explain why its a bad idea but logically if one attempt to comply with the normative language one will discover that its a bad idea without being told why.

[daidoji]

I can make another issue for the watcher spec being just as ambiguous if you'd like (originally in this issue I had included all the normative requirements for watcher too but this issue became too large so I constrained it to witnesses only). If all the agent types EXCEPT the witness were well defined with normative requirements, then it might be possible to implement the spec and have the understanding you believe you're communicating. As it is I don't think the watchers, judges, or jurors reach that level of normative rigor either.

I could make the perfectly valid assumption that "oh a witness has a normative requirement to witness the KEL to the AID which it witnesses" THEREFORE the delegator KEL that I had to process to verify is really just an extension of the KEL of the delegatee (albeit one outside of the atomic KEL unit) so providing that is witnessing the part of KEL that this witness is witnessing. There is no normative requirement that I only witness KELs as atomic units. So as a witness I'll provide witness this "piece" of the KEL as well once the KAWA threshold is satisfied even though its an atomic KEL unit (the delegatee's KEL and the delegator's KEL that I'm witnessing). Its not "remote" because these events were signed, verified, and validated by the witness. There is no normative requirement that the DAG concept of a KEL with all the superseding recovery mechanisms be constrained to an event stream that starts at the "icp" event for example.

The meta-problem security wise for all these types of agents is to provide mitigations to security threats based on the context of what one is doing. Clearly if we think about the agents as roles (constrained APIs or behaviors) your reading comes into play. However, if I think about the agents as just actors in the system, then another reading comes into play where the agent types have pools of data that have some security context associated with them. ie) I could get all my information from the controller of an AID itself (but then I'm vulnerable to a set of security threats), I could get all my information from witness pools (but then I'm vulnerable to another set of security threats), I could get my information from watcher networks (but then I'm open to another set of security threats). This tiered data availability is a completely valid reading to me and if its not valid the normative requirements don't quite communicate that this understanding causes things not to work.

If you have the intention that these things be well constrained and defined, more rigor in the normative requirements are needed to prevent the ambiguity that would cause me to believe this is an inappropriate decision.

[SmithSamuelM]

@daidoji I appreciate your feedback. Let me approach it from a different perspective. The primary distinction between the roles of the different components is the validation logic that determines what must be validated in order to accept an event into a KEL held by one of the components. The validation rules for a controller of its own KEL are different than the validation rules for some other KEL. Likewise the validation rules for a witness of a KEL for which it is a witness are different than the validation rules for some other KEL. Likewise the validation rules for a delegator of a KEL are different that they are for some other KEL. The default validation rules apply when validator is not the controller, delegate, delegator, or witness to that KEL. This default rules are the baseline rules for a watcher of a KEL. So when an event for that KEL is received for validation the validation rules are different depending on the role of the validator with respect to that KEL. The problem is that API service endpoints are attached to services that are acting as one of the roles. It is problematic to send an event to a service endpoint for which the service is acting as a witness of that event and expect that service to validate it as if it were a controller or a delegator or a watcher. The problem is the APIs do not always respect that validation rules so they need to be fixed.

There are various ways for an implementation to respect those validation rules. This has nothing to do with the normative specification of the validation rules themselves. One of the reasons the KAPI repos exists is for the community to work out how it wants to have an interoperable set of service endpoints that respect the normative rules for validation.

How one chooses to build service endpoints and apis is not part of the KERI spec. It is true that interoperability via APIs is not guaranateed that way. But that is a different specification that needs to be developed further and the KAPI spec is the first start.

When I say that KERI is not finished its largely because we as a community have not done the work to build interoperable APIs.

The current keripy made some expedient choices in order to get GLEIF vLEIs into production. One of those choices defeated a security property of witness pools with respect to delegators. The validation rules have been fixed to restore that but the APIs themselves have yet to be fixed to respect those rules.

[daidoji]

Sure, I def hear you on all that and I think I understand your intention. I just think that maybe that intention isn't coming through clearly in the spec itself. I've read it probably 20 times by now in its entirety and I didn't really understand that point until the dev meeting on Tuesday. If we want to close this issue after adding some non-normative text we can, but others in the future may have the same types of difficulties.

[SmithSamuelM]

Please suggest some non-normative test that would make it clear. I suffer from the curse of too much knowledge. What seems obvious too me, will not be obvious to others and I can’t tell what is or is not obvious to others.

[daidoji]

Well why not just add the normative requirements (and break out into sections) the specific duties of witness, watcher, judge, juror, validator roles along with the normative requirement that those roles do nothing other than those specific duties. Like if witnesses are constrained to just witnessing KELs atomically, then "Witnesses MUST only verify specific KELs atomically. All other data needed to verify within a security context such as delegation events within other KELs that link to the KEL being verified or events otherwise located in other KELs/backers are out of scope for any node performing the witness role".

Etc.. etc.. with watchers/judges/jurors. If these are specific roles and they have specific duties, why not just constrain the spec to only those duties for agents that perform each role?

[SmithSamuelM]

@daidoji A spec that did what you said would be lenghty and IMHO deserves its own specification. Those requirements become API dependent and an API specification brings in other dependencies like the type of API. No doubt various implementers will want those APIs to be compatible with those requirements.

What I have asked of you is to write some informative text that would fill in the blanks of the normative requirements for the validation rules for the core KERI spec because it wasn’t clear to you unlil recently. and I asked for you to explain it in your own words so that it would be clear to others in similar circumstances.

Finally, creating a detailed normative specification of the APIs for all the KERI infrastructure components is a lot of work. So when you say

Well why don’t you add the normative requirements

My first thought is, are you going to hire me as a consultant to do this? Or contribute to the new KERI foundation?
The 5Ws is exactly this work.It would be foolish to finalize a normative API spec withour first implementing it enough to suss out the details of how such a normative spec should look. This is only just begineeing. It will take some time and resources to complete. The KERI foundation is how some members of the community have decided to address this missing peice.

You may not be aware but the vast majority of the work that I do on KERI is on my own dime, by spending my retirement funds. So the pace that extra work gets done is only as fast as I can do it given the limited resources I have to spend on it.

So I think it would be great for those who want to further develop KERI infrastructure components and have interoperability at the API lievel to help here. At leastif they are in a hurry.

What I have asked of you is to write some informative text that would fill in the blanks of the existing normative requirements for the validation rules for the core KERI spec because it wasn’t clear to you unlil recently. and I asked for you to explain it in your own words so that it would be clear to others in similar circumstances. The core KERI spec is about the syntax and validation rules for Key Events and other message types. Not about how to build the infrastructure that implements all the components.

The components are described in informative text in the indroduction to the KERI spec. The KERI protocol spec and the CESR spec define the messages which are a dependency for infrastructure but not the infrastructure. That is still a work in progress.

[SmithSamuelM]

@daidoji how a Juror and a Judge behave is Validator specific. The high level function is described but not the low level details. The low level details are a function of the needs of the Validator with respect to a given trust decision. So the composition of a Jury pool, the rules a judge applies to that pool are all validator specific. The level of trust/risk that a Validator wants with respect to making a trust decision is a function of the value/liability/risk of the trust decision which is often highly application specific. So while KERI provides duplicity evident verifiable data structure via KEY Event Logs. It does not constrain a validator in how it wants to configure its jury pool and judges.

Its up to the community to develop interoperabile specifications for jury pools and judge behavior algorithms and apis that have defined behavior for defined trust decision applications. First you have to decide what types of trust decisons in what applications and in what ecosystems.

GLEIF spent a lot of resources developing an ECO system governance framework and defining the types of trust decisions they wanted. As a result they decided they did not need, at least initially, watchers, jury pools or judges. That they had enough trust in the QVI participants providing witnesses to not implement those components. The plan was to add watchers which is now a work item that some members of the GLEIF ecosystem are putting resources into addressing.

It was only last December that I finished implementing the delegated superseding recovery validation rules for delegated KELs. This was so that I could be confident in the normatively expression in the spec. This was a longstanding open issue for KERI that took a large investment of my time to complete.

The result is that the stopgap approach used in KERIpy for delegation in the KLI api needs to be fixed to implement the correct behavior. We are working on it. But the KLI is not part of the KERI spec it is part of a nascent API spec that is in the KAPI repo.

Provenant and healthKERI both contributed some developer resources towards the first steps in building that KAPI specification.

The purpose of the Fifth W (wizard) in the 5 Ws is to provide a sane default configuation of behaviors for a vanilla KERI infrastructure. That has yet to be worked out. One reason I can’t just add normative specifications for infrastructure APIs is that I don’t know what they should be in general or in specifics because we have yet to build all of the components.

Let me suggest an analogy. A building framer has a set of resources and constraints. They can build buildings given the building plans account for their resources and constraints. For example lumber that comes in standard sizes, brackets, hammers and nails, saws. But until the framer gets a set of plans it don’t know exactly how to configure the lumber.

An Architect who generates the plans therefore must know apriori the resources and constraints of the Framer but that is not enough. An architect must also understand the functionality of the building whcih is defined by who wants the building say the prospective residents of that building..

So the architect must figure out how to design a building that satisfies the constraints of the framer as well as the constraints of the residents. An inexperienced Architect will likely get it wrong. An experienced architect will have designed enough similar buildings to know what are practical feasible constraints from the residents and what are not. IMHO one should build some prototypes at least to gain that experience.

The KERI core specification provides the standard lumber sizes and the tools aka the framer constraints but not the constraints of the residents aka the application that defines what the building (aka infrastructure) should do. For that we need to know the application (aka the residents) constraints. This is what the API spec or API specs do. And we don’t have those yet and frankly don’t have enough experience even information about the applicationsto define APIs in general yet. We have a very limited set relatively speaking of applications constraints that came from the GLEIF application. Little else.

I have no idea what your application is for example. So I can only speculate what would be good for your application. No doubt your application whatever it may be, may be sufficiently similar to other applications in a family of applications that an API set could be designed that satisfied the constraints of the family. That remains to be worked out by the community. What the KERI and CESR specifications do is define the framer constraints, the building blocks and tools for building but not the building itself (aka infrastructure).