Skip to content
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 40 additions & 0 deletions specs/activity/protocol-activity.md
Original file line number Diff line number Diff line change
Expand Up @@ -1952,6 +1952,7 @@ Activity [entities](#entity) communicate extra metadata about the activity, such
| number | N/A | Number |
| clientInfo | N/A | Skype client info |
| streamInfo | N/A | Streaming text metadata |
| context | N/A | Screen and visual context metadata which can be used as Query context |

### string and number

Expand Down Expand Up @@ -2200,6 +2201,45 @@ Voice streaming uses `event` activities with [Media.* events](#reserved-events-f
}
```

### context
Comment thread
sveluvil marked this conversation as resolved.
The context entity contains contextual information between users and agents to improve understanding of ongoing conversations. This enables more relevant, personalized, and accurate responses, especially in multimodal interactions. By embedding context directly in the activity message, agents can leverage contextual information such as images, custom objects, and audio/video-related context.
Context can only be shared with activity type `message`, and it can be send from any channel.
There is no strict limit on the number of images that can be shared. However, it is recommended to keep the number as low as possible, as this directly impacts token consumption.

| Property | Type | Required | Description |
|------------------|---------|----------|---------------------------------------------------------------------------------|
| `type` | string | Yes | Must be `"context"` |
| `screenContext` | object | Yes | Screen context details |

The context entity in an activity message can contain a `screenContext` object. This can be a list of screenshots of the page the user is currently on. The images are shared either in base64-encoded format or as a URL.
Comment thread
sveluvil marked this conversation as resolved.
Comment thread
sveluvil marked this conversation as resolved.

| Property | Type | Required | Description |
|------------------|------------------|----------|---------------------------------------------------------------------------------|
| `images` | array of string | Yes | An array of images either in base64-encoded format or as URL |

`A9263`: If the context contains no elements, it MUST be ignored.

`A9264`: If the context entity contains any syntactic errors, it MUST be ignored.

`A9265`: If the `screenContext` object within a `context` entity contains no properties, it MUST be ignored.

```
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: ```json

// Sending context information as part of the activity message
Comment thread
sveluvil marked this conversation as resolved.
{
"type": "message",
"text": "What pizzas are available for $150?",
"entities": [
{
"type": "context",
"screenContext": {
"images": ["iVBORw0KGgoAAAANSUhEUgAAAAUA...", "https://example.com/product-page"]
}
}
]
}
```


# Appendix III - Protocols using the Invoke activity

The [invoke activity](#invoke-activity) is designed for use only within protocols supported by Activity Protocol channels (i.e., it is not a generic extensibility mechanism). This appendix contains a list of all protocols using this activity.
Expand Down
Loading