Understanding HUD Environments
An Environment
in the HUD SDK represents the runtime instance (like a web browser or an operating system desktop) where an Agent interacts to complete a Task.
The Environment
is the core component your agent interacts with. It provides:
CLA
format (see Adapters) to the runtime.Observation
objects (like screenshots, text) from the runtime.close
) and evaluating task success (evaluate
).Environments are created using hud.gym.make()
. You typically pass a Task object, which defines the required environment type (gym
attribute), setup actions, and evaluation logic.
You can also create environments directly using known Gym IDs:
Environments created this way won’t have a default Task
associated unless you explicitly reset them with one later using env.reset()
. The gym.make()
function also automatically links the environment to an active Job if one was defined using the @register_job
decorator.
The HUD SDK provides several standard environment types, specified via the gym
attribute in a Task or directly in hud.gym.make()
:
"hud-browser"
: Provides a remote Chromium browser instance managed via Playwright. Ideal for web navigation, form interaction, and testing web applications.
"qa"
: A non-interactive environment for question-answering tasks where the agent provides a direct textual response.
CustomGym
: Allows defining and running your own Custom Environments using Docker, either locally or remotely. This provides maximum flexibility for specific testing needs.The gym
attribute in a Task tells hud.gym.make()
which environment to instantiate.
The standard interaction flow involves the Agent and the Environment:
env.step()
without actions after gym.make()
. This returns the starting Observation
.predict(observation)
method processes the observation and decides on the next action(s).env.step(actions)
with the agent’s action(s) (which should be in CLA
format, usually handled by an Adapter). This executes the actions and returns the next Observation
, reward
, terminated
flag, and info
dict.predict
call. The agent should output a ResponseAction
, which the environment stores. The subsequent env.evaluate()
call then checks this stored response. The environment itself remains largely passive for QA.env.step(actions: list[CLA] | None = None)
: Executes actions (or gets initial state). Returns (Observation, reward, terminated, info)
.env.evaluate(config: FunctionConfigs | None = None)
: Runs evaluation logic defined in the Task (or the provided config
). Returns evaluation result.env.close()
: Shuts down the environment. Saves the Trajectory if linked to a Job.env.get_urls()
: Returns URLs (url
, live_url
) for accessing/viewing the environment.env.reset(configs: FunctionConfigs | None = None)
: Resets state, often running setup steps. Mostly used internally or for environments created without an initial Task.env._setup(...)
/ env._invoke_all(...)
: Internal methods for running setup/evaluate/custom configurations defined in a Task.The Observation
object returned by env.step()
contains:
screenshot
(str | None): Base64-encoded PNG string.text
(str | None): Optional text output (e.g., terminal, accessibility tree).gym
), setup
, and evaluate
logic.step
and predict
methods.step
are in the correct CLA
format.@register_job
or gym.make(job=...)
.