Temper Bench v2
The first version of our benchmark, which was used to evaluate Temper 0.5B, eventually stopped meeting our expectations. It had 100 tasks at three levels: 50 trivial, 30 easy and 20 medium. It did not overlap with the training dataset and had no leaks, but its tasks were prepared by the same DeepSeek models that generated the training data, so the benchmark matched the dataset in style. On top of that, most of it consisted of short algorithmic exercises such as primality checks, factorials, GCD or FizzBuzz. They test knowledge of syntax well, but say little about how a model handles real code, and that was what mattered most to us.
We wrote the second version to be a measure of applied work. Instead of algorithmic exercises, it uses small self-contained tasks that resemble common parts of real projects, such as computing a delivery fee, parsing a config, parsing an HTTP request or building a cache that several threads use at once. Each task has at least five tests, always including empty input and edge cases. The benchmark turned out hard even for larger models: Qwen2.5-Coder-7B solves 46% of basic tasks on the first try, 22% of medium ones and only 10% of advanced ones.
Task examples
Write normalize_phone(raw: &str) -> Option<String> for US phone numbers. Ignore everything that isn't a digit. If there are 10 digits, or 11 digits starting with 1, format the last 10 as (555) 123-4567. Anything else gives None. Reply with a single file containing the solution.
Hidden tests
#[cfg(test)]
mod tests {
use super::*;
fn some(s: &str) -> Option<String> {
Some(s.to_string())
}
#[test]
fn plain_ten_digits() {
assert_eq!(normalize_phone("5551234567"), some("(555) 123-4567"));
}
#[test]
fn various_separators() {
assert_eq!(normalize_phone("555-123-4567"), some("(555) 123-4567"));
assert_eq!(normalize_phone("(555) 123 4567"), some("(555) 123-4567"));
assert_eq!(normalize_phone("555.123.4567"), some("(555) 123-4567"));
}
#[test]
fn country_code() {
assert_eq!(normalize_phone("+1 555 123 4567"), some("(555) 123-4567"));
assert_eq!(normalize_phone("15551234567"), some("(555) 123-4567"));
}
#[test]
fn eleven_digits_without_one() {
assert_eq!(normalize_phone("25551234567"), None);
}
#[test]
fn wrong_length() {
assert_eq!(normalize_phone("555123456"), None);
assert_eq!(normalize_phone("115551234567"), None);
assert_eq!(normalize_phone(""), None);
assert_eq!(normalize_phone("call me"), None);
}
#[test]
fn letters_ignored() {
assert_eq!(normalize_phone("tel: 555 123 4567 (home)"), some("(555) 123-4567"));
}
}Reference solution
pub fn normalize_phone(raw: &str) -> Option<String> {
let digits: String = raw.chars().filter(|c| c.is_ascii_digit()).collect();
let digits = match digits.len() {
10 => digits.as_str(),
11 if digits.starts_with('1') => &digits[1..],
_ => return None,
};
Some(format!("({}) {}-{}", &digits[..3], &digits[3..6], &digits[6..]))
}Implement a loader for a small key = value configuration format, with errors reported through Result and the ? operator. Use this API:
pub fn parse_config(text: &str) -> Result<Config, ConfigError>;Config is a struct with the public fields name: String, retries: u32 and timeout_ms: u64. ConfigError is an enum with these variants: MissingKey(&'static str), BadLine(usize) and BadNumber { key: String, source: ParseIntError }, where ParseIntError is std::num::ParseIntError. Both types derive Debug and PartialEq. Rules: - Each line is trimmed. Blank lines and lines that start with # are ignored. - Every other line must contain =. Key and value are trimmed. A line without = gives ConfigError::BadLine with its 1-based number in the text, counting blank and comment lines. - The keys retries and timeout_ms are parsed as numbers of their types. A value that does not parse gives BadNumber with the key. The key name is taken as text. - Unknown keys are ignored. When a key appears more than once, the last value wins. - After all lines are read, a missing name gives MissingKey("name"), then a missing retries gives MissingKey("retries"), then a missing timeout_ms gives MissingKey("timeout_ms"). Check them in this order. Reply with a single file containing the solution.
Hidden tests
#[cfg(test)]
mod tests {
use super::*;
const FULL: &str = "name = api\nretries = 3\ntimeout_ms = 1500\n";
#[test]
fn parses_full_config() {
assert_eq!(
parse_config(FULL),
Ok(Config {
name: "api".to_string(),
retries: 3,
timeout_ms: 1500,
})
);
}
#[test]
fn skips_blank_and_comment_lines() {
let text = "# header\n\n name=api \nretries=0\n# note\ntimeout_ms=5\n";
let config = parse_config(text).unwrap();
assert_eq!(config.name, "api");
assert_eq!(config.retries, 0);
}
#[test]
fn line_without_equals_reports_its_line() {
let text = "name = api\nretries 3\ntimeout_ms = 1\n";
assert_eq!(parse_config(text), Err(ConfigError::BadLine(2)));
}
#[test]
fn bad_number_names_the_key() {
let text = "name = api\nretries = x\ntimeout_ms = 1\n";
assert!(matches!(
parse_config(text),
Err(ConfigError::BadNumber { ref key, .. }) if key == "retries"
));
}
#[test]
fn negative_retries_is_bad_number() {
let text = "name = api\nretries = -1\ntimeout_ms = 1\n";
assert!(matches!(parse_config(text), Err(ConfigError::BadNumber { .. })));
}
#[test]
fn missing_keys_checked_in_order() {
assert_eq!(parse_config(""), Err(ConfigError::MissingKey("name")));
assert_eq!(
parse_config("name = a\ntimeout_ms = 1\n"),
Err(ConfigError::MissingKey("retries"))
);
assert_eq!(
parse_config("name = a\nretries = 1\n"),
Err(ConfigError::MissingKey("timeout_ms"))
);
}
#[test]
fn last_value_wins_and_unknown_keys_ignored() {
let text = "name = old\nname = new\nretries = 1\ntimeout_ms = 2\ncolor = blue\n";
let config = parse_config(text).unwrap();
assert_eq!(config.name, "new");
}
}Reference solution
use std::num::ParseIntError;
use std::str::FromStr;
#[derive(Debug, PartialEq)]
pub struct Config {
pub name: String,
pub retries: u32,
pub timeout_ms: u64,
}
#[derive(Debug, PartialEq)]
pub enum ConfigError {
MissingKey(&'static str),
BadLine(usize),
BadNumber { key: String, source: ParseIntError },
}
fn number<T: FromStr<Err = ParseIntError>>(key: &str, value: &str) -> Result<T, ConfigError> {
value
.parse::<T>()
.map_err(|source| ConfigError::BadNumber {
key: key.to_string(),
source,
})
}
pub fn parse_config(text: &str) -> Result<Config, ConfigError> {
let mut name = None;
let mut retries = None;
let mut timeout_ms = None;
for (index, raw) in text.lines().enumerate() {
let line = raw.trim();
if line.is_empty() || line.starts_with('#') {
continue;
}
let (key, value) = line.split_once('=').ok_or(ConfigError::BadLine(index + 1))?;
let (key, value) = (key.trim(), value.trim());
match key {
"name" => name = Some(value.to_string()),
"retries" => retries = Some(number::<u32>(key, value)?),
"timeout_ms" => timeout_ms = Some(number::<u64>(key, value)?),
_ => {}
}
}
Ok(Config {
name: name.ok_or(ConfigError::MissingKey("name"))?,
retries: retries.ok_or(ConfigError::MissingKey("retries"))?,
timeout_ms: timeout_ms.ok_or(ConfigError::MissingKey("timeout_ms"))?,
})
}Write a parser for plain HTTP/1.x requests in Rust. The method, path, header names and values and the body all borrow from the input string. Use only std. Use this API:
#[derive(Debug, PartialEq)]
pub struct Request<'a> {
pub method: &'a str,
pub path: &'a str,
pub headers: Vec<(&'a str, &'a str)>,
pub body: &'a str,
}
#[derive(Debug, PartialEq)]
pub enum RequestError {
BadRequestLine,
BadVersion(String),
BadHeader(usize),
Truncated,
LengthMismatch { declared: usize, actual: usize },
}
pub fn parse_request<'a>(raw: &'a str) -> Result<Request<'a>, RequestError>;The input uses \n as the line separator, and no other handling of line endings is done. Lines are numbered from 1, and the request line is line 1. - The request line must have exactly three parts separated by single spaces: the method, the path and the version. A part that is empty, or a line with more or fewer than three parts, gives BadRequestLine. - The version must be HTTP/1.1 or HTTP/1.0. Any other version gives BadVersion with the version text. - The headers follow the request line, one per line, and they end at the first empty line. A header line must contain a :. The name is the text before the first :, and it must not be empty and must not contain a space. Otherwise the result is BadHeader with the line number. The value is the text after the first :, with the spaces at its start and end removed. - If the input ends before an empty line is found, the result is Truncated. - The body is everything after the empty line that ends the headers. It may be empty. - If a header named Content-Length (compared without regard to ASCII case) is present, only the first such header counts. Its value must be a decimal number, or the result is BadHeader with its line number. The number must equal the length of the body in bytes, or the result is LengthMismatch with the declared number and the actual length. Without that header, the body length is not checked. Reply with a single file containing the solution.
Hidden tests
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn get_with_headers_and_no_body() {
let raw = "GET /index.html HTTP/1.1\nHost: example.org\nAccept: text/html \n\n";
assert_eq!(
parse_request(raw),
Ok(Request {
method: "GET",
path: "/index.html",
headers: vec![("Host", "example.org"), ("Accept", "text/html")],
body: "",
})
);
}
#[test]
fn body_after_blank_line() {
let raw = "POST /x HTTP/1.0\n\nhello";
let req = parse_request(raw).unwrap();
assert_eq!(req.body, "hello");
assert!(req.headers.is_empty());
}
#[test]
fn content_length_must_match_body_bytes() {
let raw = "PUT /a HTTP/1.1\ncontent-length: 5\n\nhéllo";
assert_eq!(
parse_request(raw),
Err(RequestError::LengthMismatch { declared: 5, actual: 6 })
);
let ok = "PUT /a HTTP/1.1\nContent-Length: 6\n\nhéllo";
assert_eq!(parse_request(ok).unwrap().body, "héllo");
}
#[test]
fn first_content_length_header_counts() {
let raw = "PUT /a HTTP/1.1\nContent-Length: 2\nContent-Length: 9\n\nab";
assert_eq!(parse_request(raw).unwrap().body, "ab");
}
#[test]
fn content_length_that_is_not_a_number_is_bad_header() {
let raw = "PUT /a HTTP/1.1\nContent-Length: two\n\n";
assert_eq!(parse_request(raw), Err(RequestError::BadHeader(2)));
}
#[test]
fn request_line_needs_exactly_three_single_spaced_parts() {
assert_eq!(parse_request("GET /x\n\n"), Err(RequestError::BadRequestLine));
assert_eq!(parse_request("GET /x HTTP/1.1\n\n"), Err(RequestError::BadRequestLine));
assert_eq!(parse_request("GET /x HTTP/1.1 extra\n\n"), Err(RequestError::BadRequestLine));
}
#[test]
fn unsupported_version() {
assert_eq!(
parse_request("GET / HTTP/2.0\n\n"),
Err(RequestError::BadVersion("HTTP/2.0".to_string()))
);
}
#[test]
fn header_without_colon_reports_line() {
assert_eq!(
parse_request("GET / HTTP/1.1\nHost example\n\n"),
Err(RequestError::BadHeader(2))
);
}
#[test]
fn header_name_with_space_or_empty_name_is_bad() {
assert_eq!(
parse_request("GET / HTTP/1.1\nBad Name: x\n\n"),
Err(RequestError::BadHeader(2))
);
assert_eq!(
parse_request("GET / HTTP/1.1\n: x\n\n"),
Err(RequestError::BadHeader(2))
);
}
#[test]
fn missing_blank_line_is_truncated() {
assert_eq!(
parse_request("GET / HTTP/1.1\nHost: a"),
Err(RequestError::Truncated)
);
}
#[test]
fn fields_point_into_the_input() {
let raw = String::from("DELETE /item HTTP/1.1\n\n");
let req = parse_request(&raw).unwrap();
let start = raw.as_ptr() as usize;
assert_eq!(req.path.as_ptr() as usize - start, 7);
}
}Reference solution
#[derive(Debug, PartialEq)]
pub struct Request<'a> {
pub method: &'a str,
pub path: &'a str,
pub headers: Vec<(&'a str, &'a str)>,
pub body: &'a str,
}
#[derive(Debug, PartialEq)]
pub enum RequestError {
BadRequestLine,
BadVersion(String),
BadHeader(usize),
Truncated,
LengthMismatch { declared: usize, actual: usize },
}
pub fn parse_request<'a>(raw: &'a str) -> Result<Request<'a>, RequestError> {
let mut lines = raw.split('\n');
let request_line = lines.next().ok_or(RequestError::BadRequestLine)?;
let parts: Vec<&str> = request_line.split(' ').collect();
if parts.len() != 3 || parts.iter().any(|part| part.is_empty()) {
return Err(RequestError::BadRequestLine);
}
let (method, path, version) = (parts[0], parts[1], parts[2]);
if version != "HTTP/1.1" && version != "HTTP/1.0" {
return Err(RequestError::BadVersion(version.to_string()));
}
let mut headers = Vec::new();
let mut declared_length: Option<(usize, &str)> = None;
let mut consumed = request_line.len() + 1;
let mut found_blank = false;
for (index, line) in lines.enumerate() {
consumed += line.len() + 1;
if line.is_empty() {
found_blank = true;
break;
}
let number = index + 2;
let (name, value) = line.split_once(':').ok_or(RequestError::BadHeader(number))?;
if name.is_empty() || name.contains(' ') {
return Err(RequestError::BadHeader(number));
}
let value = value.trim_matches(' ');
if declared_length.is_none() && name.eq_ignore_ascii_case("Content-Length") {
declared_length = Some((number, value));
}
headers.push((name, value));
}
if !found_blank {
return Err(RequestError::Truncated);
}
let body = raw.get(consumed..).unwrap_or("");
if let Some((number, value)) = declared_length {
let declared: usize = value.parse().map_err(|_| RequestError::BadHeader(number))?;
if declared != body.len() {
return Err(RequestError::LengthMismatch {
declared,
actual: body.len(),
});
}
}
Ok(Request {
method,
path,
headers,
body,
})
}Limitations
We tried to make Temper Bench v2 honest and high-quality. The tasks are written from scratch, solutions are checked by the compiler and hidden tests, and the benchmark has been checked for leaks. But we are not a research lab and have not built benchmarks before, so it is too early to put it on par with HumanEval or MBPP, which the community has vetted for years. We made it because Rust has few ready alternatives beyond translations of those same sets. A final verdict on a model is still best made on your own tasks, by trying it in practice.